My problem:
Even though I have verified that each NVIDIA GPU is appropriately connected to a PCIe Gen 3.0 slot with 8 lanes (NVIDIA control panel verifies this) when I turn on BOINC to start computing, I see on MSI Afterburner that the usage of 2 of the GPUs immediately goes up to near 100%, but the other two stay at 0 for the GPU not driving the display and around 5% for the GPU hooked up to the display. This while I see 4 separate GPU tasks being worked on like normal in the BOINC Manager "Tasks" screen; 2 of them are completed quickly, the other 2 take forever or eventually error out. In particular I have observed this with different combinations of GPU tasks from Einstein, PrimeGrid, NumberFields, and SRBase, I think it also happens with Asteroids and Amicable Numbers.

The ONLY project that this doesn't happen with is Moo! Wrapper because it downloads singular tasks that can utilize 4 NVIDIA GPUs at once. When these are running, all 4 GPUs show a usage percent from 70-90% and all of their "Bus Interface" readouts in GPU-Z show the optimal "PCIe x 16 3.0 @ x8 3.0"

The Computer:
Motherboard: Gigabyte X99-Gaming G1 WIFI (rev. 1.0)
CPU: Intel Xeon E5-2698 v3 (40 PCIe lanes)
GPUs: 1x 1660; 2x 1660 Supers; 1x 1660 Ti
Operating System and drivers: Newest Version with updates of Windows 10 Pro, latest NVIDIA driver from Dec 12, 2023, last BIOS released for Motherboard

What I think is the cause of the problem that I can't change:
Using GPU-Z I can see that the "Bus Interface" for the 2 cards not being utilized shows "PCIe x 16 3.0 @ x8 1.1" while they are working on the tasks. This leads me to believe that Windows doesn't think the GPUs need their link bandwidth and doing a power saving thing. In addition, if I use the little render test in GPUz, it immediately increases the usage of the GPU hooked up to the display to 90-100% and the Bus Interface goes to "PCIe x 16 3.0 @ x8 3.0".

What I have done to attempt to fix with no luck:
1. Set the Power Plan in windows settings->Power Options to "High Performance" and gone into the advanced power settings and set the PCI Express->Link State Power Management setting to "Off" which should be disabling ASPM (Active State Power Management).
2. In NVIDIA Control Panel->Manage 3D settings->Power management mode: set to "Prefer maximum performance"
3. In the BIOS set "PCIe Slot Configuration" from "Auto" to "Gen 3"

My assumption about what is the issue or what settings affect it may be wrong, please throw out any ideas including registry tweaks - if I bomb the windows install its easy for me to do a clean install.

Do you have a cc_config.xml file and if yes, what's in it?
If it doesn't have the <use_all_gpus>1</use_all_gpus> option enabled in the <options> section, make sure it is enabled.
See https://boinc.berkeley.edu/wiki/Client_configuration for more information about this file.

If you do not have a cc_config.xml file, you can make a fully filled one by going BOINC Manager, Options menu, Event Log options, and without changing anything, click Save.
A full cc_config.xml file with the <use_all_gpus> option pre-enabled (<use_all_gpus>1</use_all_gpus>) will be added to your data directory. All you now have to do is restart your BOINC client, and all your GPUs will be available for use.

Hello, Jord.
My cc_config.xml file in C:\ProgramData\BOINC looks like this:

<cc_config>
<options>
<use_all_gpus>1</use_all_gpus>
</options>
</cc_config>

I found this out a long time ago when I was first building PCs for BOINC. I did not know about the Event Log Options Save function creating one though, that would have been really helpful back in the day.
I also don't think this is the issue because BOINC sees the GPUs and assigns them tasks, as noted in my original post and they are all 4 used for Moo! Wrapper 1 task 4 GPU tasks.

Please see screenshots here:
https://imgur.com/a/8VvsXg9