So I have two systems running Rocky 8, and were happily trucking along running GPU jobs alongside the CPU intensive tasks. I did a regular yum update, and now neither systems BOINC can see the GPUs anymore. Nvtop and nvidia-smi noth see the cards, but BOINC can't I have tried rebooting, reinstalling the drivers, but nothing seems to work, Here's my logfile from one system It has an Nvidia 1070Ti, and a 1080 that were doing great up until now:

Wed 20 Sep 2023 10:28:32 AM CDT | | Starting BOINC client version 7.20.2 for x86_64-pc-linux-gnu
Wed 20 Sep 2023 10:28:32 AM CDT | | This a development version of BOINC and may not function properly
Wed 20 Sep 2023 10:28:32 AM CDT | | log flags: file_xfer, sched_ops, task
Wed 20 Sep 2023 10:28:32 AM CDT | | Libraries: libcurl/7.61.1 OpenSSL/1.1.1k zlib/1.2.11 brotli/1.0.6 libidn2/2.2.0 libpsl/0.20.2 (+libidn2/2.2.0) libssh/0.9.6/openssl/zlib nghttp2/1.33.0
Wed 20 Sep 2023 10:28:32 AM CDT | | Data directory: /var/lib/boinc
Wed 20 Sep 2023 10:28:32 AM CDT | | No usable GPUs found
Wed 20 Sep 2023 10:28:32 AM CDT | | app version refers to missing GPU type NVIDIA
Wed 20 Sep 2023 10:28:32 AM CDT | Asteroids@home | Application uses missing NVIDIA GPU
Wed 20 Sep 2023 10:28:32 AM CDT | | app version refers to missing GPU type NVIDIA
Wed 20 Sep 2023 10:28:32 AM CDT | Einstein@Home | Application uses missing NVIDIA GPU
Wed 20 Sep 2023 10:28:32 AM CDT | | app version refers to missing GPU type NVIDIA
Wed 20 Sep 2023 10:28:32 AM CDT | Einstein@Home | Application uses missing NVIDIA GPU
Wed 20 Sep 2023 10:28:32 AM CDT | | app version refers to missing GPU type NVIDIA
Wed 20 Sep 2023 10:28:32 AM CDT | Einstein@Home | Application uses missing NVIDIA GPU
Wed 20 Sep 2023 10:28:32 AM CDT | Einstein@Home | Missing coprocessor for task h1_0945.80_O3aC01Cl1In0__O3ASHF1a_946.00Hz_17108_0
Wed 20 Sep 2023 10:28:32 AM CDT | Einstein@Home | Missing coprocessor for task h1_0945.80_O3aC01Cl1In0__O3ASHF1a_946.00Hz_17131_0
Wed 20 Sep 2023 10:28:32 AM CDT | Einstein@Home | Missing coprocessor for task h1_0945.80_O3aC01Cl1In0__O3ASHF1a_946.00Hz_17135_1
Wed 20 Sep 2023 10:28:32 AM CDT | Einstein@Home | Missing coprocessor for task h1_0945.80_O3aC01Cl1In0__O3ASHF1a_946.00Hz_17125_0
Wed 20 Sep 2023 10:28:32 AM CDT | Einstein@Home | Missing coprocessor for task h1_1173.80_O3aC01Cl1In0__O3ASHF1a_1174.00Hz_29653_1
Wed 20 Sep 2023 10:28:32 AM CDT | Einstein@Home | Missing coprocessor for task h1_1173.80_O3aC01Cl1In0__O3ASHF1a_1174.00Hz_29654_1
Wed 20 Sep 2023 10:28:32 AM CDT | | app version refers to missing GPU type NVIDIA
Wed 20 Sep 2023 10:28:32 AM CDT | GPUGRID | Application uses missing NVIDIA GPU
Wed 20 Sep 2023 10:28:32 AM CDT | | app version refers to missing GPU type NVIDIA
Wed 20 Sep 2023 10:28:32 AM CDT | World Community Grid | Application uses missing NVIDIA GPU
Wed 20 Sep 2023 10:28:32 AM CDT | | libc: version 2.28
Wed 20 Sep 2023 10:28:32 AM CDT | | Host name: ceres5
Wed 20 Sep 2023 10:28:32 AM CDT | | Processor: 12 GenuineIntel Intel(R) Xeon(R) W-2133 CPU @ 3.60GHz [Family 6 Model 85 Stepping 4]
Wed 20 Sep 2023 10:28:32 AM CDT | | Processor features: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req md_clear flush_l1d arch_capabilities
Wed 20 Sep 2023 10:28:32 AM CDT | | OS: Linux Rocky Linux: Rocky Linux 8.8 (Green Obsidian) [4.18.0-477.21.1.el8_8.x86_64|libc 2.28]
Wed 20 Sep 2023 10:28:32 AM CDT | | Memory: 15.07 GB physical, 7.73 GB virtual
Wed 20 Sep 2023 10:28:32 AM CDT | | Disk: 1.82 TB total, 1.77 TB free
Wed 20 Sep 2023 10:28:32 AM CDT | | Local time is UTC -5 hours
Wed 20 Sep 2023 10:28:32 AM CDT | | Config: use all coprocessors
Wed 20 Sep 2023 10:28:32 AM CDT | World Community Grid | General prefs: from World Community Grid (last modified 10-Apr-2023 13:44:13)
Wed 20 Sep 2023 10:28:32 AM CDT | World Community Grid | Computer location: work
Wed 20 Sep 2023 10:28:32 AM CDT | World Community Grid | General prefs: no separate prefs for work; using your defaults
Wed 20 Sep 2023 10:28:32 AM CDT | | Reading preferences override file
Wed 20 Sep 2023 10:28:32 AM CDT | | Preferences:
Wed 20 Sep 2023 10:28:32 AM CDT | | max memory usage when active: 13892.27 MB
Wed 20 Sep 2023 10:28:32 AM CDT | | max memory usage when idle: 13892.27 MB
Wed 20 Sep 2023 10:28:32 AM CDT | | max disk usage: 900.00 GB
Wed 20 Sep 2023 10:28:32 AM CDT | | (to change preferences, visit a project web site or select Preferences in the Manager)
Wed 20 Sep 2023 10:28:32 AM CDT | | Setting up project and slot directories
Wed 20 Sep 2023 10:28:32 AM CDT | | Checking active tasks
Wed 20 Sep 2023 10:28:32 AM CDT | Asteroids@home | URL https://asteroidsathome.net/boinc/; Computer ID 747124; resource share 100
Wed 20 Sep 2023 10:28:32 AM CDT | DENIS@home | URL https://denis.usj.es/denisathome/; Computer ID 229958; resource share 100
Wed 20 Sep 2023 10:28:32 AM CDT | Einstein@Home | URL https://einstein.phys.uwm.edu/; Computer ID 13150201; resource share 100
Wed 20 Sep 2023 10:28:32 AM CDT | GPUGRID | URL https://www.gpugrid.net/; Computer ID 609811; resource share 100
Wed 20 Sep 2023 10:28:32 AM CDT | RNA World | URL http://www.rnaworld.de/rnaworld/; Computer ID 133266; resource share 100
Wed 20 Sep 2023 10:28:32 AM CDT | Rosetta@home | URL https://boinc.bakerlab.org/rosetta/; Computer ID 6259074; resource share 100
Wed 20 Sep 2023 10:28:32 AM CDT | SiDock@home | URL https://www.sidock.si/sidock/; Computer ID 50274; resource share 100
Wed 20 Sep 2023 10:28:32 AM CDT | Universe@Home | URL https://universeathome.pl/universe/; Computer ID 667455; resource share 100
Wed 20 Sep 2023 10:28:32 AM CDT | World Community Grid | URL http://www.worldcommunitygrid.org/; Computer ID 8716853; resource share 100
Wed 20 Sep 2023 10:28:32 AM CDT | | Setting up GUI RPC socket
Wed 20 Sep 2023 10:28:32 AM CDT | | Checking presence of 2389 project files
Wed 20 Sep 2023 10:28:37 AM CDT | DENIS@home | Starting task HuVeMOp_20230920165941057386_ORd_k_12-Schmidt_575-conf_306_1
Wed 20 Sep 2023 10:28:37 AM CDT | DENIS@home | Starting task HuVeMOp_20230920165941057386_ORd_k_12-Drouin_EPI_10000-conf_706_0
Wed 20 Sep 2023 10:28:37 AM CDT | DENIS@home | Starting task HuVeMOp_20230920165941057386_ORd_k_12-Drouin_EPI_10000-conf_806_0
Wed 20 Sep 2023 10:28:39 AM CDT | Rosetta@home | Starting task 08aapRun_af2_hal_8_1283_SAVE_ALL_OUT_2962668_255_0
Wed 20 Sep 2023 10:28:39 AM CDT | DENIS@home | Starting task HuVeMOp_20230920165941057386_ORd_k_12-Drouin_EPI_10000-conf_306_0


Restarting the boinc-client service doen't work either. What am I missing?

Have you checked that you have OpenCL component available? Einstein tasks need that. Though Asteroids doesn't and has native CUDA apps.

Are you doubly sure that the drivers are installed?

~$ sudo dmesg | grep nvidia

~$ clinfo

Should see the discovery in the Event Log startup if the cards are seen by the system. Don't know if setting the Event Log coproc_debug would show anymore enlightening information.

[root@ceres5 ~]# dmesg | grep nvidia
[ 3.554258] nvidia: loading out-of-tree module taints kernel.
[ 3.554300] nvidia: loading out-of-tree module taints kernel.
[ 3.554351] nvidia: module license 'NVIDIA' taints kernel.
[ 3.554418] nvidia: module license 'NVIDIA' taints kernel.
[ 3.572209] nvidia: module verification failed: signature and/or required key missing - tainting kernel
[ 3.589182] nvidia-nvlink: Nvlink Core is being initialized, major device number 241
[ 3.592229] nvidia 0000:17:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=io+mem
[ 3.824523] nvidia 0000:b3:00.0: enabling device (0140 -> 0143)
[ 3.824697] nvidia 0000:b3:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[ 4.519343] nvidia-uvm: Loaded the UVM driver, major device number 237.
[ 4.527296] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 535.86.05 Fri Jul 14 20:20:58 UTC 2023
[ 4.757095] [drm] [nvidia-drm] [GPU ID 0x00001700] Loading driver
[ 4.757166] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:17:00.0 on minor 0
[ 4.757295] [drm] [nvidia-drm] [GPU ID 0x0000b300] Loading driver
[ 4.757361] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:b3:00.0 on minor 1


clinfo shows both cards as ready.

Got the same problem with intel igpu and Einstein@Home on the latest update. The vision I went back to - which is a couple old now - doesn't auto suspend gpu compute on use. A bad trend...

---

To anyone who was following this, I saw a new nvidia driver available via yum/dnf, and it corrected my problem after a reboot. Both GPUs ACTIVE and working.