Disabling unattended upgrades of CUDA toolkit
CUDA Driver/library mismatch
Per default Ubuntu runs automatic upgrades on APT installed packages every night, a process called unattended upgrades. This will sometimes cause issues when the unattended upgrade process tries to update the CUDA toolkit, as it will be able to update the libraries, but cannot update the driver, as it is in use by some application. The result is that the CUDA driver cannot be used because of the driver-library mismatch and the Agile Live applications will fail to start.
In case an Agile Live application fails to start with some CUDA error, the easiest way to see if this is because of a driver-library mismatch, is to run nvidia-smi
in a terminal and check the output:
$ nvidia-smi
Failed to initialize NVML: Driver/library version mismatch
In case this happens, just reboot the machine and the new CUDA driver should be loaded instead of the old one at startup. (It is sometimes possible to manually reload the CUDA driver without rebooting, but this can be troublesome as you have to find all applications using the driver)
How to disable unattended upgrades of CUDA toolkit
One way to get rid of these unpleasant surprises when the applications suddenly no longer can start, without disabling unattended upgrades for other packages, is to blacklist the CUDA toolkit from the unattended upgrades process. This will make Ubuntu still update other packages, but CUDA toolkit will require a manual upgrade, which can be done at a suitable time point.
To blacklist the CUDA driver, open the file /etc/apt/apt.conf.d/50unattended-upgrades
in a text editor and look for the Unattended-Upgrade::Package-Blacklist
block. Add "cuda"
and "(lib)?nvidia"
to the list, like so:
// Python regular expressions, matching packages to exclude from upgrading
Unattended-Upgrade::Package-Blacklist {
"cuda";
"(lib)?nvidia";
};
This will disable unattended upgrades for packages with names starting with cuda
, libnvidia
or just nvidia
.
How to manually update CUDA toolkit
Once you want to manually update the CUDA toolkit and the driver, run the following commands:
$ sudo apt update
$ sudo apt upgrade
$ sudo apt install cuda
This will likely trigger the driver-library mismatch in case you run nvidia-smi
. Then reboot the machine to reload the new driver.
Sometimes a reboot is required before the cuda
package can be installed, so in case the upgrade did not work the first time, try a reboot, then apt install cuda
and then reboot again.