Installation

Docker is required on the host. Running models on an NVIDIA GPU also requires the NVIDIA driver and Container Toolkit on that host.

1. Docker

Check docker version to confirm setup. If it is not installed, install Docker.

2. NVIDIA Driver

Confirm NVIDIA driver works on the host (nvidia-smi should succeed). If not, install NVIDIA driver.

3. NVIDIA Container Toolkit

Required to use an NVIDIA GPU inside Docker. Check if you already have it installed (Verify GPU Access in Docker).

Install toolkit (Ubuntu / Debian)

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's|deb https://|deb [ signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg ] https://|g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

Install toolkit (RHEL / Rocky / Amazon Linux)

curl -fsSL https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo \
  | sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo
sudo yum install -y nvidia-container-toolkit

Enable GPU Access in Docker

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Verify GPU Access in Docker

docker run --rm --gpus all nvidia/cuda:12.6.3-base-ubuntu22.04 nvidia-smi

Any recent nvidia/cuda base image is fine.

For other distros, rootless Docker, or version-specific installs, see the NVIDIA Container Toolkit install guide.