How to Install NVIDIA Container Toolkit and Use GPUs with Docker Containers

To use an NVIDIA GPU with Docker containers, you'll need to set up the NVIDIA Container Toolkit, which allows Docker to access NVIDIA GPUs for running GPU-accelerated applications. In this article, we will explore how to use an NVIDIA GPU in Docker, including the installation process, necessary software, and how to verify GPU usage.

NVIDIA GPU Support: Access NVIDIA GPUs from within Docker containers.

CUDA Toolkit Integration: Provides necessary components to run CUDA applications.

Compatibility: Works with different versions of Docker and various NVIDIA GPUs.

The NVIDIA Container Toolkit is essential if you are working with Docker or other containerization platforms and need access to the GPU's computational capabilities inside your containers. It allows users to build and run GPU accelerated containers. The toolkit includes a container runtime library and utilities to automatically configure containers to leverage NVIDIA GPUs.

CUDA-capable GPU: Ensure you have an NVIDIA GPU installed.

Docker: Docker must be installed on your system. You can install Docker by following the official guide here.

NVIDIA Driver: Ensure that the appropriate NVIDIA driver is installed on your system. You can check the installation with the following command: nvidia-smi.

$ curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

$ sudo apt-get update

$ sudo apt-get install -y nvidia-container-toolkit

$ sudo nvidia-ctk runtime configure --runtime=docker

The nvidia-ctk command modifies the /etc/docker/daemon.json file on the host. The file is updated so that Docker can use the NVIDIA Container Runtime.

$ sudo systemctl restart docker

Run a test container:

$ sudo docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi
$ sudo docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi

This command should output information about your GPU, confirming that the container has access to the GPU.

Now, you can run containers with GPU access using the NVIDIA runtime. For instance, to run a container with specific GPU access:

To access all GPUs:

$ docker run --gpus all nvidia/cuda:12.1-base nvidia-smi

To access a specific GPU (e.g., GPU 0):

$ docker run --gpus '"device=0"' nvidia/cuda:12.1-base nvidia-smi

Tips and Best Practices

Check GPU Usage: Use nvidia-smi inside your container to ensure GPU usage is as expected.

Manage GPU Resources: Use --gpus flag options to specify GPU limits, e.g., --gpus "device=0,1" for multi-GPU setups.

Monitoring: Integrate GPU metrics with your monitoring setup to ensure efficient GPU utilization.

GPUs can be specified to the Docker CLI using either the --gpus option starting with Docker 19.03 or using the environment variable NVIDIA_VISIBLE_DEVICES . This variable controls which GPUs will be made accessible inside the container.

Following these steps will enable you to leverage NVIDIA GPUs within your Docker containers, allowing you to run GPU-accelerated applications seamlessly. Let me know if you need any further assistance with this setup!