Kubernetes includes experimental support for managing AMD and NVIDIA GPUs (graphical processing units) across several nodes.
This page describes how users can consume GPUs across different Kubernetes versions and the current limitations.
Kubernetes implements Device PluginsSoftware extensions to let Pods access devices that need vendor-specific initialization or setup to let Pods access specialized hardware features such as GPUs.
As an administrator, you have to install GPU drivers from the corresponding hardware vendor on the nodes and run the corresponding device plugin from the GPU vendor:
When the above conditions are true, Kubernetes will expose
nvidia.com/gpu as a schedulable resource.
You can consume these GPUs from your containers by requesting
<vendor>.com/gpu just like you request
However, there are some limitations in how you specify the resource requirements
when using GPUs:
limitssection, which means:
requestsbecause Kubernetes will use the limit as the request value by default.
requestsbut these two values must be equal.
Here’s an example:
apiVersion: v1 kind: Pod metadata: name: cuda-vector-add spec: restartPolicy: OnFailure containers: - name: cuda-vector-add # https://github.com/kubernetes/kubernetes/blob/v1.7.11/test/images/nvidia-cuda/Dockerfile image: "k8s.gcr.io/cuda-vector-add:v0.1" resources: limits: nvidia.com/gpu: 1 # requesting 1 GPU
The official AMD GPU device plugin has the following requirements:
To deploy the AMD device plugin once your cluster is running and the above requirements are satisfied:
kubectl create -f https://raw.githubusercontent.com/RadeonOpenCompute/k8s-device-plugin/v1.10/k8s-ds-amdgpu-dp.yaml
You can report issues with this third-party device plugin by logging an issue in RadeonOpenCompute/k8s-device-plugin.
There are currently two device plugin implementations for NVIDIA GPUs:
The official NVIDIA GPU device plugin has the following requirements:
nvidia-container-runtimemust be configured as the default runtime for Docker, instead of runc.
To deploy the NVIDIA device plugin once your cluster is running and the above requirements are satisfied:
kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/1.0.0-beta4/nvidia-device-plugin.yml
You can report issues with this third-party device plugin by logging an issue in NVIDIA/k8s-device-plugin.
The NVIDIA GPU device plugin used by GCE doesn’t require using nvidia-docker and should work with any container runtime that is compatible with the Kubernetes Container Runtime Interface (CRI). It’s tested on Container-Optimized OS and has experimental code for Ubuntu from 1.9 onwards.
You can use the following commands to install the NVIDIA drivers and device plugin:
# Install NVIDIA drivers on Container-Optimized OS: kubectl create -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/stable/daemonset.yaml # Install NVIDIA drivers on Ubuntu (experimental): kubectl create -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/stable/nvidia-driver-installer/ubuntu/daemonset.yaml # Install the device plugin: kubectl create -f https://raw.githubusercontent.com/kubernetes/kubernetes/release-1.14/cluster/addons/device-plugins/nvidia-gpu/daemonset.yaml
You can report issues with using or deploying this third-party device plugin by logging an issue in GoogleCloudPlatform/container-engine-accelerators.
Google publishes its own instructions for using NVIDIA GPUs on GKE .
If different nodes in your cluster have different types of GPUs, then you can use Node Labels and Node Selectors to schedule pods to appropriate nodes.
# Label your nodes with the accelerator type they have. kubectl label nodes <node-with-k80> accelerator=nvidia-tesla-k80 kubectl label nodes <node-with-p100> accelerator=nvidia-tesla-p100
If you’re using AMD GPU devices, you can deploy Node Labeller. Node Labeller is a controllerA control loop that watches the shared state of the cluster through the apiserver and makes changes attempting to move the current state towards the desired state. that automatically labels your nodes with GPU device properties.
At the moment, that controller can add labels for:
GPU Family, in two letters acronym (-family)
RV - Raven
kubectl describe node cluster-node-23
Name: cluster-node-23 Roles: <none> Labels: beta.amd.com/gpu.cu-count.64=1 beta.amd.com/gpu.device-id.6860=1 beta.amd.com/gpu.family.AI=1 beta.amd.com/gpu.simd-count.256=1 beta.amd.com/gpu.vram.16G=1 beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux kubernetes.io/hostname=cluster-node-23 Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock node.alpha.kubernetes.io/ttl: 0 …
With the Node Labeller in use, you can specify the GPU type in the Pod spec:
apiVersion: v1 kind: Pod metadata: name: cuda-vector-add spec: restartPolicy: OnFailure containers: - name: cuda-vector-add # https://github.com/kubernetes/kubernetes/blob/v1.7.11/test/images/nvidia-cuda/Dockerfile image: "k8s.gcr.io/cuda-vector-add:v0.1" resources: limits: nvidia.com/gpu: 1 nodeSelector: accelerator: nvidia-tesla-p100 # or nvidia-tesla-k80 etc.
This will ensure that the Pod will be scheduled to a node that has the GPU type you specified.
Was this page helpful?
Thanks for the feedback. If you have a specific, answerable question about how to use Kubernetes, ask it on Stack Overflow. Open an issue in the GitHub repo if you want to report a problem or suggest an improvement.