# GPU Nodes on Kubernetes Without Wasting Money

URL: https://donedeploy.com/gpu-nodes-kubernetes/
Published: 2026-09-07

GPUs are usually the most expensive line on an AI infrastructure bill, and they are easy to waste. A GPU node that sits idle overnight, or a small model holding a whole card to itself, costs the same as one doing useful work. Kubernetes gives you good tools to avoid this, but none of them are switched on by default. Here is how we approach it.

## Separate GPU nodes from everything else

Put GPU machines in their own node pool and taint them, for example `nvidia.com/gpu=present:NoSchedule`. Only workloads that tolerate the taint and request a GPU land there. Without this, ordinary pods drift onto expensive nodes and keep them from scaling down.

Install the NVIDIA GPU Operator, or your cloud’s managed equivalent on EKS, GKE or AKS. It handles drivers, the device plugin, container toolkit and monitoring exporters, so nodes come up ready to serve.

## Share a GPU when one model does not need all of it

A small model or a development workload often uses a fraction of a modern GPU. There are three main ways to share:

Method How it works Isolation Good for MIG (Multi-Instance GPU) Splits supported GPUs (A100, H100 and similar) into hardware partitions Strong: separate memory and compute Several small production models Time-slicing Device plugin advertises one GPU as several; workloads take turns None for memory Development, light or bursty use MPS (Multi-Process Service) Concurrent processes share compute Limited Throughput oriented sharing

MIG is the safest choice for production because one workload cannot exhaust another’s memory. Time-slicing is simple and works on most GPUs, but a pod that allocates too much memory can starve its neighbours, so set your inference server’s memory limits carefully.

## Scale to zero when nobody is using it

Many internal AI tools see traffic only during working hours. Paying for GPUs at night and at weekends makes no sense for them.

- **Cluster Autoscaler** or **Karpenter** (on AWS) can remove GPU nodes when no pods need them, as long as the node pool minimum is zero.
- **KEDA** can scale a deployment to zero replicas based on queue length, request rate or a schedule, and back up when work arrives.
- A request queue or gateway in front of the model lets the first request wait while a node starts, instead of failing.

The trade-off is cold start time. Provisioning a GPU node, pulling a large image and loading weights can take several minutes. Reduce it by keeping images lean, caching weights on a fast volume or a local disk, and using a scheduled warm-up before working hours.

## Use spot capacity where interruptions are tolerable

Spot or preemptible GPU instances are significantly cheaper than on-demand, but the cloud can reclaim them at short notice. They suit:

- batch jobs such as embedding generation, evaluation runs and fine-tuning with checkpoints;
- inference replicas behind a load balancer, provided at least one replica runs on on-demand capacity.

Spread across several instance types and zones to improve availability, and handle the termination notice so pods drain cleanly. GPU spot capacity can be scarce in some regions, so always have an on-demand fallback.

## Right-size the card

Not every model needs the largest GPU. A quantised 7B or 8B model may run well on an L4 or A10 class card, which costs a fraction of an H100. Test your actual model and context length on two or three instance types and compare cost per thousand requests, not cost per hour.

## Watch utilisation, not just availability

Deploy the DCGM exporter (included with the GPU Operator) and graph GPU utilisation, memory use and power per node in Prometheus and Grafana. Low utilisation over a week is a direct signal that you can share, shrink or schedule. Pair it with per-namespace cost reporting, for example through OpenCost or your cloud’s cost allocation tags, so teams see what their models cost.

## A checklist

1. Dedicated, tainted GPU node pool.
2. GPU Operator installed and DCGM metrics collected.
3. MIG or time-slicing for small workloads.
4. Node pool minimum of zero where traffic allows, with KEDA or a queue.
5. Spot for batch and redundant replicas, on-demand for the baseline.
6. Instance types chosen by cost per request.

DoneDeploy builds and tunes GPU node pools on EKS, GKE, AKS and self-managed clusters, including the autoscaling and cost reporting around them. If your GPU bill has outgrown the work it does, [let’s talk](https://donedeploy.com/lets-talk/).