vLLM
High-throughput inference server for open LLMs. vLLM is an open-source library and server for running large language models efficiently on GPUs. It uses techniques such as PagedAttention and continuous batching to serve many requests at once, and exposes an OpenAI-compatible API. It supports most popular open-weight models and quantised formats.
When it is the right choice
- You are self-hosting open-weight models for production traffic
- You want an OpenAI-compatible endpoint so applications can switch between hosted and self-hosted models
- You need to get the most requests out of expensive GPUs
- Overkill for a single developer testing models locally; Ollama is simpler there
How we use it
- Package vLLM in containers and run it on Kubernetes GPU nodes with health checks and rolling updates
- Export vLLM metrics to Prometheus and alert on queue depth, latency and GPU memory
- Tune batching, context length and quantisation against your real traffic
- Automate model rollouts with GitOps so each change is reviewed and reversible
Kubernetes
NVIDIA GPUs
Llama
PrometheusWhere it fits in our work
Related articles
CI/CD for Models and PromptsEvaluation gates, model registries, canary releases and rollback: applying CI/CD discipline to prompts and models in LLM applications.
GPU Nodes on Kubernetes Without Wasting MoneyNode pools, MIG, time-slicing, scale to zero and spot instances: practical ways to keep Kubernetes GPU costs under control for AI workloads.
vLLM vs Ollama vs TGI: Choosing an Inference ServerHow vLLM, Ollama and Hugging Face TGI differ in throughput, hardware support and operations, and which one fits your workload.vLLM: frequently asked
Can our existing OpenAI client code talk to vLLM?
Yes. vLLM provides an OpenAI-compatible API, so in most cases you change the base URL and model name. Some advanced features differ, so we test your actual calls.
How do we scale vLLM?
Each replica serves one model on one or more GPUs. We scale replicas on Kubernetes based on queue length or latency, and keep warm capacity because loading a model takes time.
Running vLLM, or deciding whether to? Tell us what you have; the first consultation is free.
Let's talk »
