TGI (Text Generation Inference)
Hugging Face's inference server for LLMs. Text Generation Inference (TGI) is a server from Hugging Face for serving large language models on GPUs. It provides continuous batching, streaming responses, quantisation support and Prometheus metrics, and it powers parts of Hugging Face's own hosted inference. It is distributed as a container image.
When it is the right choice
- You already run TGI or Hugging Face Inference Endpoints and want to keep a consistent stack
- You want a container that serves a Hub model with little configuration
- You need built-in metrics and tracing for model serving
- Before choosing it for a new project, check the project's current maintenance status and compare with vLLM
How we use it
- Run TGI containers on Kubernetes GPU nodes with readiness checks and autoscaling
- Scrape TGI metrics into Prometheus and build latency and throughput dashboards
- Pin images and model revisions so every deployment is reproducible
- Benchmark TGI against vLLM on your workload when planning upgrades
Kubernetes
NVIDIA GPUs
PrometheusRelated articles
From Idea to AI Feature in Six WeeksHow an AI project runs end to end in six weeks: discovery, design, prototype, infrastructure, rollout, then measuring and improving.
CI/CD for Models and PromptsEvaluation gates, model registries, canary releases and rollback: applying CI/CD discipline to prompts and models in LLM applications.
Fine-Tuning or Prompting? A Decision Guide for Small TeamsWhen prompting, RAG or fine-tuning is the right way to adapt a language model, and what each one costs a small engineering team.TGI (Text Generation Inference): frequently asked
Should we choose TGI or vLLM?
Both serve open models with batching and an HTTP API. For new projects vLLM usually has wider community activity; check TGI's current status and benchmark both on your model.
Can TGI serve any Hugging Face model?
It supports many popular text generation architectures, but not every model. Check the supported models list for your model first.
Running TGI (Text Generation Inference), or deciding whether to? Tell us what you have; the first consultation is free.
Let's talk »
