# Running Open Models in Production: A Practical Starting Point

URL: https://donedeploy.com/running-open-models/
Published: 2026-08-18

Open weight models such as Llama, Mistral, Qwen, Gemma and DeepSeek have become good enough for a wide range of business tasks: summarising documents, classifying tickets, extracting fields from emails, answering questions over internal knowledge. Running one yourself gives you control over data, cost and behaviour. It also makes you responsible for a new kind of workload. This post sets out a sensible first path from “we downloaded a model” to “it serves real traffic”.

## Start with the task, not the model

Before choosing weights, write down what the model has to do and how you will know it did it well. Collect 50 to 200 real examples with the answers you expect. That small set does more for your decision than any public leaderboard, because it measures the model on your data.

With the examples in hand, try a few candidates of different sizes. A smaller model that passes your examples is cheaper and faster to run than a large one that passes them slightly better. Check the licence too: Llama, Gemma and some others carry their own terms, while many Mistral and Qwen releases use Apache 2.0. Read the actual licence for the exact version you deploy.

## Size the hardware honestly

Memory is the first constraint. The weights must fit in GPU memory, and the KV cache (the per-request memory that grows with context length and concurrency) needs room on top of that. A rough guide for the weights alone:

Precision Memory per billion parameters FP16 / BF16 about 2 GB 8-bit about 1 GB 4-bit about 0.5 GB

Quantised models (AWQ, GPTQ, GGUF formats) cut memory sharply, usually with a modest quality cost. Test the quantised version against your examples rather than assuming it is fine.

If a model needs more memory than one GPU has, you will need tensor parallelism across several GPUs, which adds cost and operational complexity. For a first deployment, prefer a model that fits on a single card.

## Pick a serving layer

Do not write your own HTTP wrapper around a model. Use an inference server:

- **vLLM** for high throughput GPU serving with continuous batching and an OpenAI compatible API.
- **Text Generation Inference (TGI)** from Hugging Face, a similar production server.
- **Ollama** or **llama.cpp** for development machines, CPU, or small single user setups.

An OpenAI compatible endpoint is worth insisting on. It lets your application switch between a self-hosted model and a hosted API by changing a base URL, which keeps your options open.

## Treat it like any other service

A model server is a stateless service with a large startup cost. The usual practices apply, with a few adjustments:

1. **Package it reproducibly.** Pin the container image and the exact model revision. Store weights in object storage or a registry you control, not a public download at boot.
2. **Plan for slow starts.** Loading tens of gigabytes takes time. Set readiness probes that wait for the model to load, and keep a warm replica if users cannot tolerate cold starts.
3. **Limit concurrency.** Configure maximum batch size and context length so a burst of long prompts cannot exhaust GPU memory.
4. **Put a gateway in front.** Authentication, rate limits and request logging belong in a gateway or proxy, not in each application.
5. **Measure from day one.** Track time to first token, tokens per second, queue depth, GPU memory and error rates.

## Keep quality under watch

A model that passed your examples last month can disappoint next month, because your inputs drift. Keep the evaluation set in version control, add new failure cases as you find them, and rerun it before every model or prompt change. Sample real outputs regularly and have someone who knows the domain review them.

## A realistic first milestone

A good first milestone is modest: one model, one use case, one GPU node, an OpenAI compatible endpoint behind authentication, basic metrics and an evaluation set that runs in CI. Everything else, from autoscaling to multiple models, builds on that foundation.

DoneDeploy sets up this kind of first production deployment on AWS, GCP, Azure or your own hardware, with the infrastructure and monitoring you would expect for any other service. If you are weighing an open model for a real use case, [let’s talk](https://donedeploy.com/lets-talk/).