# Hosted LLM APIs or Self-Hosted Open Models?

URL: https://donedeploy.com/hosted-or-self-hosted-llm/
Published: 2026-09-25

Every team building with language models faces the same early decision: call a hosted API from OpenAI, Anthropic or Google, or run an open model such as Llama, Mistral, Qwen, Gemma or DeepSeek on your own infrastructure. Both are reasonable. The right answer depends on your workload, your data and your team, and it often changes as a product grows. Here is how to think it through.

## Capability

The leading hosted models (GPT, Claude and Gemini families) are generally strongest on hard reasoning, long and complex instructions, coding and agent style tool use. Open models have closed much of the gap and handle many focused tasks very well: classification, extraction, summarisation and retrieval augmented question answering.

The practical test is your own evaluation set. If an open model passes it, capability is not the deciding factor. If only the frontier models pass, the decision is mostly made for you, at least for now.

## Cost

Hosted APIs charge per token. You pay nothing when idle and costs scale linearly with use. That is ideal for prototypes, spiky traffic and low volume.

Self-hosting charges per GPU hour, whether the GPU is busy or not. It becomes cheaper only when utilisation is high and steady, and only if you count the full cost: engineering time, monitoring, on-call and upgrades, not just instance prices.

Situation Usually cheaper Prototype or low volume Hosted API Unpredictable, bursty traffic Hosted API High, steady volume on a narrow task Self-hosted Batch jobs that can run on spot capacity Self-hosted

Run the numbers with your real token counts. Prompt caching and batch discounts from the providers can change the comparison considerably.

## Privacy and compliance

This is often the real driver. Hosted providers offer business terms that typically exclude API data from training, plus options for data retention, regional processing and dedicated deployments. Azure OpenAI, Amazon Bedrock and Google Vertex AI let you use major models inside your existing cloud account and region, which helps with GDPR and procurement.

Self-hosting keeps data entirely within infrastructure you control. For some regulated data, contractual requirements or customer expectations, that is the simplest way to meet them. Read the provider terms with whoever handles compliance before assuming either way.

## Latency

Hosted APIs add network round trips and can slow down under provider load, though they run on large, well tuned fleets. A self-hosted model close to your application can give consistent, low time to first token, particularly for small models, but only if you provision enough capacity for peaks. An overloaded self-hosted server is slower than any API.

## Effort

A hosted API needs an API key, a client library and sensible retry logic. Self-hosting needs GPU capacity, an inference server such as vLLM, autoscaling, monitoring, security patching, model upgrades and someone who understands all of it. That effort is ongoing, not one off.

## Control

Self-hosting gives you a fixed model version that changes only when you decide. Hosted models are updated and eventually retired on the provider’s schedule, which means re-running your evaluations when that happens. Self-hosting also lets you fine-tune freely and inspect behaviour more deeply.

## You do not have to pick one

Many teams end up with both:

- A hosted frontier model for complex or low volume tasks.
- A self-hosted open model for high volume, simple or sensitive tasks.
- A gateway (LiteLLM, or one built in-house) that routes requests, tracks cost per team and allows fallback between providers.

Build against an OpenAI compatible interface from the start and this becomes routing configuration rather than a migration project.

## A simple way to decide

1. Build the evaluation set first.
2. Prototype on a hosted API to learn what the feature needs.
3. Measure volume, latency needs and data constraints once real usage exists.
4. Move specific workloads to self-hosting when the numbers or compliance justify it.

DoneDeploy helps teams make this call with their own data and then build either path, or both, on AWS, GCP, Azure or on-premises. If you want a second opinion on your setup, [let’s talk](https://donedeploy.com/lets-talk/).