Serving and AI platform
Ollama
Run open LLMs locally with one command. Ollama is an open-source tool for downloading and running open-weight language models on a local machine or server. It handles model files, quantisation formats and GPU or CPU execution, and exposes a simple HTTP API. Developers use it to prototype with local models and to run small private deployments.
When it is the right choice
- Developers want to test open models on their own machines
- You need a simple private model server for an internal tool with light traffic
- You want offline or air-gapped model access on a single machine
- Not built for high-concurrency production serving; vLLM or TGI fit that better
How we use it
- Set up Ollama for teams to prototype against the same models they will run in production
- Run Ollama on a private server for internal automations with low traffic
- Provision it with Ansible so the setup is repeatable
- Move workloads to vLLM on Kubernetes when traffic or latency needs grow
Llama
Gemma
Qwen
DockerFrom the blog
From Idea to AI Feature in Six WeeksHow an AI project runs end to end in six weeks: discovery, design, prototype, infrastructure, rollout, then measuring and improving.
CI/CD for Models and PromptsEvaluation gates, model registries, canary releases and rollback: applying CI/CD discipline to prompts and models in LLM applications.
Fine-Tuning or Prompting? A Decision Guide for Small TeamsWhen prompting, RAG or fine-tuning is the right way to adapt a language model, and what each one costs a small engineering team.
Related articles
From Idea to AI Feature in Six WeeksHow an AI project runs end to end in six weeks: discovery, design, prototype, infrastructure, rollout, then measuring and improving.CI/CD for Models and PromptsEvaluation gates, model registries, canary releases and rollback: applying CI/CD discipline to prompts and models in LLM applications.Fine-Tuning or Prompting? A Decision Guide for Small TeamsWhen prompting, RAG or fine-tuning is the right way to adapt a language model, and what each one costs a small engineering team.Questions
Ollama: frequently asked
Is Ollama suitable for production?
For low-traffic internal tools it can be. For many concurrent users we recommend a server designed for batching, such as vLLM.
Does Ollama need a GPU?
No, it can run on CPU, and it supports Apple silicon and NVIDIA GPUs. Speed depends heavily on the hardware and model size.
Running Ollama, or deciding whether to? Tell us what you have; the first consultation is free.
Let's talk »
