Serving and AI platform
Ray Serve
Scalable model serving built on the Ray framework. Ray Serve is the model serving library of Ray, an open-source framework for distributed Python computing. It lets you compose several models and business logic into one application, scale each part independently and allocate fractional GPUs. On Kubernetes it is usually run with the KubeRay operator.
When it is the right choice
- Your application chains several models or steps that need to scale separately
- Your team already uses Ray for data processing or training
- You want to write serving logic in plain Python rather than configure several servers
- Not needed when you only serve one LLM behind an API; vLLM alone is simpler
How we use it
- Run Ray clusters on Kubernetes with KubeRay, provisioned and versioned as code
- Combine Ray Serve with vLLM for LLM steps inside a larger pipeline
- Collect Ray metrics in Prometheus and trace requests across steps
- Tune replica counts and GPU allocation from production metrics
Kubernetes
vLLM
NVIDIA GPUs
PrometheusFrom the blog
From Idea to AI Feature in Six WeeksHow an AI project runs end to end in six weeks: discovery, design, prototype, infrastructure, rollout, then measuring and improving.
CI/CD for Models and PromptsEvaluation gates, model registries, canary releases and rollback: applying CI/CD discipline to prompts and models in LLM applications.
Fine-Tuning or Prompting? A Decision Guide for Small TeamsWhen prompting, RAG or fine-tuning is the right way to adapt a language model, and what each one costs a small engineering team.
Related articles
From Idea to AI Feature in Six WeeksHow an AI project runs end to end in six weeks: discovery, design, prototype, infrastructure, rollout, then measuring and improving.CI/CD for Models and PromptsEvaluation gates, model registries, canary releases and rollback: applying CI/CD discipline to prompts and models in LLM applications.Fine-Tuning or Prompting? A Decision Guide for Small TeamsWhen prompting, RAG or fine-tuning is the right way to adapt a language model, and what each one costs a small engineering team.Questions
Ray Serve: frequently asked
How is Ray Serve different from vLLM?
vLLM is an engine for serving language models efficiently. Ray Serve is a general framework for composing and scaling models and code, and can use vLLM inside it.
Do we need Kubernetes to use Ray Serve?
No, Ray can run on plain VMs. For production we usually recommend Kubernetes with KubeRay for upgrades, recovery and resource management.
Running Ray Serve, or deciding whether to? Tell us what you have; the first consultation is free.
Let's talk »
