SLOs and Error Budgets: Agreeing How Reliable Is Reliable Enough
How to set your first SLOs and error budgets: pick the user journeys, choose an SLI you can measure, set a target you can defend, alert on burn rate and agree what happens when the budget runs out.

Ask a team how reliable their service should be and the first answer is usually “always up”. Ask how reliable it is today and the answer is usually a shrug, or an uptime figure from a status page that pings the home page once a minute. Neither answer helps anyone decide whether to ship the next feature or spend the week fixing the flaky checkout.
Service level objectives (SLOs) and error budgets exist to settle that argument with a number both product and engineering have agreed in advance. Most guides explain the definitions well. This post is about the part teams get stuck on: choosing the first few, and making them change a decision rather than decorate a dashboard.
The three terms, briefly
- SLI (service level indicator): a measurement of what users experience, written as a ratio. Good events divided by valid events. For example, the share of checkout requests that returned successfully in under 500 ms.
- SLO (service level objective): the target for that ratio over a window. For example, 99.5% of checkout requests are good over a rolling 30 days.
- Error budget: what the SLO allows to go wrong. At 99.5%, half a percent of requests in the window may fail or be slow before you have missed the objective.
An SLA is different: it is a contract with customers, with penalties attached. Keep SLOs internal and stricter than any SLA, so you notice a problem before it costs money.
Step 1: pick the journeys that matter
Do not start with services. Start with what a user is trying to do. For most products, three to five journeys cover what people would complain about: signing in, loading the main page or dashboard, searching, paying, and an API call that a customer’s integration depends on.
One SLO on a journey users care about is worth more than twenty on internal components. A queue consumer that falls behind matters only because some user-facing thing gets slow or stale, and that is where you measure it.
Step 2: choose an SLI you can actually measure
The best SLI is measured as close to the user as you can manage. In practice the choice usually looks like this:
| Where you measure | Good for | Watch out for |
|---|---|---|
| Load balancer or ingress logs | Availability and latency of HTTP requests, cheap to start | Misses failures that happen before a request arrives, such as DNS or a broken front end |
| Application metrics or traces | Per-endpoint detail, custom definitions of “good” | Counts nothing when the application itself is down |
| Synthetic checks | Journeys with little traffic, end to end flows | A probe is not a user, and a few probes give coarse numbers |
| Real user monitoring | What browsers and apps really saw | More setup, and noisier data |
Most teams start with load balancer or application data, because it is already there. If your services are instrumented with OpenTelemetry, request counts and durations per route are enough for availability and latency SLIs.
Two definitions do most of the work:
- Availability: requests that did not fail on the server side, divided by all valid requests. Decide up front which status codes count as failures, and exclude traffic that is not a real user, such as health checks.
- Latency: requests faster than a threshold, divided by all valid requests. A threshold (“under 500 ms”) is easier to reason about as a budget than an average or a percentile.
Step 3: set a target you can defend
The target is a business decision, not a technical one, and higher is not automatically better. Each extra nine costs a great deal more and leaves a much smaller margin for change:
| SLO over 30 days | Error budget, as time |
|---|---|
| 99% | about 7.2 hours |
| 99.5% | about 3.6 hours |
| 99.9% | about 43 minutes |
| 99.95% | about 22 minutes |
| 99.99% | about 4.3 minutes |
Two rules keep the first target honest:
- Look at the last few months before choosing. If the service achieved 99.7% without anyone being upset, a 99.99% target only creates a permanent failure.
- Do not promise more than your dependencies give you. A service that relies on a database, a cloud region and a payment provider cannot be more reliable than all of them together, unless you have designed around their failures.
Start a little below what you achieve today, run it for a month or two, and adjust once you have seen how it behaves.
Step 4: alert on burn rate, not on every error
An SLO also gives you better alerts. Instead of paging on “error rate above 1%”, page when the error budget is being spent fast enough to matter. That rate is the burn rate: a burn rate of 1 spends exactly the budget over the window; a burn rate of 10 would spend it in a tenth of the time.
The pattern from Google’s SRE workbook is a good default to start from:
- Page when the burn rate is high over both a long and a short window, for example about 14 times over the last hour and the last five minutes. That is a real outage eating the budget quickly.
- Page at a lower rate over longer windows, for example about 6 times over six hours and thirty minutes, to catch a serious slow burn.
- Open a ticket, not a page, for a slow burn over days. Someone should look this week, not at 3 a.m.
The short window matters: it stops the alert from firing long after the problem has been fixed. Tools such as Sloth and Pyrra can generate these Prometheus rules from a short SLO definition, and Datadog and Grafana Cloud have SLO features of their own. Keep the definitions in Git next to your other alert rules, so a change to a target is reviewed like any other change.
Step 5: agree what happens when the budget runs out
This is the step most teams skip, and it is the one that makes SLOs worth having. Write down an error budget policy before you need it, and get product and engineering to sign it together. A simple version:
- Budget healthy: ship as normal. Planned risky work, such as migrations or a major upgrade, is fine.
- Budget under pressure: risky changes need a second look, and the team checks which recent changes spent the budget.
- Budget exhausted: feature releases pause for that service, except fixes and changes that improve reliability, until the budget recovers or the team and product owner agree an exception in writing.
The policy cuts both ways. If a service sits well inside its budget month after month, that is a signal you can release faster or spend less on redundancy, not a reason to celebrate quietly.
It also gives your deploy pipeline something useful to read. A release step that checks the remaining budget before rolling out a large change fits naturally after the gates described in our CI process.
Mistakes we see most often
- Too many SLOs. Forty objectives nobody reads are worse than four that drive a weekly decision.
- Measuring the server, not the user. CPU and memory are causes, not experiences. Keep them on dashboards, out of SLOs.
- Targets copied from someone else. 99.99% sounds good in a planning meeting and becomes a constant red light in production.
- No policy. An SLO with no agreed consequence is a chart. People notice when it goes red, then carry on as before.
- Never revisiting them. Review targets each quarter. Traffic, architecture and what customers expect all change.
SLOs for AI features
The same approach works for LLM features, with a few extra SLIs: time to first token, the share of responses that pass your automated checks, and the share of requests that fell back to a cheaper or backup model. We covered what to measure in Observability for LLM Applications; an SLO is the target you set on top of those measurements.
Where to start this week
- List the three user journeys that would cause the most complaints if they broke.
- For each, write one availability SLI and one latency SLI using data you already collect.
- Look at the last 90 days and set targets slightly below what you achieved.
- Draft a one-page error budget policy and put it in front of the product owner.
Setting up SLIs, dashboards and burn-rate alerts around user-facing symptoms is part of our AI observability work, with the rules kept in your Git. If you want help choosing the first few objectives, book a free call and we will work through them with you.


