SLOs and Error Budgets: A Negotiation Tool, Not a Target

From the field: a 50-minute argument, one number

After that meeting, the first thing I did was open three months of load balancer logs. The question was simple: how many order requests came back successful and on time? The answer was very different from what our dashboard said.

The dashboard showed uptime: 99.98%. The health check said “I am alive” every 10 seconds, and it was telling the truth. But the order requests told another story. In June, 14,200 of 6.1 million order requests either failed or took longer than 1 second. Success rate: 99.77%. I was measuring the wrong thing. I was measuring whether the server was alive, not whether the customer’s order went through.

The second surprise was where those 14,200 bad requests came from. My guess was “the database”. It was not:

On every deploy, the monolith shut its pods down one by one. But a pod that was shutting down died before it finished the requests it was holding. Once a week, while the market was open, a few hundred customers had their orders cut off halfway. Nobody had noticed, because each one was a spike of a few seconds, and no alert looked long enough to catch it.

I went into the second meeting with this table. The discussion took 10 minutes. The product manager looked at it and asked one thing: “If we fix the deploy, how much does this number drop?” Now we were asking the same question.

The SLI: what the user sees

An SLI (service level indicator) is what you measure. An SLO (service level objective) is the target you set for it. The order matters: a good target on the wrong SLI only gives you false confidence backed by a correct number.

A good SLI has three properties. It measures what the user feels. It is written as a ratio: good events divided by all events. And it is measured as close to the user as possible. This is where we ended up for the order service:

• Good event: the order request returned a successful response within 1 second.
• All events: every order request, except 4xx user errors. An order rejected for insufficient balance is not a service failure. It is proof the service works.
• Where we measure: the load balancer. The application cannot log the request of a pod that has already died. The load balancer can.

We did not put an SLO on everything. Orders and withdrawal requests got 99.9%; account statements and reports got 99.5%. If a statement opens three minutes late, the customer complains. If an order goes out three seconds late, the price has moved. Give them the same target and both lose their meaning.

The maths: how many minutes is 99.9%?

The product manager first suggested 99.99%. It sounds reassuring. When we did the maths, the whole room went quiet at the same moment:

# SLO: in the last 30 days, 99.9% of orders succeed within 1 s
target        = 0.999
window_min    = 30 * 24 * 60              # 43,200 minutes
budget_min    = window_min * (1 - target) # 43.2 minutes of full outage

# Same maths by request (this is what we actually measure)
monthly_reqs  = 6_100_000
budget_reqs   = monthly_reqs * (1 - target) # 6,100 bad requests

# June, the reality
bad_reqs      = 14_200
consumed      = bad_reqs / budget_reqs      # 2.33 --> 233% of the budget

# if the target were 99.99%
budget_reqs   = 610                         # one bad deploy was ~2,150

The last line ended the discussion. If a monolith deploys once a week and one deploy creates 2,150 bad requests, a budget of 610 is gone on the first Tuesday of the month. 99.99% was not a target. It was a wish. The table shows the difference:

We agreed on 99.9%, with one condition: June’s reality was 99.77%. So on the day we set the target, we were spending more than twice the budget. We did not hide it. That was exactly why the first two sprints went to reliability. In those two sprints we focused on one thing: a pod that is shutting down first stops taking new requests, then finishes the ones it holds, and only then dies. Bad requests per deploy dropped from 2,150 to about 200. A target should be a bit better than today, not out of reach. People start ignoring an impossible target in the first week.

What happens when the budget runs out?

The real power of an SLO is not the target. It is writing down, in advance, what happens when the budget is gone. An SLO without a policy is a graph nobody looks at. Ours is one page, and the product manager and I both signed it:

The policy was tested in September. On 9 September, a bad release burned 70% of the budget in 40 minutes. The window was already at 38%, so it reached 108% the same day. The freeze started the next morning. The root-cause fix shipped on 12 September, and the 7-day burn rate dropped below 1 on 17 September. The freeze lasted seven days (10–17 September). Two features moved to the next sprint.

The argument I expected never came. That morning the product manager wrote one line in the team channel: “It is our rule.” A rule written in July, on a calm day, when nobody was in the middle of a fire, was not up for negotiation in the middle of a fire in September. Not because it was a great rule, but because it was written before.

Burn rate: speed, not a threshold

After setting the SLO, we looked at our old alerts: “fire if the error rate is above 5% for 10 minutes.” This threshold gets two things wrong. For a 3-minute spike it may fire, or stay silent, for the wrong reasons. And it never sees a small problem that lasts all day. Yet that small problem is usually what empties the budget.

Burn rate is how many times faster than normal you are spending the budget. At 1, the budget lasts exactly 30 days. At 14.4, it lasts two days. We built the alert on this speed:

# burn_rate = error_rate / (1 - target)
# target 99.9% --> an error rate of 1.44% is a burn rate of 14.4

# FAST BURN: page the on-call engineer (at night too)
#   2% of the budget is gone in 1 hour
long_window = 1 hour,   short_window = 5 min,  burn_rate > 14.4

# SLOW BURN: team channel (checked in the morning)
#   5% of the budget is gone in 6 hours
long_window = 6 hours,  short_window = 30 min, burn_rate > 6

# Both windows must agree: the long one answers "is it really happening",
# the short one answers "is it still happening".
# When the problem stops, the short window goes quiet at once,
# so the alert does not keep firing for hours.

In October we saw it work. A configuration error in one type of derivatives order made 0.6% of all order traffic fail. The old 5% threshold would never have seen it. The slow-burn alert reached the team channel in the 6th hour, and the fix went out the next morning. The total cost was about 9% of the budget. Without the alert, we would most likely have found it in the customer support queue.

For now, only the order service has a burn rate alert. The other services still use fixed thresholds. It is tempting to add it everywhere at once, but a burn rate alert on the wrong SLI just tells you about the wrong thing faster. SLI first, alert second.

How it goes wrong

• One number for everyone. A single SLO for the average of all services. The order service’s bad day gets averaged away by the report service’s good day.
• A 100% target. Zero budget means no change is allowed. In practice, you either never ship or everyone ignores the target.
• Exception inflation. Every “this feature is too important” release during a freeze makes the next freeze weaker. Our exception list has three lines, and there is no fourth.
• An unused budget. If you use less than 10% of the budget, that is also a signal: the target is too loose, or you ship too slowly. The budget is there to be spent.

What did not work for me

• Using uptime as the SLI. That was where we started: 99.98%. The customer saw 99.77%. The gap was pods that were about to die but still said “I am alive”.
• Tying the freeze to the 30-day window. Our first policy said “the freeze ends when the budget is back under 100%”. In a rolling window, bad days stay in the window for 30 days, so this rule would have locked us for a month. We tied the end to the root-cause fix and the last 7 days’ speed instead.
• An alert on “50% of the budget left”. Both late and noisy. Half the budget can go in two separate incidents or in one slow leak. The question an alert should answer is not “how much is left”, but “how fast is it going”.

What to track

Checklist

Conclusion

In that 50-minute argument in July, we were both right, and neither of us could prove it. The product manager knew customers wanted new features. I knew the service was fragile. What was missing was not information. It was a shared unit.

The error budget gave us that unit. Today the order service has used 39% of its 30-day budget. At this quarter’s planning, nobody asked for a reliability sprint; there was enough budget, and the features shipped. The same number started the freeze in September.

If you set an SLO like a target, you try to hit it. If you set it like an agreement, you make decisions with it. The difference: a target asks “how good are we?”, a budget asks “what do we do now?”