On-Call: Handover, Not Heroics

From the field: an empty runbook and 40 minutes

From the outside, what I did that night looks like success: the alert fired, someone woke up, the service recovered. From the inside there are three separate failures.

First, the alert was called PAYMENT_LATENCY and that name told me nothing. Which service, which threshold, what the user feels — none of it. Second, the runbook link existed but the page was empty: someone had created the page, written the title, and left the rest for later. Third, and worst: I did not fix the problem, I erased it. A restart does not remove the root cause. It removes the evidence.

When I told the team in the morning, the answer was: “Yes, that alert always does that. We restart it.” So this was not something I did not know. It was something written down nowhere. The knowledge lived in three people’s heads, and there were five people in the rotation.

The three questions

A person woken at night works with zero context. No coffee, no team, half a brain. That person has three questions, and the only job of an on-call system is to have answered them in advance:

1. What broke? The alert name and message should describe what the user feels. Not CPU_HIGH, but payment-api p95 > 2s for 5 min.
2. What should I do? The runbook. The next three commands, in order.
3. Am I allowed? Can I do this alone at night, or do I have to wake someone up?

The third one is skipped most often. If the authority is not written down, one of two things happens: the engineer freezes and waits for morning, so the outage gets longer; or they act without permission, so something bigger breaks next time. Both are failures of the system, not of the person.

A runbook is instructions, not a document

Our page was empty because we thought of a runbook as a document. Writing a document takes a long time, so it gets postponed. Writing running instructions takes ten minutes:

ALERT: payment-api-p95-high
--------------------------------------------------------------
MEANS      : payment-api p95 > 2s for 5 minutes. The user is
             waiting on the payment screen. No error, but they
             cannot finish.
CHECK FIRST: 1) Grafana "payment-api" dashboard: did error rate rise?
             2) Dependency: card-service p95
             3) Any deploy in the last 2 hours?
DO         : - error rate normal AND a deploy within 2 hours
               --> roll back the release (command below)
             - card-service also slow --> not our problem,
               call the card-service on-call
             - neither --> connection pool saturation,
               command below
COMMANDS   : rollback:  deploy rollback payment-api
             pool:      kubectl -n prod get pods -l app=payment-api
ESCALATE   : no progress in 20 minutes --> team lead
             money is being lost --> immediately, do not wait
DO NOT     : change the schema, fix data by hand, change config permanently

Then we made one rule, and it turned out to be the most useful thing we did: if you cannot write the runbook, the alert does not page a human. If nobody can write down what to do, either the alert is meaningless or nobody knows the answer. In both cases you have no right to wake someone up at night. That alert moves to a dashboard instead.

In the week we applied the rule, the paging list went from 60-something alerts to 38. For none of the 23 we deleted did anyone later say “but we needed that one”. Collecting alerts is easy; deleting them needs an excuse, and the rule gives you one.

Night authority, written in advance

You also need an escalation threshold, because this is where people hesitate most. Ours is three lines:

• The 20-minute rule. No progress in 20 minutes, wake up one more person. No hesitation, no asking for permission.
• Money means now. If payments, orders or any money movement are affected, there is no time condition. Escalate immediately.
• Waking someone up is never a mistake. Nobody is ever criticised for an unnecessary escalation. You have to write this line down, or the rule does not work — people are more afraid of waking a colleague than of extending an outage.

Handover: 15 minutes, three items

The cheapest part of on-call, and the one most often skipped. Once a week, between the two engineers, 15 minutes. A fixed agenda:

1. Open incidents. What is unresolved or half-finished? “We restarted it and we do not know why it broke” is an open incident too.
2. What did I silence this week? Which alert, why, and for how long. This is the most important item on the list.
3. Risk that built up. “Disk is at 80%, it will be full in two weeks.” It will not break this week. It will break during your week.

We learned why the second item matters the hard way. A silence is the one thing in the system that quietly becomes permanent: you mute an alert “just for two hours”, it gets forgotten, and three months later that alert never fires and nobody knows. A silence that is not mentioned at handover is a deleted alert.

The first rotation: a shadow week

Putting a new joiner on-call alone in their third week means leaving all three questions above unanswered. Our rule: the first rotation is a shadow.

During the shadow week their phone does not ring, but they see every alert and they are next to the real on-call engineer during every fix. They have one job: open the runbook for every alert that week and fix what is missing. That gives you two things at once — the person learns the system, and the runbooks stay fresh. The best person to update a runbook is the one reading it for the first time.

What to track

You cannot judge on-call health without measuring it for three months in a row. Whether the load is shared fairly is a separate topic, and I covered it in the burnout post. These four numbers are about how on-call works:

What did not work for me

• Splitting alerts into priorities (P1/P2/P3). Neat on paper. In practice everyone marked everything P1. Only one distinction worked: does this alert wake a human, yes or no? No more than two levels.
• A shared on-call pool. “One rotation for all services” looked fair. Then the on-call engineer got alerts from seven services they did not know, and had to call someone for every single one. On-call should follow the same boundary as ownership.
• Keeping runbooks in the wiki. A runbook in a wiki never gets updated, because it does not change together with the code. We moved them into the repo, so the pull request that changes the service also changes the runbook. Same ownership logic as in the monorepo post.

Checklist

Conclusion

I lost 40 minutes that night, and not only because the alert was bad. The alert reached the right person at the right time. The problem was that there was no system behind it: no instructions, no boundary of authority, and no habit of moving what three people knew out of their heads and onto a page.

You do not fix on-call by finding tougher people. You fix it by putting instructions, written while awake, in front of the person who wakes up.

The test is simple: if someone who joined last month were on-call tonight, how far could they get alone? If the answer is “not very far”, you have an on-call rota. You do not have an on-call system.