Postmortem: Change the System, Don’t Find the Culprit

From the field: the same failure, four months apart

Here is the timeline of the first incident:

07:50  first error - mobile app: "cannot connect"
07:54  reported in the support channel     <-- NO alert, a customer found it
08:12  diagnosis: TLS certificate expired at 03:00
08:31  new certificate issued
08:58  rollout finished, error rate back to normal

       68 minutes total, ~14,000 failed requests

I wrote the postmortem the next day. Six pages. The timeline was pulled from the logs minute by minute, the root cause was correct, nobody was blamed. Everyone nodded in the meeting. It really was a good document.

Four months later, on a Friday morning, a certificate between internal services expired. The same failure, for the second time. I opened the document again. The actions section had nine items, and all nine were owned by “the infrastructure team”. Two of the nine were closed — the two easiest ones.

I should have known better. I had written exactly this in the post about delegation: work owned by everyone is owned by nobody. I broke my own rule inside my own document.

What a postmortem is not

In most teams a postmortem turns into one of three things, and none of them work:

• A court. People discuss who did what, and the root cause ends in someone’s name. At the next incident, nobody speaks.
• An archive. The document is written, filed, and never opened again. Writing feels like an action, but it is not one.
• A ceremony. A meeting happens, someone says “let’s be more careful”, everyone leaves. Care is not a system property.

A postmortem has one job: find the change that makes this failure impossible, or at least expensive, to repeat for the same reason. The document is a side product. How the meeting opens is a separate topic, and on its own it decides everything — I wrote about that in the post on psychological safety. Here I am looking at the document itself, and at what happens after it.

Five Whys breaks at the third step

Five Whys is a good tool, but almost every team trips at the same place: at the third question the answer is a person, and the chain stops there. In our incident it went exactly like this:

1. Why was there an outage?   -> The TLS certificate expired.
2. Why did it expire?         -> It was not renewed.
3. Why was it not renewed?    -> No reminder had been set up.
                                 ^^^ BREAK: the answer is a person.
                                     If the chain stops here, the action
                                     becomes "let's be more careful".

   (correct continuation, from the same step)
3. Why did renewal depend on a human remembering?
                              -> There was no automatic renewal.
4. Why was there no automatic renewal?
                              -> This certificate was issued by hand,
                                 outside the automation.
5. Why were hand-issued certificates invisible?
                              -> There was no certificate inventory.
                                 Nobody knew the full list.

   ACTION: build the inventory + alert 30 days before expiry  (one name, one date)

Here is the difference. At the end of the first chain there is a person. At the end of the second one there is a missing system. The action from the first chain cannot be implemented. The action from the second one takes a week and covers every certificate of the same kind.

A practical rule: whenever the answer is someone’s name, or “forgot”, “missed it”, “was not careful”, ask the question again — “why did this depend on a human remembering?” That single sentence turns a court into an engineering meeting.

The one-page template

I wrote six pages and nobody opened them twice. The template we use now is one page, and it is narrow on purpose. Anything longer does not get written:

POSTMORTEM - <incident name>        <date>  |  author: <one name>
------------------------------------------------------------------
IMPACT     : what broke, for how long, how many users / requests
TIMELINE   : first error -> detected -> diagnosed -> fixed
             (a clock time on every line; from logs, not from memory)
DETECTED BY: an alert, or a customer?
             a customer --> that alone is an action item
ROOT CAUSE : one sentence. "forgot" / "was not careful" is not allowed.
WHY NOW    : this system ran for months; what changed this time?
ACTIONS    : 3 items maximum
             <what> - <one name> - <date> - <backlog id>

Two lines are there on purpose. “Detected by” makes it visible when a customer found the problem first. In our incident there was no alert, and that was a bigger problem than the outage itself. “Why now” is the question people skip most often: the system had been running for months, so what changed that day? The answer is usually the root cause itself.

Actions: one name, one date, the same queue

Three rules, starting with the one that helps most:

1. The owner is one name. The moment you write a team name, the action dies. The person is responsible for closing the action, not for the failure — keep those two apart.
2. Actions go into the normal backlog. Do not start a separate list. We did, and four months later nobody had opened it. In the same queue, you at least have to defend the item at refinement.
3. Three items maximum. Writing nine actions is the polite way of doing none. When you have to pick three, you start thinking about which one actually prevents the repeat.

What to track

You cannot tell whether your postmortem culture works by how it feels. Four numbers are enough:

There is one more thing worth looking at, even if you do not put a number on it: who found the incident? Of the last ten, how many were reported by an alert and how many by a customer? For a while ours was 6 out of 10 by customers. None of the postmortems said so, because the template had no line for it. What you do not measure does not get discussed.

What did not work for me

I tried three things and all three failed. I am writing them down because you will probably try them too:

• Making the template richer. I added sections for “contributing factors”, “alternative scenarios” and “lessons learned”. The document got longer and nobody read it. A wider template does not improve quality. It turns filling it in into a chore.
• A postmortem for every incident. For a while we wrote one even for problems under 20 minutes. By the sixth week the quality had collapsed. Now the threshold is clear: anything that affects customers or lasts more than 30 minutes — plus any incident with an unknown root cause, however short.
• Tracking actions in a separate list. That was my most expensive mistake. The list became a graveyard of its own. When we moved the items into the normal backlog, the close rate went up by itself.

Checklist

Conclusion

After the second certificate incident, the postmortem meeting took 25 minutes. The document was one page. It had three actions, each with a name and a date. All three closed — one of them a week late.

It was not a better document than the first one. It was worse: less detail, less analysis, fewer pages. But it did the one thing the first document never did. It changed the system.

Track this and ignore the rest: did the same root cause come back? If it did, the first postmortem was not done. It was written — and that is a different thing.