Definition of Done: A Quality Contract, Not a Checklist

From the field: a story with six ticked boxes

The story was: “In a withdrawal request, reject the request if the name of the IBAN owner does not match the name of the account owner.” A customer can only withdraw money to an account in their own name; this is a regulatory rule. The bank’s name check service did the matching.

It was in Sprint 12, from 10 to 21 November. It moved to Done on Thursday 20 November. On that day, our DoD had six items:

1. Code reviewed (at least one approval)
2. Unit and integration tests green in CI
3. Code coverage above 80%
4. Deployed to the test environment
5. Acceptance criteria checked in the test environment
6. Approved by PO

All six were true. Code coverage was 91%. We had tried twelve scenarios in the test environment against the bank’s sandbox service. We showed it in the review on 21 November, and the PO approved it. It went live with the weekly release on Tuesday 25 November at 10:00.

The sandbox returned the name exactly as we sent it. The production service returned only the first given name and the surname. “Fatma Nur Kaya” was three words in our records and “FATMA KAYA” in the bank’s answer. Our comparison ignored upper and lower case, but it expected the full name to match. Every customer with two given names was rejected. All twelve test scenarios used customers with one given name.

It lasted 26 hours. From Tuesday 10:00 to Wednesday 12:00, 212 of 1,460 withdrawal requests were wrongly rejected, which is 14.5%. There was no metric for the rejection rate, so there was no alert either. There was also no switch to turn the check off. To disable it, we had to ship a rollback release, and that took until 12:00. The real fix took a day and a half in Sprint 13.

I had written the DoD. In the Daily Standup post, I even gave its summary: the tests passed, CI/CD deployed to the relevant environment, and the feature was checked there. I chose the words “relevant environment”, and in my head it meant the test environment. Our definition said a story was done. It did not say it was done where the customer sees it.

What a DoD is for: a contract

The Scrum Guide defines the DoD as the commitment of the increment. The moment an item meets the DoD, an increment is born. An item that does not meet it cannot be released. It cannot even be shown in the sprint review; it goes back to the product backlog. This is a much stronger sentence than a checklist.

This is how I put the difference. A checklist is for the memory of the person doing the work, so they do not forget something. A contract is written for the people who rely on the work. The moment we say “done”, other people start to rely on things:

• The PO plans the next sprint assuming the work runs.
• The call centre answers customers based on that work.
• Other teams build their own work on top of it.
• The velocity chart counts it as done, and future estimates depend on it.

On 26 November, all four had relied on something wrong. The call centre did not even know about the new rejection message, so they did not know what to tell customers. The list was not wrong. It just did not say what anyone was relying on.

What it should look like: three layers

In the Sprint 13 retro on 5 December, we split the DoD into three layers. Each layer means a different “done”, and we check it at a different time:

We also added a column to the board: “Live, being watched”. A card only reaches Done through that column. With this setup, the IBAN story would never have reached Done. On the first day, the rejection rate alert would have fired.

STORY
  - Code reviewed (at least one approval)
  - Unit and integration tests green in CI
  - At least one example with production-like data
    (customer with two given names, Turkish letters,
     upper case, spaces)
  - Log line and metric for the new behaviour

SPRINT
  - Regression tests green
  - Release notes written
  - If customers can see the change, call centre
    informed (what changed, what to tell customers)

PRODUCTION
  - Checked in production with at least one real transaction
  - Visible on the dashboard, alert fires above a threshold
  - Switch-off path exists and was tested (flag or rollback)
  - Watched for 24 hours, then "done"

Removed: "code coverage above 80%" (the IBAN story had 91%)

Who checks each layer?

Writing the layers was easy. Deciding who checks each one took a week. In the end, we set it up like this:

• Story layer: the person who did the work and the person who reviewed the code. The reviewer asks about the code, and also about the example with production-like data.
• Sprint layer: one developer, a different one each sprint, one day before the review. This person writes the release notes and the two-line note for the call centre.
• Production layer: the person who shipped the release. After 24 hours, they look at the dashboard. If the rejection rate or error rate is below the threshold, they move the card to Done themselves.

We added one more rule. The person who moves the card to Done writes the layer’s result on the card in one line: “Tried with 3 real requests in production, rejection rate 0.8%, alert threshold 3%.” A sentence, not a ticked box. Six months later, anyone who opens that card can read what people relied on.

A feature flag is a setting that turns a feature on or off without a new release. I will not go into testing separately here. In the Epic, Story, Task post, I suggested putting tests inside the DoD instead of on a separate card. That rule still stands. The question here is not where testing sits. It is where “done” ends.

Why did we remove the coverage item? Because it told us nothing. A story with 91% coverage rejected one in seven withdrawal requests. The item gave a feeling of safety, not something you could rely on. Only items that someone really relies on go into the contract.

Who changes the DoD

The Guide is clear. If the organisation has a standard, the team follows it as a minimum. If not, the Scrum team creates the DoD. We have no organisation standard, so we set our own rules:

• A change is proposed in the retro. The action rule from the Retrospective post applies here too: definition, owner, deadline.
• The team decides, and the PO listens. The PO can ask for an item to be added. The PO cannot relax an item to meet a date.
• It does not change in the middle of a sprint. It applies from the next sprint. A DoD that changes mid-sprint also quietly changes that sprint’s commitment.
• Every addition has a cost, and we write the cost down. The new items added about half a person-day per story. In Sprint 14, we planned seven items instead of nine.

The last point got the most push-back. “So we will do less work?” No, we will count less work as done. The difference was an item that said “done” in Sprint 12 but needed a day and a half more work in production.

When it breaks: keeping the work open

On 26 November, our first reflex was to open a new bug ticket. If we had done that, the story would have stayed “done”. The Sprint 12 count would have stayed at seven. The bug would have taken capacity from Sprint 13 as “other work”. On paper, nothing would have been broken.

Instead, we set this rule:

1. Within 7 days of going live, if a bug breaks the acceptance criteria, we reopen the same story. After 7 days, it is a normal bug ticket.
2. The reopened story is removed from the done count of its sprint. The table in the Definition of Ready post says “7 / 8” for Sprint 12. Under today’s rule, that number is 6 / 8. I am not editing that post; this post is the correction.
3. Relaxing the DoD is recorded. If work has to go live without meeting one DoD item, it can. But it does not count as done. Three things go on the card: which item is missing, who made the decision, and when the missing part will be finished. The card stays open until then.

The rule had a side effect I did not expect. In the days after a release, the team started to look at the dashboard on their own, because work that came back from production now came off their own sprint count. Nobody asked them to. The number did.

How it breaks

What to watch

Sprint 14 is one sprint. Zero is not a result; it is a start. The production layer also has an uncomfortable cost. We ship a weekly release, so work that misses the Tuesday 16 December release cannot be “done” inside the sprint. The seventh item was finished as code, went live on 23 December, and will count in Sprint 15. We accepted this on purpose.

The row I really watch is the last one: who sees the next bug first. If the alert sees it, the production layer of the DoD is working. If a customer sees it, it is not.

Checklist

Conclusion

The message from the call centre on the morning of 26 November was not a test failure. It was a broken contract. Six boxes were ticked, and all six were true. But none of them said what the people relying on the work were relying on.

Now a card waits 24 hours before it moves to Done. This is not slowness. It is the price of never again learning the meaning of “done” from the call centre.