Shipping: Release Management, Not Deploys

From the field: 9 minutes versus 4 seconds

What we did that Friday was technically correct: we spotted the error, made a decision, rolled the release back, and the system recovered. Nobody panicked. We still lost 9 minutes, and all 9 of them were avoidable.

Here is why. The only way to get the change in front of users was a deploy, so the only way back was also a deploy. A deploy is a build operation — an image is pulled, pods are replaced, health checks are awaited. You can make it faster, but you cannot make it take seconds. Meanwhile “do not show this feature to users” is a read operation. It is one value being read, and it takes seconds.

The following week we shipped the same flow again, this time behind a flag. It stayed closed for three days, then we opened it to 5% of users. On the second morning the same class of error appeared again — and this time switching it off was a field change affecting around 40 users. The same bug, two very different costs.

Deploy is not release

My favourite side effect of this split: the argument about “this will not be ready, let us move the release” disappears. The code ships, sits closed, and opens when it is ready. You no longer have to park unfinished work on a branch — and a parked branch is the one that conflicts most when it finally merges.

A note: why shipping often matters so much, and how a shared deploy train locks teams together, is in the monolith to microservices post. I am not repeating it here.

Four types of flag, four lifetimes

“Feature flag” is not one thing. Keeping them all in one list makes the delete rule impossible, because the moment someone says “but some of them are permanent”, all cleanup stops. Four types, and how long each should live:

1) RELEASE FLAG   : open a new feature gradually
   life: days to weeks    --> DELETED once fully open

2) KILL SWITCH    : emergency off switch for a feature
   life: permanent        --> never deleted, but TESTED ONCE A YEAR

3) EXPERIMENT     : compare two variants (A/B)
   life: the experiment   --> DELETED when there is a result

4) PERMISSION     : "this customer may use this feature"
   life: permanent        --> not really a flag; it is authorisation
                             and does not belong in the flag system

The fourth type caused most of our confusion. “Enterprise customers can see this screen” is not a release decision, it is an authorisation rule. As long as it lives in the flag system it can never be deleted, and it inflates the count. When we moved them out, 14 of our 61 flags left the list in one go.

The flag graveyard and the delete rule

Creating a flag costs one person five minutes. Deleting one is nobody’s job. The result is predictable: we had 61 active flags and 23 of them were over a year old. Two contradicted each other — one opened the new flow, the other forced the old one, and you could not tell which users saw which without reading the code.

We tried a cleanup campaign. It did not work. What worked was making the decision when the flag was created:

Breaking the build sounds harsh, but it is the part that matters. During the two months when CI only warned, not a single flag was deleted; the warnings sank to the bottom of the list. In the week it started breaking the build, 18 flags were cleaned up.

Gradual rollout: write the threshold first

Doing a gradual rollout as “let us open it to a few users and see” is almost the same as not doing one. “See” does not say who is watching, for how long, or what they are watching. So we wrote the steps down:

STEP  TRAFFIC   MIN TIME    AUTOMATIC ROLLBACK THRESHOLD
-----------------------------------------------------------
  1      1%     15 minutes   error rate > 1%  OR  p95 > 800ms
  2      5%     30 minutes   error rate > 1%  OR  p95 > 800ms
  3     25%      2 hours     same
  4    100%        -         same  (watched for 24 hours)

RULE: if the threshold is crossed, it rolls back automatically.
      Nobody approves. A threshold that asks a human at 03:00
      is a threshold that does not work.
RULE: no step is skipped. "It is urgent" does not skip a step;
      it can shorten one, and that is requested in writing.

Two details matter. First, the minimum time: to move to the next percentage, “nothing looks wrong” is not enough, you have to wait. Most problems appear with time rather than with traffic — a memory leak, a queue filling up, a table growing. Second, automation: if you leave the threshold to a person, it only works during the hours when someone is awake.

The on-call engineer being able to turn a flag off alone at night is part of the same design. That is one of the things I meant in the on-call post by “anything reversible is allowed at night”.

What to track

What did not work for me

• Spreading flag checks through the code with if statements. It felt practical at first. Six months later the same flag was read in seven places and one of them used the old default. Flag reads should go through one place: not the branching, but the decision itself should be central.
• “We do gradual rollouts for important changes.” Who decides what is important? Nobody. In the end the change that looked most harmless — a single configuration line — caused the biggest outage. The rule is now reversed: everything user-visible goes out gradually, and exceptions are requested in writing.
• Thinking blue-green solved it. Switching traffic instantly is nice, but it is all or nothing. If you cannot ship to a 1% slice, you discover your bugs on all users at once. The two are not alternatives: blue-green is infrastructure, gradual rollout is the decision layer on top.

Checklist

Conclusion

The problem that Friday was not that the new payment flow had a bug. Every flow has a bug one day. The problem was that the only way to undo it was to wait for a server. We had tied shipping the code and showing it to users to the same moment, so undoing a product decision required running a build operation.

Release management does not exist to stop you making mistakes. It exists so you can choose what a mistake costs. The same bug can reach 1,100 users or 40. The difference is not code quality. It is how you ship.

Measure it with one question: how many seconds would it take to undo the last change you shipped? If the answer is in minutes, you do not have release management. You have deploys.