Rewriting: A Migration Plan, Not a Clean Slate

From the field: a rewrite that lasted four hours

The commission engine is the part that calculates the fee we charge a customer for every executed order. It was nine years old: 4,200 lines inside stored procedures in the database. Every change to the fee rates took three days, because nobody wanted to touch that code. In January we decided to rewrite it. I still think that decision was right.

The plan was wrong. The plan was: write the new engine on a clean slate, build test cases from the fee rate document, and when the tests pass, flip the switch one morning. Five months later, on Monday 16 June, we flipped it. Every test was green.

The market opened at 10:00. By 11:10, 18,400 orders had been executed, and on 570 of them the commission was different from before (3.1%). The total amount was small: 1,940 TL. But it was 570 customers, and each one needed a separate refund and a separate explanation. The operations team needed two days to finish that.

The differences had three sources, and none of them was in the fee rate document. Accounts opened before 2019 were on an old rate. For partly executed orders, the old engine rounded each part separately. A group of corporate customers had a daily cap in their contracts. Nobody had hidden these rules. They just were not written down anywhere, except in the code.

My mistake: I treated the fee rate document as the specification. But the real specification was the numbers customers had paid for nine years, and the only thing that produced those numbers was the old code. A clean slate meant throwing all that knowledge away on purpose.

Why a rewrite is tempting, and why it is dangerous

A rewrite is tempting because the cost of the old system is visible every day: slow changes, deploys that people fear, one person who knows the code. The cost of the new system is not visible yet. On paper it is always clean.

The dangerous part: during a rewrite you own two systems. The old one keeps running, fee changes keep arriving, and you have to write every one of them in two places. The ownership question from the build or buy post doubles here. The longer the transition, the longer your team carries two systems at once.

So the real question is not “should we rewrite?” It is: in which steps do we move from the old system to the new one, and how do we go back at each step? If you cannot answer that, you do not have a rewrite plan. You only have a plan to write a new system.

The migration plan: five stages

On the second attempt we did not flip a switch. We put the new engine next to the old one and moved traffic stage by stage. Each stage had a way back and a condition for moving to the next one:

This is usually called the strangler approach: the new system does not replace the old one at once. It wraps around it piece by piece and takes its place. In our case the pieces were not code modules but customer segments. The engine could not be split, but the customers could.

My favourite stage is the second one. The first real customers are us: the personal investment accounts of company staff. If someone gets charged the wrong commission, the complaint comes from down the corridor, not from 570 customers.

We also tried the way back once at every stage. Which account got its answer from which engine was kept in a single configuration table. Going back meant changing one row in that table. In June, 45 minutes passed between the first message and the rollback, because bringing the old engine back needed a deploy. On the second attempt, before entering each stage, we ran the rollback in production with the staff accounts, not in a test environment: 40 seconds. A way back you have never tried is a way back on paper. The middle of an incident is the wrong moment to find out whether it works.

Shadow: two systems, one answer

The idea of the shadow stage is simple. Every order goes to both the old and the new engine. The old one answers the customer, and the two results are compared. Simple, but there are three details:

def calculate_commission(order):
    old = old_engine.calculate(order)            # this one answers

    try:
        # the new engine ONLY calculates: no writes, no queue, no log table
        new = new_engine.calculate(order, side_effect_free=True)
        compare(order, old, new)
    except Exception as err:
        record_diff(order, kind="NEW_ERROR", detail=err)   # customer not affected

    return old

def compare(order, old, new):
    # normalise: timestamps, field order, digits below one kurus
    o, n = normalise(old), normalise(new)
    if o.amount != n.amount:
        record_diff(order, kind="AMOUNT", old=o.amount, new=n.amount,
                    hints=[order.account_type, order.is_partial, order.rate_code])

No side effects. While in shadow, the new engine must not write anywhere. We learned this the hard way. On the first day, the new engine’s audit log also wrote to the table the old engine used, and that day’s commission report came out doubled. It took us a day to notice.

Normalise. On the first day the difference rate was 40%. Most of it was noise: milliseconds in timestamps, the order of fields, digits below the smallest coin. If you compare without normalising, the real differences get lost in the noise.

Record hints. “There is a difference” is not enough. We added fields like account type, partial execution and rate code to every difference record. When we grouped differences by these fields, the categories appeared on their own.

Every difference is a decision

During six weeks of shadow we compared 2.1 million orders and found 14 categories of difference. The difference rate was 3.1% in the first week (the same as in June, because the code was the same) and fell to 0.004% by the sixth week. The real work was putting each category into one of three boxes:

The middle row taught us the most. The old engine had two bugs, and our first reflex was “let us do it properly in the new one”. We did not. Customers had seen that number for years. Changing it quietly on the day of the switch would invite the question “why did my commission change?” on the support line. First the new engine copied the old one exactly, bugs included. We fixed the bugs after the switch, as a separate change, with an announcement.

The rule: do not migrate and fix at the same time. If you do both at once, then when a difference appears you cannot tell which one caused it.

How it breaks

The last item is the sneakiest. The switch ends, the new system runs, and the old one stays up “just in case”. A year later you find two reports, a night job and an integration nobody knew about, all depending on it. You keep maintaining two systems, only now without knowing it.

Shutdown criteria: written at the start

We wrote down the shutdown date for the old engine before the shadow stage began, on the first page of the migration document. Even if the date moved, the criteria did not:

The call counter found something unexpected: the accounting team’s month-end reconciliation script was calling the old engine directly. It was not on any architecture diagram. Without the counter we would have found out at the first month-end after the shutdown, on 31 October.

What to track

What did not work for me

• Building test cases from the document. We wrote 300 cases and all of them passed. They tested the rules we knew. The only thing that found the rules we did not know was real traffic.
• Running the shadow on a sample. To save cost, in the first week we compared only 5% of orders. Rare rules like the corporate cap never fell into that 5%. When we moved to 100%, two more categories appeared.
• Freezing the old system during the shadow period. We said “no fee changes for six weeks”. In the third week a new campaign rate arrived. Instead of a freeze we made a rule: every change to the old system goes into the new one in the same PR.

Checklist

Conclusion

The plan on 16 June would have taken five months. With the migration plan the total came to nine months: the old engine was switched off on 10 October. In return, on the second attempt the number of customers charged the wrong commission was zero.

The clean slate idea assumes the old system is only a burden. But the old system is also a memory. It holds every exception nobody wrote down, and every number customers are used to. Rewriting does not mean throwing that memory away. It means moving it, line by line, into the new system.

The test: if you switched to the new system tomorrow morning and something went wrong, could you go back to the old one in five minutes? If you could not, you do not have a migration plan. You only have a hope.