Monolith to Microservices: What Five Months Taught Us

First, stop: do you actually need microservices?

Nobody asked us this question, and we did not ask ourselves. Microservices were "the modern thing" at the time. If asked today, my answer starts with this table:

The rule: microservices are not a performance tool, they are an independence tool. Without teams that need to ship independently, what you get is not speed but a monolith distributed over a network. In our case the last three rows genuinely applied; the decision was right. The way we did it was not.

Why we started: a deploy-train story

An eight-year-old monolith, four teams, one deploy a week. The numbers were bad, but the numbers were not the real problem:

• Deploy 45 minutes, rollback 52 minutes. Hence once a week; nobody dared ship more often.
• An average of 60–80 changes per weekly package. When something broke, finding which one took hours.
• The payment team's urgent fix waited on the reporting team's half-finished work. Or the other way round.
• One test environment. While one team tested, the others waited.

So the problem was not "the system is slow"; it was "four teams are handcuffed to each other".

Our second day-one decision came from the same place: services would not call each other; they would publish events. We chose Kafka as the broker. A service that calls another over HTTP depends on the other being up at that moment; a service that listens to events does not. If teams were going to be independent at deploy time, they had to be independent at runtime too. Right decision. We did not know the price.

Step by step: from wrong to right

Attempt 1 — split by layer (broken)

web-api  --(waits)-->  business-rules-service  --(waits)-->  data-access-service  -->  DB

# "They will talk through events", we had said.
# Layers cannot talk through events: the rule needs the data layer's ANSWER.

Attempt 2 — split by domain, kept the database (distributed monolith)

order-service     --> publish(OrderChanged { id: 4711 })
shipping-service  --> listen, then:  SELECT * FROM orders WHERE id = 4711   <- shared table
report-service    --> listen, then:  SELECT * FROM orders WHERE id = 4711   <- shared table

Nobody had made a mistake. The customer team changed their own table. But the table was not theirs — it was the shared contract of four services, and a system that looked event-driven hid that. The sentence we learned: the service boundary is drawn in the data, not the code. An event must carry the data the consumer needs; an event that says "go and look" is not an event, it is an invitation to the table.

Attempt 3 — started from the core (stalled)

Attempt 4 — from the edge, gradually (right)

# inside the monolith, notification code:
if hash(orderId) % 100 < NOTIFICATION_SERVICE_PERCENT:   # 5 -> 25 -> 100
    pass                       # the new service listens to OrderCreated and sends it
else:
    sendNotificationLegacy()   # monolith code, still in place

The order was this and would be the same again: from least dependent to most dependent. The core goes last, because the core can only move once its surroundings are empty.

First cost: the same message arrives twice

Early in month two, a Tuesday morning message from customer service: "Two shipments went out for the same order; the customer received two boxes." Not one order; 17 double shipments in a week. On the campaign side something quieter: more than 300 customers had loyalty points credited twice, and nobody noticed — nobody complains about that.

The cause was not a bug, it was the contract itself. The shipping service received the event, created the label, wrote it to the database, and got deployed before telling the broker "processed". The broker got no answer and delivered the same event again; the new replica processed it as if seeing it for the first time. The broker we used had done exactly what it promised: at-least-once delivery. The monolith never had this problem, because a function call is not "redelivered".

onOrderCreated(event):
    label = shipping.create(event.orderId)
    db.save(label)
    ack()   # <- die before reaching this line and
            #    the event comes back,
            #    the label is created a second time

onOrderCreated(event):
    BEGIN
      INSERT INTO processed_events (event_id)
        VALUES (event.id)   -- UNIQUE:
                            -- the second arrival fails here
      label = shipping.create(event.orderId,
                              header "Idempotency-Key: " + event.id)   -- external system: same key,
                                                                     -- same label, no new label
      db.save(label)
    COMMIT
    ack()

    # unique violation -> already processed, ack quietly, move on

Three things matter. The event's identity is set by the producer, never generated by the consumer. The "seen" record lives in the same transaction as the database side of the work; in a separate transaction, a crash in between still produces two labels. And the third, which we noticed last: a call to an external system sits inside the transaction but is not part of it.

After the first dedup, double shipments fell from 17 a week to 2; not to zero. The remaining two looked like this: the INSERT went through, the carrier's API cut the label, and the process died before COMMIT. The database rolled back — the dedup record with it — but the carrier did not. The event came again, the API was called again, second label. The fix is for the external system to be idempotent too: we pass the event ID to the carrier's API as an idempotency key; a second request with the same key returns the same label instead of cutting a new one. If the external system does not support that, the work has to be split in two: commit a "request being sent" record first, then call, then write the result — that is the only way a retry is safe. The details are in message delivery guarantees; the lesson here is about order: this should have been built with the first event consumer, not after seventeen double shipments.

Duplicates do not come only from the consumer side. If the process dies between writing the order and publishing the event, the event never leaves; we solved that with an outbox — and the outbox relay retries too, so it can publish the same event twice. Duplicates can come from both ends, and the only defence is in the consumer. Simple rule, one table per service: every service that consumes events must produce the same result when the same event arrives twice. Not a preference; the entry fee for going event-driven.

Second cost: ordering

Two weeks after dedup, a new complaint: "The order I cancelled arrived at my door." The log showed this:

The cause: we were not setting a partition key when publishing; the broker spread events as it liked. Two events for the same order landed on two partitions, and two consumer replicas processed them at their own pace. Kafka guarantees order only within a partition; we knew that, but "it won't happen to us". The 170 milliseconds between the two events were enough.

publish(topic="orders", key=orderId, event)   # every event for one order on the same partition

One case remained that ordering alone cannot fix: if a cancellation genuinely arrives first, instead of dropping it as "unknown order", record it and wait. When OrderCreated arrives, check for a pending cancellation first. Ordering is guaranteed now, but that guard stays; a check is cheap, a double shipment is not.

Third cost: "I placed an order and it's not in my list"

This one generated the most support tickets. The order is written, the event is published, and the "My orders" page is served from a separate read model that processes the event 1–3 seconds later. The customer taps "Place order", goes straight to the list: empty. Orders again. 12 duplicate orders a week, all through this path.

The second member of the same family was worse. Having said "everything is an event", we waited for the payment result as an event too: the order screen showed a spinner until PaymentCompleted arrived and asked the server every second. Payment confirmation reached the screen in 6 seconds at p95; the monolith did it in 80 milliseconds. In that window customers closed the tab, the money was taken, and the order sat in "pending".

POST /orders
  -> write order
  -> publish(OrderCreated)
  -> 202 Accepted

# screen: until PaymentCompleted arrives,
# GET /orders/4711/status every second

POST /orders
  -> stock.check()     # HTTP, 300 ms timeout
  -> payment.take()    # HTTP, circuit breaker
  -> write order + outbox(OrderCreated)
  -> 201 { orderId, status: "accepted" }

# shipping, notifications, points, reports, read model:
# listen to the event, at their own pace

The measure became: a step that cannot continue without its answer is synchronous; everything else is an event. Stock and payment pass that test; the other seven things do not. The two synchronous calls got timeouts and a circuit breaker; if the payment service slows down, the order screen says "we can't take this right now" in 300 ms instead of spinning for six seconds.

For "not in my list", two small things: after ordering, the screen goes to the order detail rather than the list, and the detail is read directly from the order service, not the read model (read your own writes). The list still lags 1–3 seconds; nobody places a second order in that window any more.

Fourth cost: one transaction spread over four services

In the monolith, placing an order was a single database transaction: reduce stock, write the order, use the coupon, record the payment — all or nothing. Spread over events, that guarantee quietly vanished; on the happy path everything worked, so nobody noticed.

For the first three weeks we fixed these by hand — 20 to 30 records a week. Then we moved to compensating events: when PaymentFailed is published, the stock service handles RestoreStock and the campaign service releases the coupon. Every forward step has a backward step, and every service listens for its own compensation. That is the subject of the saga post; what matters here is that nobody had priced this before the move. A guarantee the monolith gave for free is a feature you have to design in an event-driven system.

Invisibility: debugging from 10 minutes to 3 hours

An HTTP chain at least gives you a stack trace. With events there is nothing: a shipping label came out wrong — which event produced it, and which event triggered that one? By month three the same job took three hours on average; we were looking at the same second in six services' logs and matching the customer's order by hand.

Correlation IDs and distributed tracing are covered in that post; an event-driven system adds one step. One ID is not enough; you need three, and all three travel in the event header:

event_id        this event's own identity          (dedup uses this)
correlation_id  the workflow's identity; born with the first request, NEVER changes down the chain
causation_id    the event_id of the event that triggered this one

OrderCreated        event_id=E1   correlation_id=C1   causation_id=-
PaymentCompleted    event_id=E2   correlation_id=C1   causation_id=E1
ShippingLabelCut    event_id=E3   correlation_id=C1   causation_id=E2

A consumer copies the correlation ID unchanged into the events it produces and writes the ID of the event it consumed into causation. Correlation answers "which events did this customer's request spawn" in a single query; causation lets you walk the chain backwards to find which event produced the wrong label. We added it in month three; counting the bugs we "assumed did not exist because we could not see them" during those three months, the bill for that delay was among the highest.

The cost nobody budgeted: operations

A monolith has one pipeline, one dashboard, one on-call book. Fourteen services have fourteen — plus the queues:

The queue brought its own diseases. One malformed event (an old schema from a stale producer) locked the shipping consumer for 40 minutes: it retried the same message over and over while 2,000 events waited behind it. The dead-letter queue was set up that day — all of this is in queue pathologies; the note here is that every one of them happened to us, one by one, in the first three months.

Pages nearly tripled in the first three months; not because the system was more fragile, but because it had more parts. We only pulled that back with a platform team: a shared pipeline template, a shared logging/monitoring package, a one-command service skeleton — with the dedup table, the outbox and the lag alarm already inside. That team was not in the plan; it was created out of necessity in month four. Today it would be created in week one.

And the human side: fourteen services, four teams. Each team owns three or four services. That is Conway reminding you of himself — if you do not draw service boundaries along team boundaries, team boundaries start bending themselves to the services.

What we merged back

This is a little hard to write, but it is the most useful lesson. In month five we merged two services back:

• Basket and pricing. Every basket change published BasketChanged, pricing answered with PriceCalculated, the basket waited for it — an event ping-pong between two services, and the user saw the new price 1-2 seconds later. They had never changed independently: 40 deploys in four months, 38 on the same day. They became one service; the price is instant, nothing was lost.
• Shipping and delivery tracking. Same team, same data, two services that changed on the same day. The reason for splitting was "there will be separate teams eventually". There were not.

The rule crystallised afterwards: if two services behave like one service in the deploy history, they should probably be one service. It is easy to measure — look at the deploy history. If the freedom to ship separately is never used, the price of separation is paid for nothing.

From the field: the numbers after five months

The middle column is the one most write-ups leave out. In month two everything was worse: slower, more pages, longer debugging, inconsistent data, double shipments. Some people said "let's go back" and they were not wrong. A migration abandoned halfway gives you the worst of both worlds; carried through, it solves the first column's real problem — teams handcuffed to each other. The only genuine gains in the last column are rows two and four; the rest is recovering what we lost in month two.

If I started again today

Checklist

Conclusion

After five months we saw that we had made the right decision in the wrong order. The decision had two parts and both were right: four teams handcuffed to one deploy train was unsustainable, and services should talk through events rather than call each other. The order was wrong: layers, then a shared database, then the core — three attempts, roughly ten weeks. And we learned the three costs of events — exactly-once execution, ordering, immediate consistency — not by designing for them but through double shipments, a cancelled order arriving at the door, and an empty order list.

The sequence to remember: draw the boundary inside the monolith first, then split the data, then start at the edge and add the network; dedup, partition key and IDs arrive with the first event consumer; decide up front which steps will not be events. Microservices are not pieces of code talking through a queue; they are teams that can ship without waiting for each other. If the teams are still waiting, neither the service count nor the queue matters.