How Many Times Does a Message Arrive?

Why you have to choose a guarantee

Start with a simple situation: you sent a message and no reply came. What happened?

• The message never arrived.
• The message arrived, was processed, and the acknowledgement was lost on the way back.

On the sending side these two are indistinguishable. Both look the same: silence. And you have exactly two decisions available:

• Send again → the message is never lost but may be processed twice. That is at-least-once.
• Do not send → it is never processed twice but may be lost. That is at-most-once.

There is no third option. The whole subject comes out of that one ambiguity.

At-most-once: the lost tick problem

Price feeds usually arrive over UDP, and UDP promises nothing: if a packet drops it is gone and nobody resends it. The first time you hear this it sounds wrong — why not use TCP?

Because a repeated tick is useless. Getting the EURUSD price from 50 ms ago has no value; a newer one already arrived. The time TCP would spend retransmitting does not recover data, it delays the whole stream. Here loss is acceptable; latency is not.

class Quote {
    String  symbol;
    double  bid, ask;      // fine for prices; NEVER double for fills/balances, BigDecimal/numeric
    long    sourceTime;    // provider's stamp
    long    arrivalTime;   // when we received it
    long    sequence;      // increasing number from the provider
}

// On every tick: FILTER first, THEN look for a gap (sequence is the reliable key, not time)
if (incoming.sequence <= last.sequence) return;     // duplicate or late old tick: drop
if (incoming.sequence > last.sequence + 1) {        // only a FORWARD gap (no negative counts)
    ticksLostTotal.inc(incoming.sequence - last.sequence - 1);
    if (deltaFeed) suspect.put(symbol, true);       // delta: until a snapshot arrives
}
if (!deltaFeed) suspect.put(symbol, false);         // full quote: the new tick is the current price, doubt ends
last = incoming;
lastAlive.put(symbol, now());

// On a heartbeat: the price did not change but the provider is alive.
// Heartbeats carry a sequence too (in most protocols "next expected number"): if there is a gap,
// liveness is NOT updated, otherwise while ticks drop on UDP the heartbeats keep an old price looking fresh
if (hb.nextExpected > last.sequence + 1) {          // only a forward gap
    ticksLostTotal.inc(hb.nextExpected - last.sequence - 1);
    last.sequence = hb.nextExpected - 1;            // do not count the same gap again
    suspect.put(symbol, true);                      // delta: until a snapshot, full quote: until the next tick
} else {
    lastAlive.put(symbol, now());                   // no gap: count as alive
}

// On a snapshot: state rebuilt, sequence continues from here
// (on reconnect / at start of day the provider resets the sequence; if you do not set last.sequence here,
//  the "incoming.sequence <= last.sequence" line above drops every tick)
suspect.put(symbol, false);
last.sequence = snapshot.sequence;

// BEFORE opening an order, three checks:
long silence = now() - lastAlive.get(symbol);                  // tick OR gap-free heartbeat
if (silence > STALE_THRESHOLD_MS)     reject("stale quote: " + silence + " ms");
long sourceAge = now() - quote.sourceTime - clockOffset;       // two different clocks: subtract the measured offset
if (sourceAge > SOURCE_THRESHOLD_MS)  reject("stale at source"); // separate, tolerant threshold (e.g. 2 s)
if (suspect.get(symbol))              reject("suspect: waiting for snapshot / next tick");

At-least-once: the fill that arrived twice

Now the real subject. Orders, fills, balance movements — losing these is not acceptable, so you use at-least-once. Which means they will arrive twice.

How it happens, one by one:

All four are normal operation. None of them is a bug. So chasing "make sure it does not arrive twice" is the wrong path; the right one is answering "what happens if it arrives twice?"

Why exactly-once is a myth (and what actually exists)

When you hear "we support exactly-once", the question to ask is: in delivery, or in processing?

• Exactly-once delivery: not possible. The ambiguity above — was it the ack or the message that was lost — does not go away.
• Exactly-once processing: possible. And the method is known: at-least-once delivery + a consumer that discards duplicates.

Kafka's exactly-once falls into the second category, with one condition: both the read and the write must be inside Kafka. If you read from Kafka and write to PostgreSQL, that guarantee does not cover you: the DB write and the offset commit live in two different systems and you can crash between them. Two known ways out: store the offset in the same DB transaction, or the idempotent consumer below. (Its cousin, the "write to DB + publish an event" problem, is a separate post: dual write.)

Cure 1: an idempotency key (request path)

The side sending the order generates a unique key per logical order and sends the same key on retries.

-- Key table: uniqueness TOGETHER WITH the account
CREATE TABLE request_keys (
  account_id   bigint NOT NULL,
  key          text   NOT NULL,       -- client_order_id
  request_hash bytea  NOT NULL,       -- hash of symbol+side+lots+price
  result       jsonb,                 -- a copy of the first response
  created      timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (account_id, key)
);

BEGIN;
  INSERT INTO request_keys (account_id, key, request_hash)
  VALUES (:account, :client_order_id, :hash)
  ON CONFLICT (account_id, key) DO NOTHING
  RETURNING key;
  -- If no row came back: this key was seen before.
  --   same request_hash      -> DO NOT act, return the stored result.
  --   different request_hash -> 422 "same key, different request" (never silently return the old result)

  -- If a row came back: first time, place the order
  INSERT INTO orders (...) VALUES (...);
  UPDATE request_keys SET result = :response
   WHERE account_id = :account AND key = :client_order_id;
COMMIT;

Four subtleties, all learned the hard way:

• The client generates the key, not you. Generate it server-side and every retry produces a new key, so the protection never fires. And the client must generate it per logical order — once, when the form opens — not per click; otherwise a double click arrives with two different keys.
• The key is unique together with the account, not on its own. FIX's ClOrdID is only unique within a session/firm too. With the key as a global PK, when two accounts generate the same client_order_id the second is not just wrongly rejected, it receives the first one's stored response: another account's order details. That is a security hole.
• Same key, different content → error. If the client sends the same key with different lots or a different symbol, silently returning the old response is wrong; store a hash of the request and return an explicit error on mismatch (Stripe and friends do this). In an order system "1 lot or 10 lots" cannot stay silent.
• Store the first response and return it again. If you return an "already exists" error, the client reads it as a failure and retries again. The second request must receive the first request's answer.

Cure 2: an idempotent consumer (event path)

For a fill notification arriving on a queue there is no client; the key travels inside the event (fill_id, deal_id, event_id). The consumer checks whether it has seen that id.

CREATE TABLE processed_events (
  kind      text NOT NULL,            -- 'FILL', 'CANCEL', ...
  event_id  text NOT NULL,            -- execution id (deal_id / exec_id)
  created   timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (kind, event_id)        -- ids of different kinds may collide
);

BEGIN;
  INSERT INTO processed_events (kind, event_id)
  VALUES ('FILL', :fill_id)
  ON CONFLICT (kind, event_id) DO NOTHING
  RETURNING event_id;
  -- No row? This fill was already processed -> COMMIT and exit

  -- First time: do the real work
  INSERT INTO fill_records (...) VALUES (...);
  UPDATE positions
     SET lots = lots + :lots, avg_price = ...
   WHERE id = :position_id;
COMMIT;

The dedup table must not grow forever

With millions of events a day, that table becomes your main load a year later. Practical approach:

• Pick a window longer than the longest possible replay. Most duplicates arrive within minutes, but not all: reprocessing from the DLQ days later, a consumer that stayed down for a long time, replaying a topic from the start. Kafka's default retention is 7 days; the window must be at least retention + DLQ dwell time. Ours is 30 days.
• Clean up the old, but not with one DELETE. An unindexed DELETE ... WHERE created < ... does not lock the table; it does a seq scan, a long transaction, heavy I/O and bloat. Delete in chunks (PostgreSQL has no DELETE ... LIMIT, you need a subquery):

  DELETE FROM processed_events
   WHERE (kind, event_id) IN (SELECT kind, event_id FROM processed_events
                               WHERE created < now() - interval '30 days' LIMIT 10000);
  -- loop until 0 rows, short pause in between
  -- an index on created is REQUIRED: without it every round's subquery scans the whole table,
  -- at millions of rows more expensive than one big DELETE

• If you partition, do not break the PK. Dropping an old partition is the cheapest cleanup, but in PostgreSQL the PRIMARY KEY of a partitioned table must include the partition key. Partition by created and the PK becomes (kind, event_id, created); since created DEFAULT now(), a duplicate arrives with a different timestamp and the constraint never catches it — dedup dies silently. The right way is a column the event itself carries and that stays the same on a retry (event_time, the execution time): partition on it and make the PK (kind, event_id, event_time). If you cannot, stay unpartitioned and delete in chunks as above.
• Think about outside the window. A duplicate arriving after the window will be reprocessed. If that is unacceptable, add a unique constraint on the real table too — a second line of defence.

So which guarantee for which data?

The easiest way to decide is one question: is losing this message worse, or processing it twice?

Note: both exist in the same system, and they should. Making the price feed at-least-once kills latency; making orders at-most-once kills money. Standardising on one guarantee means being wrong on one of the two sides.

From the field: how we closed the double fill

Back to the opening incident. The same fill arrived twice and two records were created; there was no dedup at all that day. The first instinct was "fix the bridge so it stops sending twice". Wrong instinct: the bridge was behaving correctly, resending because it got no ack.

What we actually did (the second one is a separate incident, a few weeks later):

1. A unique constraint on deal_id. Not a check in the application, a prohibition in the database. However many copies run, the rule lives in one place.
2. Dedup and the position update moved into the same transaction. The first version wrote the dedup row and then updated the position; a crash in between produced "processed but not written" — this time loss, not a duplicate, and much harder to spot: reconciliation caught it.
3. The duplicate counter became a metric. duplicate_events_total is not zero and we do not expect it to be. But when the number jumps, it is usually the first sign of a problem on the bridge or the network. So dedup is not only a guard, it is also a sensor.

Checklist

Conclusion

Choosing a delivery guarantee looks like a technical preference but it is really a commercial one: what are you willing to lose? You accept loss on prices because a new one is coming. You do not accept it on fills because that is money.

And one sentence to keep: exactly-once is not a delivery guarantee, it is a consumer design. Nobody can sell it to you; you write it yourself. The core is one ON CONFLICT DO NOTHING; everything around it — one transaction, a per-account key, the stored response, cleanup — is the real work.