Leader Election: Who Does This Job?

Where the problem comes from

Scaling a service horizontally usually feels free: two copies do twice the work. But not all work divides. Some jobs must happen exactly once, and they break quietly as the copy count grows.

Examples that will feel familiar from the trading side:

The right-hand column looks scarier at first, but the left one is more insidious: it produces no error, it just costs money. Health checks going out from four copies is invisible on day one; it becomes visible when the bridge provider starts rate limiting you.

Three easy but wrong answers

1. "Let's just run one copy"

It works — until that copy dies. If the pod crashes, Kubernetes starts a new one and 20–60 seconds pass. If the node crashes, it is worse: with the default eviction thresholds, moving the pod takes about 5 minutes. During that window nobody health-checks anything. Acceptable? For some jobs, yes. For bridge health checks, no.

2. "Let's flag it with an environment variable"

if os.getenv("IS_LEADER") == "true":
    start_health_check_loop()

3. "First come first served, put a flag in Redis"

Right direction, half the answer. If you set a flag with SETNX leader 1 and no expiry, the flag survives a crash forever and nobody can become leader. The full story is in the distributed lock post; the difference here is that leadership is not a lock but a timed lease.

The right model: leadership is leased

The lease idea: leadership is not granted forever, it is rented for a period. The leader renews the rent regularly. If it cannot renew — because it crashed, got partitioned or paused — the lease expires and someone else takes over.

CREATE TABLE leadership (
  job_name   text PRIMARY KEY,
  owner      text NOT NULL,
  expires_at timestamptz NOT NULL,
  epoch      bigint NOT NULL              -- ← fencing token
);

-- Every copy runs this every 5 seconds.
-- :me = pod name + a UUID generated at process start (NOT the pod name alone)
INSERT INTO leadership (job_name, owner, expires_at, epoch)
VALUES ('bridge_health', :me, now() + interval '15 seconds', 1)
ON CONFLICT (job_name) DO UPDATE
   SET owner      = :me,
       expires_at = now() + interval '15 seconds',
       -- epoch: unchanged on renewal, +1 when the leader changes OR the lease expired
       epoch      = leadership.epoch
                  + CASE WHEN leadership.owner = :me
                          AND leadership.expires_at >= now() THEN 0 ELSE 1 END
 WHERE leadership.owner = :me            -- either it is already mine (renewal)
    OR leadership.expires_at < now()     -- or it expired (takeover)
RETURNING epoch;

-- If a row came back: I am the leader, keep the returned epoch as my fencing token.
-- If no row came back: someone else leads, wait.

Twenty lines and no extra infrastructure. Five details matter:

• The lease must be at least 3× the renewal interval. Above: renew every 5 seconds, lease 15 seconds. Miss two renewals and I am still leader; on the third I lose it. A tighter ratio means a single network hiccup hands leadership over for nothing; this is called flapping.
• Takeover cannot happen before expiry. The two conditions in the WHERE clause guarantee that.
• :me must identify the process, not the pod. In a StatefulSet the hostname is stable; a new process restarting under the same name inherits the old epoch and the fencing is useless. Add a UUID generated at process start to the identity.
• The epoch increments even when the old owner renews an expired lease. If the lease expired, someone else may have been leader in between. Incrementing no matter who renews is the safer choice. Only a renewal before expiry keeps the epoch.
• The epoch column is not decoration. That is the next section.

Split-brain: two leaders at once

Now the bad news. The code above is correct but it does not prevent two copies from believing they are the leader. And no code can. Here is why:

A did nothing wrong; it just slept and does not know the world moved on. This is split-brain and in distributed systems it is unavoidable.

The fix: an epoch number (fencing token)

Produce an increasing number on every leadership handover and make the protected resource store it. The resource then rejects anything carrying a lower number:

-- A (epoch 7) wakes up and tries to update bridge status:
INSERT INTO bridge_status (bridge_id, health, checked_at, epoch)
VALUES ('LP-3', 'DOWN', now(), 7)
ON CONFLICT (bridge_id) DO UPDATE
   SET health     = EXCLUDED.health,
       checked_at = EXCLUDED.checked_at,
       epoch      = EXCLUDED.epoch
 WHERE bridge_status.epoch <= EXCLUDED.epoch;   -- ← the table says 8, condition fails

-- 0 rows affected. A could not write.
-- The application sees that, concludes "I am no longer leader", stops the loop.

Why an upsert and not a plain UPDATE? Because with a plain UPDATE, "0 rows" means two different things: either you were fenced, or there is no row for that bridge yet. On first start the leader would think it is not the leader and stop. With the upsert, a missing row becomes an INSERT; 0 rows now means only "you were fenced".

The subtlety: A does not have to ask whether it is still the leader. It tries to write, gets rejected, and learns. Correctness now rests on order rather than on timing — the same idea as in the distributed lock post.

The gap: no protection until the new leader writes once

Look at the table again. B became leader at 00:20, but bridge_status.epoch is still 7, because B has not written anything yet. If A wakes up at 00:22, the epoch <= 7 condition passes and the stale write goes through. The token in the protected resource lags behind the handover. The fix: as soon as you take leadership, and before doing any work, raise the epoch on the protected rows:

-- B became leader (epoch 8). BEFORE entering the health check loop:
UPDATE bridge_status
   SET epoch = 8
 WHERE epoch < 8;

-- Now A's write with epoch 7 matches no row.

Not needed for side-effect-free work

Not every job needs fencing. The dividing question: does the job leave a mark outside?

Do not write this yourself (usually)

The Postgres approach above is instructive, but in reality you probably already have a tool for this:

Writing your own Raft is almost never the right answer. That code looks correct, passes tests, and abandons you six months later during a network partition.

From the field: what we did with the health check

Back to the opening problem: four streaming engines, all polling the bridges. Three steps:

1. Leadership moved to a lease. 5-second renewal, 15-second lease. Ping traffic dropped to a quarter.
2. An epoch number was added to status writes. A copy coming out of a pause and trying to write "LP-3 is down" is rejected. Before that change, a copy waking from a pause marked a healthy bridge as dead and cut traffic to it — the most expensive lesson of the lot.
3. Leadership became a metric. Every copy publishes "am I the leader". The sum on the graph must be 1. If it drops to 0, nobody is doing the job; if it hits 2, there is a split-brain. One graph, both failures. The rule is not "1 right now" but "alarm if it is not 1 for 30 seconds": even a normal handover shows 0 for a few seconds (the old leader let go, the new one has not taken over yet), and that is not a failure. Be honest in the other direction too: a split-brain shorter than the scrape interval never shows on this graph. Fencing, not the graph, is the insurance for that.

UPDATE leadership
   SET expires_at = now() - interval '1 second'
 WHERE job_name = 'bridge_health' AND owner = :me;

How many leaders? One — but for what?

A common design mistake: electing one global leader and loading every exclusive job onto it. The result is a single bottleneck inside a system that otherwise scales horizontally.

Better: elect leadership per job. One leader for health checks, another for reconciliation, another for symbol sync. The load spreads, and one job's leader getting stuck does not affect the others.

Take that one step further and you are in sharding territory: leadership says "let one copy do it", sharding says "let every copy do its own share". They are not rivals; you pick based on whether the work divides.

Checklist

Conclusion

Leader election sounds academic, but the practical version is a very simple question: who is going to do this job? Answer "all of them" and you get noise; answer "none of them" and you get a silent failure.

One thing to keep: leadership is not property, it is a rental. It expires, it changes hands, and the previous holder may not know. The robustness of your system is measured by what that previous holder can still do when it wakes up late.