Failure Modes

// 1) Scheduled job: offset by 0-30s instead of a fixed second
long offset = ThreadLocalRandom.current().nextLong(30_000);
long delay = openTimeMs - System.currentTimeMillis() + offset;
scheduler.schedule(job, delay, TimeUnit.MILLISECONDS);   // ScheduledExecutorService: delay + unit

// 2) TTL: not fixed, ±20% spread
int ttl = 300;
int spread = ThreadLocalRandom.current().nextInt(-60, 61);
redis.setex(key, ttl + spread, value);

// 3) Staggered startup: wait based on pod index, but with an upper bound
int index = readPodIndex();            // StatefulSet ordinal; a Deployment has NO ordinal
long wait = Math.min(index * 500L, 15_000L)               // with 120 pods the last one must not wait 60 s
          + ThreadLocalRandom.current().nextLong(500);    // + jitter
Thread.sleep(wait);

The third one has two traps. Leave index * 500 unbounded and with 120 gateways the last pod waits 60 seconds; you need an upper bound plus jitter. The ordinal only exists in a StatefulSet; in a Deployment you hash the pod name, and a hash may not spread evenly — 8 of 20 pods can land in the same slot. That is why, in the hashed version, jitter is not decoration but a requirement. And this wait delays startup by up to 15 seconds: the startupProbe in section 5 (failureThreshold × periodSeconds) has to cover those 15 seconds too, otherwise liveness reads the wait you added yourself as "failed to start" and kills the pod.

The three cures above fix three of the four sources in the list: cron, TTL, bulk restart. The bridges connecting to their providers is your code too, and goes into the same staggered startup. But clients reconnecting after the night is not your code; you cannot fix, from the client side, every terminal sending TLS handshake + login + subscription in the same second at 09:00:00. There is a server-side answer, in four parts:

• Accept rate limit. Accept new connections at N per second (at the load balancer or in the application); the excess is rejected, not queued. Accepting a connection and then making it wait 20 seconds means a timeout for the client and a half-finished handshake for you.
• Reject with a jittered Retry-After. The connection-level form of the load shedding in section 3: a rejection that says "come back in 3–8 seconds" spreads connections piled into one second over 5 seconds.
• Staged work per connection. Accept the handshake, then limit login and the first subscription separately; the most expensive step (sending the symbol list) comes last and is limited the most.
• A limit per IP / account. Stops one client stuck in a reconnect loop from eating the whole accept budget.

If you ship the client SDK, there is a fifth: build jittered exponential backoff into the SDK; do not leave "reconnect at once when the connection drops" to the client. An SDK that waits a fixed 1 second brings every terminal to the door in the same second after every drop.

The other half of a bulk restart is how the pods stop. If a pod receives SIGTERM during a deploy and exits at once, two things happen: its in-flight requests are cut off, and the load balancer keeps sending it requests for a few more seconds, because leaving the endpoint list is not instant. Those requests fail, the client retries, the retries land on the remaining pods. The same herd, this time on the way out; and on every deploy.

# Deployment
lifecycle:
  preStop:
    exec: { command: ["sh", "-c", "sleep 5"] }   # wait to leave the endpoint list; SIGTERM comes after this
terminationGracePeriodSeconds: 30                 # preStop (5) + finishing in-flight work + shutdown; too small = SIGKILL

# Application (Spring): on SIGTERM stop accepting, finish what you have, then exit
server.shutdown=graceful
spring.lifecycle.timeout-per-shutdown-phase=20s

The order: first leave the list (a preStop wait, or readiness switched to 503), then stop accepting new connections, then finish what is in flight, then exit. If terminationGracePeriodSeconds is smaller than that total, the kubelet kills the process with SIGKILL and all the care is wasted.

// Store, alongside the value, how long computing it took and when it expires.
// As expiry approaches, the PROBABILITY of refreshing rises.

Entry e = read(key);                 // value + computeMs + expiresAt are stored together
// NOTE: remaining, computeMs and threshold must be in the SAME unit (ms); one in seconds silently breaks the formula
if (e == null) return computeAndStore(key).value;   // first read: no value yet (returns the value, not the Entry)

long remaining = e.expiresAt - now();
double u = 1.0 - ThreadLocalRandom.current().nextDouble();        // (0, 1]: no log(0) = -infinity trap
double threshold = -e.computeMs * BETA * Math.log(u);             // BETA ~ 1.0

if (remaining < threshold) {
    e = computeAndStore(key);         // early, spontaneously, alone
}
return e.value;                        // clients that do not refresh return the valid value too

The beauty: no locks, no coordination. Only a handful of clients refresh before expiry while the rest read a valid value. The key never expires "for everyone at once". And the more expensive the computation, the earlier the refresh starts — the formula handles that by itself.

// The default is usually NONE or 30+ seconds. In trading, 30 s means infinity.
// Connect timeout and read timeout are SEPARATE things:
// if the TCP/TLS handshake hangs, the read timeout never fires.

// Redis (Jedis 4+): the timeout goes into the constructor via config; there is no setTimeout method
JedisClientConfig cfg = DefaultJedisClientConfig.builder()
    .connectionTimeoutMillis(100)       // connect
    .socketTimeoutMillis(200)           // read: measured p99 ~120 ms x 1.5
    .build();
JedisPoolConfig poolCfg = new JedisPoolConfig();
poolCfg.setMaxWait(Duration.ofMillis(100));   // pool checkout: the default is -1 = UNLIMITED
JedisPool pool = new JedisPool(poolCfg, new HostAndPort(host, 6379), cfg);

// MT5 bridge (HTTP)
HttpClient http = HttpClient.newBuilder()
    .connectTimeout(Duration.ofMillis(200))                    // connect
    .build();
HttpRequest request = HttpRequest.newBuilder(uri)
    .timeout(Duration.ofMillis(800))                           // the WHOLE exchange, until the response completes: p99 ~500 ms x 1.5
    .build();

// Database: the query timeout does NOT cover getting a connection from the pool
hikari.setConnectionTimeout(500);   // checkout: Hikari default is 30 s; DBCP2 (maxWaitMillis=-1) is UNLIMITED
stmt.setQueryTimeout(3);            // the query itself (seconds); driver-dependent, on PostgreSQL statement_timeout is more reliable

There are three separate timeouts, and the invisible one is the third: waiting for a connection from the pool. Connect and read get configured, checkout gets forgotten. The defaults depend on the library: in HikariCP it is 30 seconds (the same as infinity on the trading side); in DBCP2, and in the GenericObjectPool underneath JedisPool, it is maxWaitMillis = -1, truly unlimited. Note where that leaves you: the pool with an unlimited checkout is on this post's own Redis path; the socketTimeoutMillis(200) above only starts after you have a connection from the pool. When all 20 connections in the pool are stuck in slow queries, the 21st request quietly starts waiting — it has no timeout, because it does not even have a connection yet. This is very often the real starting point of the cascade: step 2 of the table in section 3 can just as well begin with threads waiting for the pool, not for Redis.

Skip this step and nothing else works. Neither circuit breakers nor queue bounds can engage without timeouts.

How do you choose the value? Not from the air, from measurement: take the real p99 of that call and multiply by ~1.5. If Redis has a p99 of 120 ms, use 200 ms; if the bridge has 500 ms, use 800 ms. Second rule: it must be smaller than the caller's timeout. If the client gives up on the gateway after 1 second, the gateway's budget for Redis + bridge + DB must add up to less than 1 second; otherwise the gateway works on an answer nobody is waiting for. The next step is deadline propagation: if the client gave up at 800 ms, the layer below must not carry on. Pass the remaining time down in a header (X-Remaining-Ms: 450; gRPC's grpc-timeout carries exactly this); if the deadline has passed, the lower layer does not even start. Do not put an absolute timestamp in the header: "deadline 09:00:01.800" assumes the two machines have the same clock. A 200 ms clock skew between machines is a quarter of an 800 ms budget, and depending on its direction it either kills a live request or keeps a dead one alive. Send the remaining time and let each layer count down on its own clock. Otherwise, under load, you spend the most expensive work on requests whose answers go straight to the bin.

This is what cuts step 4. A bounded queue rejects requests when full — bad, but the process survives. An unbounded queue accepts until memory runs out and then loses all of it at once.

"The queue" is not one thing; a gateway has at least five, and first you have to find which one is unbounded:

• The accept queue. In Tomcat, acceptCount (default 100) and maxConnections (8,192 with NIO). Already bounded; when full, new connections are refused.
• The thread pool's work queue. Tomcat's own pool is bounded. But every pool you open with Executors.newFixedThreadPool(n) has an unbounded LinkedBlockingQueue behind it, and newCachedThreadPool() does not limit the number of threads at all. This is Java's most classic unbounded queue.
• The in-app async queue. Your own BlockingQueue that carries orders to the bridge or processes ticks. No capacity given means unbounded.
• The Netty / socket outbound buffer. Keep writing to a slow reader and the outbound buffer grows; WRITE_BUFFER_WATER_MARK and isWritable() exist for this.
• The client-side buffer. Subscription data piling up in the terminal; it kills the client, not your server, but when it drops it turns into the reconnect herd from section 1.

Since the accept queue is already bounded, the source of step 4 in the table (memory fills) is most likely the second or third item: your own pool or your own queue. The fix is three lines:

// WRONG: Executors.newFixedThreadPool(50) -> an UNBOUNDED LinkedBlockingQueue behind it
ExecutorService pool = new ThreadPoolExecutor(
    50, 50, 0L, TimeUnit.MILLISECONDS,
    new ArrayBlockingQueue<>(1_000),                       // bound: 1000 waiting tasks
    new ThreadPoolExecutor.AbortPolicy());                   // when full: RejectedExecutionException -> 503

// NOT CallerRunsPolicy: it runs the task on the thread that submitted it (Tomcat worker, Netty event loop),
// that thread blocks, accepting stops too; it looks like "graceful backpressure", it speeds up the cascade.

// Measure: time spent waiting in the queue, not utilisation. Utilisation is a late signal;
// a price request that has waited 5 s is already worthless, there is no point serving it.
// A simplification inspired by CoDel: real CoDel does not look at the INSTANT value but at whether
// the MINIMUM waiting time over an interval (100 ms) exceeds the target. In a short burst the minimum
// stays low and nothing is dropped; under sustained backlog the minimum rises too, and then it drops.
if (queue.minWaitLastWindowMs(100) > 500 && !request.isCritical()) {
    // A FIXED Retry-After brings every rejected client back at t+2 in the same second: the herd from section 1.
    int retryAfter = 2 + ThreadLocalRandom.current().nextInt(4);    // 2-5 s, with jitter
    return Response.status(503)
                   .header("Retry-After", String.valueOf(retryAfter))   // a HEADER, not the body
                   .build();
}

Rejecting a price lookup while accepting an order beats slowly losing both. Shedding load is not a failure, it is a prioritisation decision. Two details: in the first version of this post Retry-After: 2 was fixed — it called every rejected client back two seconds later, in the same second; I was producing in section 3 the very herd I described in section 1. And queue utilisation is a late signal: by the time the queue is 80% full, the request at its head has already been waiting for seconds. Waiting time in the queue is both an earlier and a more honest signal — but its minimum over a window, not its instant value: in the first version of this post the rule was "if the request at the head is older than 500 ms", and in that form it returned 503 on every single 600 ms burst. The whole point of CoDel is not to do that: if for 100 ms even the best request has waited 500 ms, the backlog is real, then drop.

Two more rules follow naturally. LIFO under overload. With a backlog, FIFO serves the oldest request first — the one whose client has long given up. 100% CPU, 0% useful work. Taking from the tail (LIFO) under load at least answers someone who is still waiting. Drop dead requests before dequeuing them. If the request carries a deadline (the deadline propagation above) and it has passed, drop it before starting; nobody will read the answer. Both recover capacity without adding any, because the capacity was always there, it was just going to the bin.

One more gap: in the code above the only distinction is isCritical(). Load shedding does not look at whose request it is; one client's retry storm (a broken bot, an integration stuck in a loop) eats the whole queue, and the ones rejected are the other 5,000 customers. The real-life companion of load shedding is fairness per customer: a per-tenant rate limit (nobody gets more than N requests per second) or weighted fair queuing (when the queue is full, the biggest consumer is dropped first). Next to "how full is the queue" you need "who filled it".

This is the defence that comes before the circuit breaker; the fourth in the order from section 3. Until the circuit opens (minimumNumberOfCalls=20 plus the window), the shared pool of 200 threads is already gone; the seven-second story happened exactly inside that window. Like the watertight compartments of a ship: each bridge gets its own pool or semaphore, and when one fills up the others keep working.

// A separate limit per bridge: one bridge cannot take all 200 threads
BulkheadConfig bh = BulkheadConfig.custom()
    .maxConcurrentCalls(25)                       // at most 25 calls to this bridge at once
    .maxWaitDuration(Duration.ofMillis(50))       // no room? reject within 50 ms, do not queue
    .build();
Bulkhead bridgeA = Bulkhead.of("mt5-bridge-A", bh);

Supplier<Order> guarded = Bulkhead.decorateSupplier(bridgeA, () -> sendToBridgeA(order));

When bridge A hangs, at most 25 threads get stuck there; 175 stay free for B, C and D. The circuit breaker comes next and rescues those 25 as well. The same rule applies to the breaker itself: one breaker per dependency. Put all bridges behind a single global breaker and bridge A's failures also close the path to B, C and D; in this story a global breaker is useless.

maxConcurrentCalls(25) is a guess; right for the bridge's capacity today, wrong when it slows down tomorrow. The next step is an adaptive concurrency limit: instead of setting the limit by hand, derive it from latency. Netflix's concurrency-limits library (the Gradient and Vegas algorithms; the same idea as TCP congestion control) lowers the limit as soon as p50 latency starts to rise and raises it when latency falls; in Envoy the same thing is the adaptive_concurrency filter. Start with a fixed 25, but put the latency graph next to the limit: if p99 doubles before you ever reach 25, the limit is too high.

// Breaker for the MT5 bridge (synchronous HttpClient.send)
CircuitBreakerConfig.custom()
    .slidingWindowType(SlidingWindowType.TIME_BASED)   // ← COUNT_BASED keeps hours-old calls under low traffic
    .slidingWindowSize(60)                              // last 60 seconds
    .minimumNumberOfCalls(20)                           // ← do not decide before 20 calls
    .failureRateThreshold(50)                           // 50% failures -> open
    .slowCallDurationThreshold(Duration.ofMillis(400))  // ← must be BELOW the timeout (800 ms), ~50% of it
    .slowCallRateThreshold(50)                          // ← SLOW calls count as failures
    .waitDurationInOpenState(Duration.ofSeconds(10))
    .automaticTransitionFromOpenToHalfOpenEnabled(true)  // ← default false: the transition only happens when a CALL arrives
    .permittedNumberOfCallsInHalfOpenState(5)
    .recordExceptions(IOException.class)                  // ← only infrastructure errors count; HttpTimeoutException extends IOException
    .ignoreExceptions(InsufficientMarginException.class,  // ← business errors do NOT open it
                      InvalidSymbolException.class)       //   (one of the two is enough, see below)
    .build();

// Do NOT copy the same config to the Redis path: Jedis does not throw IOException.
// JedisConnectionException -> JedisException -> RuntimeException. In Lettuce, RedisConnectionException
// and RedisCommandTimeoutException are RuntimeExceptions too. For the Redis breaker:
//   .recordExceptions(JedisConnectionException.class)      // both connection loss and socket timeout come from here

• Without minimumNumberOfCalls, the first two failing calls of the morning open the circuit and the service is never really tried.
• slowCallRateThreshold matters a lot. This is where gray failure gets caught: the bridge is not returning errors, it is returning in 5 seconds. If slow calls do not count as failures, the circuit never opens. The threshold must be below the timeout: in the first version of this post both were 800 ms, and with that there was no such category as "slow but successful" — every call reaching 800 ms was already counted as a timeout failure, so the mechanism was dead on arrival. The practical ratio is 50–60% of the timeout: 400–500 ms for an 800 ms timeout.
• recordExceptions / ignoreExceptions is the setting that burns most in practice. The default: every exception is a failure. If a group of customers gets "insufficient margin" in the morning — the bridge's correct answer — the circuit opens against a perfectly healthy bridge and everyone's orders are rejected. Separate business errors from infrastructure errors; the circuit should only look at the second kind. In the example above you do not need both: the moment you set recordExceptions, nothing outside that list counts as a failure, and ignoreExceptions has priority even over that. In practice, narrowing recordExceptions is the safer choice, because a new business exception added tomorrow does not open the circuit by default; if you go with ignoreExceptions, you have to remember to add every new business error to the list. Narrowing has its own trap, and it is the sneakiest form of this item: check the exception hierarchy of the client library. The first version of this post listed IOException and java.util.concurrent.TimeoutException. The second never comes out of a synchronous HttpClient.send(); that throws HttpTimeoutException, which is an IOException. TimeoutException only appears on the sendAsync / CompletableFuture path; I was showing synchronous code and recording an async exception, harmless but wrong. The harmful part is this: copy the same list to the Redis breaker and the circuit never opens, because Jedis does not throw IOException; a lost connection and a socket timeout are both JedisConnectionException, and that is a RuntimeException. It is not on the list, so it does not count as a failure, and the breaker stays closed even if Redis stops answering for days. Do it once per dependency: trigger the failure in a test, see which class the client really throws, and write that class.
• slidingWindowType gets forgotten. A count-based window (last 100 calls) keeps hours-old calls in the calculation under low traffic; 3 failures at 02:00 are still in the ratio at 09:00. A time-based window (last 60 s) answers the question "right now".
• automaticTransitionFromOpenToHalfOpenEnabled is off by default. While it is off, the OPEN→HALF_OPEN transition does not happen when waitDurationInOpenState ends; it happens on the first call after that. In the recovery scenario below, where you cut the traffic and wait, the circuit shows OPEN on the dashboard forever; you cannot answer "have the circuits recovered" without opening traffic. Set it to true and the transition happens on a timer by itself, you see HALF_OPEN on the dashboard, and the first 5 calls of the first 10% of traffic go through as trial calls. The cost is one timer per breaker; unless you have hundreds of breakers, it does not matter.

The first version of this post never said where retry goes; but if the wrapping order is wrong, neither the circuit breaker nor the bulkhead does its job. The order Resilience4j recommends, from the outside in, is Retry(CircuitBreaker(RateLimiter(TimeLimiter(Bulkhead(call))))):

// With Decorators, the first one written is the INNERMOST; retry is written last and sits on the OUTSIDE
Supplier<Order> s = Decorators.ofSupplier(() -> sendToBridgeA(order))   // the timeout lives on the HttpRequest itself (800 ms)
    .withBulkhead(bridgeA)         // 1) is there room? if not, reject within 50 ms
    .withCircuitBreaker(cbA)       // 2) is the circuit open? if so, CallNotPermittedException, the bridge is never called
    .withRetry(retryA)             // 3) OUTERMOST: retry only if the breaker let it through and an infrastructure error came back
    .decorate();

RetryConfig retryCfg = RetryConfig.custom()
    .maxAttempts(2)                                   // 2 attempts in total, not 3
    .intervalFunction(IntervalFunction.ofExponentialRandomBackoff(100, 2.0))   // with jitter
    .retryExceptions(IOException.class)               // synchronous HttpClient: the timeout is an IOException too (HttpTimeoutException)
    // The breaker rule applies here too: once the list is given, CallNotPermittedException, BulkheadFullException
    // and business errors are ALREADY not retried; no extra ignoreExceptions needed. The default (no list)
    // retries EVERY exception -- even while the circuit is open.
    .build();

This is the build order; at run time the chain works from the outside in, that is, in the reverse of the spine from section 3: retry → breaker → bulkhead → call. Both are right; do not mix them up when reading them side by side.

Why this order? Put retry inside the breaker and every attempt is recorded by the breaker as a separate failure: 3 attempts = 3 failures, the circuit opens three times faster. Put retry outside the breaker but leave the retryExceptions list empty, and the default retries every exception, CallNotPermittedException included: the retry waits out its backoff even while the circuit is open; no call is made, but the thread waits. The second rule is less known and more expensive: retries multiply across layers. The client retries 3 times, the gateway 3 times, the bridge client 3 times: the load reaching the source is not 3×, it is 3 × 3 × 3 = 27×. The retry budget from section 1 caps this at 10% in one layer, but if each of the three layers retries within its own budget, the multiplication is still there. The rule: only the outermost layer retries; the layers in between pass the error straight up.

And what the TimeLimiter in that order is for: the library that does not accept a timeout. This is the escape hatch of the "without timeouts nothing works" thesis. If the old SDK from the provider has no time parameter on send(), you run the call in a separate pool and cut it from the outside:

// The SDK knows no timeout: the call runs in a separate thread pool, the limit comes from outside
ThreadPoolBulkhead tpb = ThreadPoolBulkhead.of("bridge-B", ThreadPoolBulkheadConfig.custom()
    .maxThreadPoolSize(25).queueCapacity(1).build());      // 25 threads, practically no queue (minimum is 1): no room = reject
TimeLimiter tl = TimeLimiter.of(Duration.ofMillis(800));

CompletableFuture<Order> f = Decorators.ofSupplier(() -> oldSdk.send(order))
    .withThreadPoolBulkhead(tpb)              // 1) separate pool
    .withTimeLimiter(tl, scheduler)           // 2) cut the future at 800 ms -> java.util.concurrent.TimeoutException
    .withCircuitBreaker(cbB)                  // 3) on this path the breaker DOES record TimeoutException.class
    .withRetry(retryB, scheduler)             // 4) outermost
    .get().toCompletableFuture();

Two honest notes. TimeLimiter cancels the future, not the thread: if the SDK ignores interrupts, the cancelled call keeps holding that thread until the answer arrives. That is exactly why the pool is limited to 25; at most 25 zombies, then rejection. And above I said "a synchronous call never throws TimeoutException"; this path is asynchronous, here it does, and the breaker's record list is written accordingly. Whichever path you are on, that path's exception.

Catching the 1.8-second GC pause in the health check is diagnosis; in a price engine the real work is making sure that pause does not happen. The order matters, from cheap to expensive:

• Measure. Read the reason for the pause with -Xlog:gc*: humongous allocation, full GC, promotion failure? The treatment depends on the cause; changing the collector without measuring is guessing.
• Lower the allocation rate. Do not create new objects per tick: reuse tick objects, stop boxing, use primitive collections, take string building out of the hot path. This is usually the most effective step, and it does not depend on the collector.
• Choose the collector. G1's pauses grow with the heap; ZGC (its generational mode arrives in JDK 21 behind -XX:+ZGenerational, and is the default from JDK 23) and Shenandoah keep pauses in the millisecond range independent of heap size, at the cost of some CPU and throughput. For a price engine that is the right trade; for a batch job it is not.
• Move large data off the heap. Big, long-lived structures like symbol history or order book snapshots go off-heap (Chronicle, Agrona, a direct ByteBuffer) or into a separate process, so the collector has fewer live objects to scan.