Failure Modes

// 1) Scheduled job: offset by 0-30s instead of a fixed second
long offset = ThreadLocalRandom.current().nextLong(30_000);
scheduler.schedule(job, openTime + offset);

// 2) TTL: not fixed, ±20% spread
int ttl = 300;
int spread = ThreadLocalRandom.current().nextInt(-60, 61);
redis.setex(key, ttl + spread, value);

// 3) Staggered startup: wait based on pod index, but with an upper bound
int index = readPodIndex();            // StatefulSet ordinal; a Deployment has NO ordinal
long wait = Math.min(index * 500L, 15_000L)               // with 200 pods the last one must not wait 100 s
          + ThreadLocalRandom.current().nextLong(500);    // + jitter
Thread.sleep(wait);

The third one has two traps. Leave index * 500 unbounded and with 200 gateways the last pod waits 100 seconds; you need an upper bound plus jitter. The ordinal only exists in a StatefulSet; in a Deployment you hash the pod name, and a hash may not spread evenly — 8 of 20 pods can land in the same slot. That is why, in the hashed version, jitter is not decoration but a requirement.

// Store, alongside the value, how long computing it took and when it expires.
// As expiry approaches, the PROBABILITY of refreshing rises.

Entry e = read(key);                 // value + computeMs + expiresAt are stored together
if (e == null) return computeAndStore(key);   // first read: no value yet

long remaining = e.expiresAt - now();
double u = 1.0 - ThreadLocalRandom.current().nextDouble();        // (0, 1]: no log(0) = -infinity trap
double threshold = -e.computeMs * BETA * Math.log(u);             // BETA ~ 1.0

if (remaining < threshold) {
    e = computeAndStore(key);         // early, spontaneously, alone
}
return e.value;                        // clients that do not refresh return the valid value too

The beauty: no locks, no coordination. Only a handful of clients refresh before expiry while the rest read a valid value. The key never expires "for everyone at once". And the more expensive the computation, the earlier the refresh starts — the formula handles that by itself.

// The default is usually NONE or 30+ seconds. In trading, 30 s means infinity.
// Connect timeout and read timeout are SEPARATE things:
// if the TCP/TLS handshake hangs, the read timeout never fires.

// Redis (Jedis 4+): the timeout goes into the constructor via config; there is no setTimeout method
JedisClientConfig cfg = DefaultJedisClientConfig.builder()
    .connectionTimeoutMillis(100)       // connect
    .socketTimeoutMillis(200)           // read: measured p99 ~120 ms x 1.5
    .build();
Jedis jedis = new Jedis(new HostAndPort(host, 6379), cfg);

// MT5 bridge (HTTP)
HttpClient http = HttpClient.newBuilder()
    .connectTimeout(Duration.ofMillis(200))                    // connect
    .build();
HttpRequest request = HttpRequest.newBuilder(uri)
    .timeout(Duration.ofMillis(800))                           // read: p99 ~500 ms x 1.5
    .build();

// Database: the query timeout does NOT cover getting a connection from the pool
hikari.setConnectionTimeout(500);   // pool checkout
stmt.setQueryTimeout(3);            // the query itself (seconds)

Skip this step and nothing else works. Neither circuit breakers nor queue bounds can engage without timeouts.

How do you choose the value? Not from the air, from measurement: take the real p99 of that call and multiply by ~1.5. If Redis has a p99 of 120 ms, use 200 ms; if the bridge has 500 ms, use 800 ms. Second rule: it must be smaller than the caller's timeout. If the client gives up on the gateway after 1 second, the gateway's budget for Redis + bridge + DB must add up to less than 1 second; otherwise the gateway works on an answer nobody is waiting for. The next step is deadline propagation: if the client gave up at 800 ms, the layer below must not carry on. Pass the remaining time down in a header (X-Deadline, grpc-timeout in gRPC); if the deadline has passed, the lower layer does not even start. Otherwise, under load, you spend the most expensive work on requests whose answers go straight to the bin.

This is what cuts step 4. A bounded queue rejects requests when full — bad, but the process survives. An unbounded queue accepts until memory runs out and then loses all of it at once.

// Measure: the age of the request at the HEAD of the queue (CoDel logic). Utilisation is a late signal;
// a price request that has waited 5 s is already worthless, there is no point serving it.
if (queue.headAgeMs() > 500 && !request.isCritical()) {
    // A FIXED Retry-After brings every rejected client back at t+2 in the same second: the herd from section 1.
    int retryAfter = 2 + ThreadLocalRandom.current().nextInt(4);    // 2-5 s, with jitter
    return error(503, "Retry-After: " + retryAfter);
}

Rejecting a price lookup while accepting an order beats slowly losing both. Shedding load is not a failure, it is a prioritisation decision. Two details: in the first version of this post Retry-After: 2 was fixed — it called every rejected client back two seconds later, in the same second; I was producing in section 3 the very herd I described in section 1. And queue utilisation is a late signal: by the time the queue is 80% full, the request at its head has already been waiting for seconds. The age of the work at the head of the queue is both an earlier and a more honest signal.

This is the defence that comes before the circuit breaker. Until the circuit opens (minimumNumberOfCalls=20 plus the window), the shared pool of 200 threads is already gone; the seven-second story happened exactly inside that window. Like the watertight compartments of a ship: each bridge gets its own pool or semaphore, and when one fills up the others keep working.

// A separate limit per bridge: one bridge cannot take all 200 threads
BulkheadConfig bh = BulkheadConfig.custom()
    .maxConcurrentCalls(25)                       // at most 25 calls to this bridge at once
    .maxWaitDuration(Duration.ofMillis(50))       // no room? reject within 50 ms, do not queue
    .build();
Bulkhead bridgeA = Bulkhead.of("mt5-bridge-A", bh);

Supplier<Order> guarded = Bulkhead.decorateSupplier(bridgeA, () -> sendToBridgeA(order));

When bridge A hangs, at most 25 threads get stuck there; 175 stay free for B, C and D. The circuit breaker comes next and rescues those 25 as well.

CircuitBreakerConfig.custom()
    .slidingWindowType(SlidingWindowType.TIME_BASED)   // ← COUNT_BASED keeps hours-old calls under low traffic
    .slidingWindowSize(60)                              // last 60 seconds
    .minimumNumberOfCalls(20)                           // ← do not decide before 20 calls
    .failureRateThreshold(50)                           // 50% failures -> open
    .slowCallDurationThreshold(Duration.ofMillis(800))
    .slowCallRateThreshold(50)                          // ← SLOW calls count as failures
    .waitDurationInOpenState(Duration.ofSeconds(10))
    .permittedNumberOfCallsInHalfOpenState(5)
    .recordExceptions(IOException.class, TimeoutException.class)          // ← only infrastructure errors count
    .ignoreExceptions(InsufficientMarginException.class,                  // ← business errors do NOT open it
                      InvalidSymbolException.class)
    .build();

• Without minimumNumberOfCalls, the first two failing calls of the morning open the circuit and the service is never really tried.
• slowCallRateThreshold matters a lot. This is where gray failure gets caught: the bridge is not returning errors, it is returning in 5 seconds. If slow calls do not count as failures, the circuit never opens.
• recordExceptions / ignoreExceptions is the setting that burns most in practice. The default: every exception is a failure. If a group of customers gets "insufficient margin" in the morning — the bridge's correct answer — the circuit opens against a perfectly healthy bridge and everyone's orders are rejected. Separate business errors from infrastructure errors; the circuit should only look at the second kind.
• slidingWindowType gets forgotten. A count-based window (last 100 calls) keeps hours-old calls in the calculation under low traffic; 3 failures at 02:00 are still in the ratio at 09:00. A time-based window (last 60 s) answers the question "right now".