Rate Limits and Fallback: A Plan, Not an Error

From the field: an IPO morning

What happened that morning was not a surprise. We were simply not ready for it. When an IPO subscription opens, support questions always go up: “Was my order received?”, “How many lots will I get?”, “Why is my money blocked?” On a normal morning we get 90 questions per minute at the peak. That morning we got 150.

Each question used about 2,500 tokens on average: the system prompt, customer context, conversation history and the answer. 150 questions means 375,000 tokens per minute. Our quota was 400,000. On paper it fit. It did not fit, because we had forgotten a job that ran at the same hour. A background job that labels the tickets from the night started at 10:00 every morning, and it used 100,000 tokens per minute on its own. Total: 475,000. That is 119% of the quota.

The real damage came from our own code. When our client got a 429, it retried three times without waiting. Every rejected question became four requests. Requests per minute went from 270 to 610, and now the request quota of 500 was full too. Even short questions could not get through. In 38 minutes, about 1,900 of roughly 5,700 questions (33%) got the “I can’t help” answer.

At 10:52 everything recovered. We did nothing. There was nothing we could do. Traffic dropped by itself, and the labelling job finished. In the meeting after the incident, the first question was “how did you fix it?” The honest answer: we did not fix it. We waited for it to pass.

What a quota means: RPM, TPM and your share

An LLM provider sells you two things: a model, and a slice of that model. With most providers the size of the slice is described by two numbers:

• RPM (requests per minute): how many requests you can send per minute. Short requests fill it.
• TPM (tokens per minute): how many tokens you can process per minute. Long prompts fill it.

Some providers also have a daily limit. Which one fills first depends on the shape of your workload. Our live assistant sent few but long requests, so it was close to TPM and far from RPM. The labelling job sent many short requests. When the two shared the same quota, we got the worst side of both.

The first lesson is simple: think of the quota as a budget, not a number. Who spends that budget? For us the answer was: live customers, internal tools and background jobs. Before the incident all three took from the same pool, in no order. Nobody had written down who comes first.

Retry: with a budget, not immediately

The most common answer to a 429 is the one we gave: try again. The problem is not the retry. It is how you retry. We made three rules:

1. Come back when the provider tells you to. If the response has Retry-After, follow it. The provider knows better than you when the quota will free up.
2. If not, exponential backoff with jitter. If a hundred rejected requests all come back at the same moment, you get a second wave. Adding a random part spreads the wave over time.
3. Total waiting time is a budget. The customer is waiting on screen. Our budget is 6 seconds. The first token already takes 1–2 seconds (I covered this in the latency post), and more than 6 seconds of waiting on top of that feels broken. When the budget is spent, retries stop and the fallback starts.

WAIT_BUDGET = 6.0      # seconds; the user is waiting on screen
MAX_RETRIES = 2        # first request + 2 retries

def answer(question):
    start = now()
    for attempt in range(MAX_RETRIES + 1):
        resp = main_model.call(question)
        if resp.status != 429:
            return resp

        # come back when the provider says so
        wait_s = resp.headers.get("Retry-After")
        if wait_s is None:
            base   = 0.5 * (2 ** attempt)      # 0.5, 1, 2
            wait_s = random.uniform(0, base)   # full jitter

        # if the budget is not enough, do not wait: go to fallback
        if now() - start + wait_s > WAIT_BUDGET:
            break
        sleep(wait_s)

    metrics.inc("fallback", reason="429")
    return fallback(question)    # rules below

The most important line in this code is break. On the morning of the incident our client did not have it. The retries were not waiting for the quota to free up. They were the thing filling it.

Fallback: a second model, a second quality

After the incident the first reflex was “let us add a second model, so when one is full the other answers.” The reflex was right; the plan was incomplete. A fallback has two questions: how much worse is it, and which questions is it allowed to answer.

To measure quality, we picked 120 real questions from the last month with two people from the support team, and sent them to all three options. The two people scored the answers separately. For the 7 answers they did not agree on, they decided together:

The table led to three decisions. First: the small model only answers general questions. It does well on “how do I open an account” or “what are the trading hours”. Withdrawals, order cancellations and tax questions never go to the fallback; they go to a human agent. A wrong answer costs more than a slow one.

Second: no customer data goes to the second provider. Our contract and data processing agreement are with the main provider. The second provider only gets questions without personal data. In practice, that makes it a backup for the small model, not for the main one.

Third: a fallback is never silent. We log which model produced every answer (I described what we log and how in the prompt logging post). When a customer complains, this is the first field we check.

Graceful degradation: four levels

The real change was to see the quota as a gauge, not an on/off switch. As usage goes up, the system gives things up in order. First the things nobody will notice, last the things the customer will notice:

Level 1 alone would have saved the incident morning. If the labelling job had stopped, the total would have been 375,000 tokens, below the quota. Background jobs now run in a separate queue and can use at most 15% of the quota, which is 60,000 tokens per minute. Live customers always come first.

Even at level 3 the customer never sees “I can’t help”. They get a shorter answer, or “I am connecting you to one of our agents”. An error message is where the plan ends. Ours does not end before that point.

The first version kept jumping between levels

In the first version, the level dropped back as soon as usage went below the threshold. Level 2 reduced token usage, usage fell below 85%, the system went back to level 1, and usage rose again. In testing it switched twice a minute. The fix: moving up is instant, but moving down requires usage to stay below the lower threshold for 10 minutes.

What to track

One alert rule: if token usage per minute stays above 80% for five minutes, notify the on-call engineer. An alert that fires on 429 only tells you what the customer already sees.

What did not work for me

• Only asking for a bigger quota. After the incident, the first thing we did was ask the provider for more. Two weeks later it went from 400,000 to 500,000. Without a plan, the new quota fills at the next peak. The increase only postpones that day.
• Connecting 429 to a circuit breaker. In the first week, a 429 closed the main model completely for one minute. Most of the time the quota was not at 100%, it was at 103%. Closing everything for a minute pushed more traffic to the fallback than we were actually losing. Degradation levels worked better.
• Using one prompt with two providers. With the same prompt, the second provider’s model wrote long paragraphs instead of short lists. We had to keep a separate prompt, and now both prompts are tested together.

From the field: one month later, a second IPO

Tuesday, 17 March, the first morning of the next IPO subscription. This time questions went up to 170 per minute, higher than the first incident. Side by side:

The last row matters: degradation is not free. In the 9 minutes of short answers, the thumbs-up rate fell by 5 points. But the customer got a short answer, not a closed door. We never reached level 3; all 14 requests went through on retry.

Checklist

Conclusion

On the morning of 16 February we closed the door on our customers for 38 minutes. The reason was not a small quota. The reason was that we had never decided what should happen when the quota ran out. Without a decision, the system chose the worst option by itself: it rejected everyone at once, then made everyone retry at once.

A quota is a limit, and one day the limit will be reached. The question is not whether it fills. The question is what goes quiet first when it does. The labelling job, the long answer, or the customer?

You write that order on a calm day. If you do not, on IPO morning the provider writes it for you.