Why Backoff Matters: A Retry Storm Can Kill a Service

I was debugging a production issue in a service that looked like a performance problem.

The service was misbehaving under real traffic, but the code path looked normal. CPU was high, and clients were seeing intermittent instability.

At first, nothing obvious stood out. Then I traced one API call and found the real issue: the upstream service was returning a temporary error because the data was not ready yet.

That one failed request turned into a retry burst.

The retry logic was reasonable in principle, but the missing backoff meant the service kept retrying on that transient failure at full speed:

1while True:
2    try:
3        response = client.get("/data")
4        break
5    except TransientError:
6        continue

That missing delay was the bug.

This article builds on the ideas discussed in Software Robustness and Timeout Retry Backoff Paradigms. That article explains the theory. This one is the practical story: what happens when a retry loop is too aggressive.

​

Why Zero Backoff is Dangerous

A retry loop is not free. Each failed request does more than simply trigger another attempt. It creates another request, parses the error, schedules another retry, and repeats immediately.

In a tight loop like that, the service keeps the CPU busy, the upstream dependency keeps receiving pressure, and the whole call path becomes less stable.

The important point is that the problem is not only the failed request itself. The problem is the frequency of retries. When the service is already under stress, retrying immediately can turn a transient error into a sustained overload situation.

The effect is visible in two places: the service becomes CPU-heavy because it is spinning on retries, and the upstream dependency receives a much larger request rate than it should.

This is exactly the behavior we want to avoid.

​

The Experiment

The easiest way to see this clearly is to simulate a small service that returns a temporary error while its data is still being prepared.

The lab uses one upstream service (called server) that responds with a temporary 503 until its data is ready, one downstream service (called client) that retries failed requests with a configurable backoff, and a range of backoff settings from 0 ms to 2000 ms.

Each scenario runs in sequence, and the script records both client and server CPU usage.

The lab covers 32 combinations: eight backoff values crossed with four concurrency levels. It is intentionally small, so it highlights the pattern rather than serving as a capacity benchmark.

The point of the experiment is not to produce a perfect benchmark. It is to make the effect visible: as the retry delay grows, the retry storm weakens and the service becomes stable again.

​

What the Data Should Look Like

The expected shape is straightforward: CPU should be highest when retries happen in a tight loop, and should fall as the delay between attempts grows. That is a prediction about the mechanism, not a measurement of request volume or latency.

The experiment measures CPU usage for the client and server containers. It does not collect request counts, retry counts, or latency distributions, so those quantities should not be inferred from the figures below.

The client service is spending more time on retries than on useful work. When the retry loop is too fast, it amplifies the error. When the loop slows down, the error becomes manageable.

​

What This Run Measured

The measurements follow that pattern. Each cell below is the median CPU usage over 120 seconds of retries for a backoff and concurrency pair. The color scale is logarithmic because the values span almost three orders of magnitude.

At eight concurrent client threads, the client median falls from 93.8% with no backoff to 23.7% at 50 ms backoff and 0.8% at 2 seconds backoff. The server median falls from 92.0% to 21.5% and then 0.8% over the same values.

Even a 10 ms delay makes a substantial difference at lower concurrency, but it is not enough to contain the pressure at eight concurrent threads.

The first few samples also show why a summary alone is not enough.

The following trace shows one-client runs at 0 ms, 100 ms, and 1000 ms backoff. The initial startup spike is visible, but the sustained CPU level after startup is the important difference.

The collector generated pseudo-timestamps while processing docker stats, so the horizontal axis is useful for comparing the shape of the traces, not for reading precise wall-clock timing.

These numbers should not be read as universal CPU limits. The containers were constrained to one CPU each, and the result depends on the machine, runtime, request workload, and sampling method. The useful conclusion is the relationship: removing the tight retry loop removes most of the avoidable CPU pressure.

​

Why This Matters in Real Systems

The real lesson is not “do not retry”. It is that retrying is not free, and a retry loop without backoff can turn a temporary error into a self-inflicted load spike.

If a dependency is briefly unavailable or still preparing data, a client service that retries too aggressively can create a local hot loop. That service burns CPU and can send the upstream dependency a burst of traffic. In a production system, that extra load can grow the queue, increase latency, and make the original error more common. The retry storm can feed itself.

That is why backoff is not a minor implementation detail. It is part of failure containment.

The retry strategy should match the failure mode. For a temporary outage, a small delay and retry may be enough. For a dependency that is still warming up, backoff should be explicit. For a system under load, jitter and rate limiting are often needed as well.

Without that, you are not tolerating faults. You are amplifying them.

​

Summary

I do not remember the exact CPU usage from that incident, but I remember the pattern.

A retry loop with zero backoff can turn one temporary dependency failure into a CPU spike and a server-side load spike. The pattern is predictable: retries multiply pressure, and pressure makes the initial failure worse.

The fix is not to stop retrying. The fix is to retry with a deliberate delay, and to choose that delay based on the failure mode and the load on the system.

If you want to measure it yourself, use the lab in https://github.com/buildsoftwaresystems/labs/tree/main/retry-without-backoff-makes-your-service-fall-over and compare client and server CPU across backoff values from 0 ms to 2 s.

The data is usually enough to make the point without any ambiguity.