Pythonium

Python, What else ?

When retries make an outage worse

When an application encounters a temporary error, retrying often seems like a good idea. An HTTP request fails? Send it again. A database connection fails? Try again. A remote API doesn't respond? Maybe it will work on the second or third call.

It's such a common technique that it's easy to add it without really thinking about it. The problem is that a retry doesn't make the error disappear. It simply adds another attempt.

And in some situations, these additional attempts can actually make the problem worse. I've experienced this myself: retries triggered a snowball effect, and things quickly turned into a disaster. Since then, I've been much more careful about how I handle retries.

An outage can cause even more requests

Let's imagine an application that calls an external API. Under normal circumstances, it gets a response within a few hundred milliseconds. But one day, the API starts having problems and responds very slowly.

The application then assumes that the request has probably failed and decides to retry. One request therefore becomes two requests. Then three if the second one also fails.

If a few users do this, it's probably not a big deal. But with several thousand simultaneous requests, the behavior becomes different. The service that is already struggling now receives even more work precisely when it has the least capacity to handle it.

You end up in a pretty vicious circle: the service slows down, clients retry, the number of requests increases, the service slows down even more, which causes new retries.

The mechanism designed to make the application more resilient can therefore contribute to bringing the service down.

The problem is even more obvious with timeouts

Retries are often associated with timeouts. Suppose an application waits a maximum of five seconds for a response from an external service. If the service doesn't respond, the request times out and the application retries.

But during those five seconds, the first request hasn't necessarily disappeared from the remote system. It may still be being processed. The application can therefore send a second request while the first one is still working. If the second one also times out, a third one can be sent.

For a single user action, the remote system can ultimately receive several operations simultaneously. The service then processes multiple instances of an operation that was only needed once. This happened to me recently with processes taking more than a minute: we ended up with around twenty processes running in parallel, without actually making the initial process move forward.

And if the operation isn't idempotent, this can create an even more serious problem. Well, if the service is properly designed, this kind of situation shouldn't normally happen.

Retrying a read and retrying a write are not the same thing

Not all operations are equally suited to retries. If a GET request fails, retrying it is generally fairly straightforward. Context obviously matters, but performing the same read again normally doesn't modify the data.

With a write operation, the situation is different. Imagine an application asking a service to create an order. The server actually creates the order, but the response never reaches the client because of a network problem.

The client then assumes that the creation failed. It retries. The server receives a new request and potentially creates a second order. From the client's point of view, there was an error. From the server's point of view, the operation had actually succeeded.

This is one of the classic problems in distributed systems: not receiving the response doesn't necessarily mean that the operation wasn't executed. This absolutely needs to be taken into account for certain requests, otherwise you can end up with some very unpleasant surprises.

Idempotence then becomes important

One way to limit this problem is to make certain operations idempotent. For example, instead of simply asking:

"Create this order."

you can send a unique request identifier.

The server can then remember this identifier and consider a new request with the same identifier to be the same operation. If the first request has already been processed, the server can return the existing result instead of creating a new order. The retry then becomes much less dangerous.

This obviously requires more work on the server side. But it's often preferable to assuming that a request can always be executed multiple times without consequences.

Retrying immediately is generally not a good idea

Another important detail is the delay between attempts. If a thousand clients retry immediately after an error, the server can receive a new wave of requests almost instantly.

It's generally better to introduce a delay. For example, an application might wait 100 ms before the first additional attempt, then 200 ms, then 400 ms. This is generally called exponential backoff.

The idea is fairly simple: the more attempts fail, the longer we wait before trying again. This gives the service some time to recover.

Randomness can also help

Even with exponential backoff, another problem can appear. If all clients use exactly the same delays, they can end up retrying at the same time.

Imagine several thousand machines all waiting 100 ms, then 200 ms, then 400 ms. The requests may then arrive in waves.

To avoid this, a random component is often added to the delay. This is called jitter. The different machines no longer retry at exactly the same time, which helps spread the load more evenly.

This small detail can be quite important when the number of clients is large.

Not every problem deserves a retry

It's also important to distinguish temporary errors from errors that aren't temporary. If an API returns a 500 error, a retry can sometimes make sense.

But if it returns an error indicating that the request is invalid, basically an HTTP 400 error, sending exactly the same request again probably won't solve anything. The same principle applies to 401 authentication errors or missing resources.

Automatically retrying every error ultimately means asking the system to repeatedly do something it already knows is impossible. You therefore need to define which errors can actually be temporary.

Retries can also multiply

The problem becomes even more subtle when an application is made up of several services.

Imagine a service A that calls B, which calls C.

A makes three attempts to B.

B itself makes three attempts to C.

A single initial request can therefore result in up to nine calls to C. With more levels, the multiplication can become much greater.

And if each service has its own retry rules, it becomes difficult to understand how many requests a user action can actually generate.

This is one of the reasons retries need to be considered at the system level and not just at the level of a single function.

Sometimes you have to stop retrying

A robust system doesn't necessarily try to succeed at all costs. It also needs to know when to give up.

A maximum number of attempts prevents an operation from remaining stuck indefinitely. A global maximum duration can also prevent a succession of retries from turning a request that should take a few seconds into an operation lasting several minutes.

In some cases, it may also be preferable to temporarily take a service out of the equation when it accumulates errors.

This is the idea behind a circuit breaker: when calls fail too often, we temporarily stop sending them in order to give the service time to recover.

The goal is no longer to make every request succeed immediately, but to prevent the entire system from going down along with the struggling service.

A retry is just a trade-off

Retries are obviously still very useful. A temporary network problem or a short period of unavailability can sometimes be resolved simply by trying again.

But a retry also adds traffic, latency, and sometimes side effects. You therefore need to know which errors should trigger a retry, how many times to retry, and how long to wait between attempts.

In production, a retry can turn a temporary error into overload, which itself causes more errors and therefore more retries.

The mechanism intended to make the system more resilient can then become part of the problem.

Retrying isn't necessarily a bad idea. You simply need to keep in mind that a solution that works normally can behave very differently when everything starts going wrong.




Popular posts