Distributed Systems Error Handling: When to Retry, Reconcile, or Crash

When I first started working with microservices, error handling confused me. If a service hit an unexpected error, my initial reaction was to throw an uncaught exception or let the program panic. I assumed the service should simply stop and exit.

This approach caused problems. My programs crashed and restarted constantly over minor network hiccups. I eventually learned how to keep systems alive using retries, but a new dilemma emerged: when should a service retry an operation, when should it give up and crash, and when should it back off and wait to reconcile?

Through trial and error, I found a reliable way to structure microservices so they can run predictably for months without unnecessary restarts.

​

The Golden Rule: Transient vs. Permanent Errors

The decision to retry or crash comes down to one simple rule: retry transient errors, and crash on permanent errors.

A transient error is a temporary issue that can resolve itself over time. Network glitches, brief external service outages, or temporary database locks are all transient. If you wait a moment and try again, the operation might succeed.

A permanent error will never recover on its own. If your application has the wrong database credentials, or a required configuration file is missing in a static environment, retrying a hundred times will not fix the problem. You must stop the application.

​

Fail Fast at Startup

To apply this rule effectively, structure your microservice into two distinct phases: startup and runtime.

Push all your hard, permanent checks to the very beginning of the program’s execution. When your application boots up, it should read configuration files, validate variables, and verify essential environment details.

If anything goes wrong during this startup phase, do not attempt to recover. Crash the application immediately. If a static configuration file is missing, fail fast and let your orchestrator (like Kubernetes) handle the restart. There is no value in starting a broken application that will not change.

​

Use Retries During Runtime

Once your service finishes its startup checks and begins serving traffic, shift your strategy. Treat most new external errors as transient.

If your service makes a network request to an external API and receives no answer, do not crash. The network might have dropped the packet, or the external service might be temporarily overwhelmed. In these runtime scenarios, you should retry the request.

To handle these runtime glitches efficiently, you need a solid retry mechanism. I wrote a detailed breakdown on how to implement timeouts, retries, and backoff.

​

Context Matters: Static vs. Dynamic Environments

Sometimes, the line between a permanent and transient error depends entirely on your system’s architecture.

Take a missing configuration file, for example. In a static deployment where configurations never change after a container spins up, a missing file is a permanent error. You must crash.

However, in a dynamic environment where configurations update at runtime, a missing file might just mean the update has not arrived yet. In this scenario, the error is transient. You should log an informational warning for the administrator, wait, and retry reading the file.

For example, imagine a Kubernetes deployment where a ConfigMap is updated independently of your application pod. If your service attempts a hot-reload exactly when the ConfigMap is being synced, the file might be temporarily locked or incomplete. Crashing the pod here causes an unnecessary outage. Catching the error, applying a short 5-second backoff, and retrying allows the sync to finish, keeping the service alive without human intervention.

The same logic applies to external services. If you expect a downstream service to be fully provisioned before your application starts, a missing connection is a permanent error—crash at startup. If you assume eventual consistency where services boot up at different speeds, treat the missing service as a transient error and retry the connection until it succeeds.

​

When to Stop Retrying: The Reconciliation Pattern

When a network connection drops, your application cannot look into the future. It does not know if the failure will last two seconds or two hours. Naturally, it assumes the error is transient, starts retrying, and (hopefully) uses jitter to randomize the request intervals.

But what happens during a sustained outage, like a 30-minute routing table failure?

Within a few minutes, your retry loops will hit their maximum backoff cap (usually 30 or 60 seconds). Once capped, your code stops expanding its delay. It just sits there, hammering the broken endpoint every minute.

If you have dozens of instances doing this, they create a massive wall of noisy traffic. The moment an engineer fixes the network, this “thundering herd” can instantly crash the recovering downstream system.

Transient errors can become structural. You cannot always loop forever.

After a set number of failed attempts, trip a circuit breaker. Stop the automated retry loop, fall back to your last known good state, and transition to a reconciliation pattern.

Instead of actively hammering the dependency, your system idles and waits for an external signal (like an orchestrator webhook or a successful background health check) to safely resume operations.

​

Conclusion

Building self-recovering microservices requires clear boundaries. Define your environment assumptions early. Identify what can change during runtime and what remains fixed.

Crash early and cleanly for fixed, unrecoverable errors during startup. Rely on smart retries for dynamic, temporary issues during runtime. And when those temporary issues drag on too long, break the loop and rely on reconciliation.

By separating your startup validation from your runtime execution, and knowing exactly when to stop retrying, you ensure your microservices stop predictably when they should and recover gracefully when they can.

Applying these boundaries is a core part of building resilient applications. For more on designing systems that survive failure, read my deep dive into Quality Attributes of Computer Programs: Implement Software Robustness.