Operability First: Designing for Safe Recovery
Designing distributed systems so operators can determine what happened, control retries, and recover safely when normal processing fails.
Consider a worker that reads a message from a queue and calls a downstream service. The service commits the requested change, but its response never reaches the worker and the worker times out. After several unsuccessful attempts, the message ends up in a dead-letter queue.
Now someone has to decide whether to replay it.
The timeout tells them that the worker didn't receive a response. It doesn't establish whether the downstream operation completed. Moving the message into a DLQ has preserved it for investigation, but it has not resolved that uncertainty. Replaying it might finish the work, duplicate a side effect, or fail in exactly the same way.
The capacity plan for this system could be perfectly sound. Its normal processing path could meet every throughput and latency target. But we'd still have an unanswered question about how to recover it.
That's the question I want brought into the design discussion earlier. When normal processing fails, what will the people operating the system know, and what will they be able to do safely?
Decide what recovery requires
In this example, the recovery design depends on the downstream operation.
If the downstream service supports idempotent requests, the worker might be able to repeat the call using the same operation identifier. But that immediately raises further questions. Where is the identifier recorded? How long is it retained? Does it remain useful if we replay the message several days later? How are the deduplication record and the actual state change coordinated?
Putting an idempotency key in a message doesn't answer those questions by itself.
Alternatively, the downstream service might expose a way to look up the operation’s outcome. Recovery could then reconcile what happened before deciding whether to submit anything again. If the service offers neither deduplication nor a reliable status lookup, automatic replay may be inappropriate. Some messages may need to remain quarantined until someone can establish their effects.
I want the system to represent that uncertainty honestly. A processing failure and a confirmed failure to apply the requested change are different states, even if both initially appear as a timeout in the worker’s logs.
These decisions affect what information we retain, which interfaces we need, and what we allow operators to repeat. They're worth making while those parts of the system are still easy to change.
Retry settings interact across boundaries
Now suppose the worker makes up to three total attempts, and the client library it uses also makes up to three attempts for each worker attempt. A single message can produce nine downstream calls before we consider queue redelivery or an operator replay.
Each setting might look reasonable to its owner. The downstream service receives the combined effect.
This is why I think of retries, concurrency limits, timeouts, and consumption rates as parts of a distributed control system. Together, they determine how much work we send toward a dependency, how long we keep trying, and what happens when responses slow down. Their implementations may live in different places, but their effects need to be considered together.
For the queue pipeline, I want to know which layer owns retrying, what the overall time budget is, and how much outstanding work the downstream service can tolerate. I also want to distinguish failures that justify another attempt from failures that need a change in input or configuration.
Backoff and jitter are useful within that design. They do not, on their own, settle how much additional traffic we're willing to generate.
The same accounting applies to recovery traffic. A replay tool can obey its own rate limit while still overwhelming a dependency that's already serving live requests. Its budget needs to reflect the capacity available during recovery, rather than the rate at which the tool can drain the queue.
I'd evaluate a retry-policy change against both completed work and the pressure it places on dependencies. An improvement in the worker’s success rate is incomplete evidence if downstream saturation and tail latency are getting worse.
Give operators evidence they can use
Return to the message in the DLQ. Before replaying it, an operator needs to establish which operation it represents, what attempts have already occurred, and whether the original reason for failure still applies.
That suggests a fairly concrete telemetry requirement. The operation identifier needs to survive the asynchronous boundary. Attempt records need to distinguish a rejected request from an indeterminate outcome. The operator should be able to connect the message to downstream evidence without reconstructing the relationship from timestamps.
For the queue as a whole, depth is useful, but message age matters too. A queue with a modest number of very old messages presents a different problem from one that's temporarily accumulating work and processing it successfully. During replay, I'd distinguish replay traffic from new arrivals so we can see whether recovery is delaying current work.
These signals should support decisions. Can we start? Should we slow down? Has the failure changed? Do we need to stop?
The recovery mechanism needs corresponding controls: a way to select the affected messages, limit the rate, pause the operation, and inspect its results. A runbook can explain how to use those controls and what evidence justifies proceeding.
An empty DLQ is not sufficient verification. In our example, messages could leave the queue after producing duplicate changes. Recovery needs a check against the intended downstream state, not merely the disappearance of failed work.
Work out who can act
Suppose the worker and downstream service belong to different teams.
The worker’s on-call engineer can replay messages but cannot inspect the downstream records. The downstream team can determine whether an operation completed but has no responsibility for the queue. Both teams have useful tools. Neither can complete the recovery alone.
This is the sociotechnical part of operability that concerns me. The recovery path depends on permissions, ownership, and coordination as well as software behavior.
Before an incident, I want agreement about who establishes the downstream state, who authorizes replay, and who can stop it when live traffic begins to suffer. Those responsibilities can be lightweight in a small team. Across organizational boundaries, they may need to be part of an explicit service agreement.
Ownership also has to survive handoffs. A procedure that depends on one engineer remembering an undocumented exception is difficult to rely on when that person is unavailable.
A recovery exercise can test this without requiring a large chaos-engineering program. For the queue example, we could arrange for the downstream change to succeed while the response is lost, then ask an operator to investigate and recover the message using the normal tools and permissions.
That exercise tests something a successful retry-path unit test cannot establish: whether the people responsible for the system can determine what happened and carry out the procedure. Missing access, insufficient records, and ambiguous instructions become visible while there is still time to fix them.
Match the investment to the consequences
There's a cost to all of this. Durable operation records require storage and lifecycle decisions. Safety checks can add latency. Recovery tools need maintenance. Exercises take time from other work.
I don't think every pipeline needs the same machinery.
A job that rebuilds a disposable cache may be recoverable by discarding its output and starting again. A workflow that changes externally owned state needs a more careful account of what repetition would do. Even within one system, some operations may be safe to retry freely within a time budget, while others require reconciliation.
The useful design question is how much recovery capability the failure consequences justify.
Sometimes the right answer is a manual procedure. For an infrequent operation, a documented investigation with sufficient evidence may be safer and cheaper than maintaining an automated replay system. Manual recovery still needs an owner, access to the relevant state, and a way to verify the result.
Performance remains a constraint throughout. Retaining more state or adding a lookup can impose real costs. I want those costs considered alongside the difficulty of recovering without them, so the team can make a deliberate tradeoff.
Improving a system already in production
Many of these changes can be made after launch. Existing systems are often where the work is most valuable.
I'd start with a consequential workflow and trace its failure path. What happens when a dependency slows down? What happens when it commits a change but the caller cannot confirm it? Where does failed work accumulate? What would an operator do with it today?
The first useful improvement may be small: limiting concurrency, removing overlapping retry layers, retaining an operation identifier, or preventing automatic replay of an ambiguous outcome. Those changes can reduce risk while more involved recovery work is being designed.
Some gaps will require changes at service boundaries. If the downstream system provides no way to recognize a repeated operation or inspect its result, worker-side instrumentation alone cannot supply that information. We may need an API change, a reconciliation process, or a more restrictive recovery procedure.
I prefer to identify that constraint directly, then work toward it through changes the team can deploy and verify. A complete rewrite is rarely a useful starting assumption.
The design document should capture the failure behavior alongside the normal path: what is safe to repeat, what remains uncertain, which limits contain the damage, and how recovery is checked. That gives implementation and operational work a shared reference.
For the engineer opening our dead-letter queue, those decisions make a practical difference. They should be able to determine which messages are safe to process, which need investigation, and what evidence will show that recovery succeeded.
I consider that part of delivering the system. The people responsible for it need a workable way through the failure path, using information and controls the design has deliberately provided.