Engineering approach

I judge architectural choices partly by what they leave the team dealing with afterward: dependencies, deployment coordination, incomplete work, and recovery. These are the questions and design practices I return to when building or changing distributed software.

I start with the workload, the behavior that must remain correct, and the environment in which the system will run. A useful boundary can make one component easier to change while introducing network calls and coordination elsewhere. I want those costs to be explicit, and I stay close to implementation to check the assumptions behind the design.

Diagnosability by design

When work spans several services or queues, I want to be able to follow one operation through the system. Where did it stop? Did a dependency reject the request, or did the response disappear after the work completed? Which results are trustworthy, and what action is safe now?

That shapes the instrumentation. Correlation and trace context need to survive asynchronous boundaries. Structured events need to identify the operation, its outcome, and the dependency involved. Metrics should distinguish useful progress from accumulating work: a busy worker pool and a growing queue tell a different story from a system completing requests at a steady rate.

I organize dashboards around how the system works, such as a queue and its workers or a workflow and its dependencies. Alerts should connect a symptom to something an operator can investigate or act on. More telemetry earns its cost when it helps answer those questions; a large volume of undifferentiated logs can leave the diagnosis just as uncertain.

Resilient workflows and pipelines

A timeout does not establish that an operation failed. Before retrying, I want to know whether the original request could have committed a side effect and how a second attempt will be recognized. That may require an idempotency key, a status query, or a different workflow.

Retry policy also affects load. I look at which failures are transient, where retries occur, and how much time and additional work the overall operation can spend. Several layers each making reasonable local retry decisions can still create excessive downstream traffic. Backoff and jitter help spread attempts, but they do not replace a budget or a decision about when to stop.

Concurrency limits and backpressure need the same attention. Where will work wait when demand exceeds capacity? How old can it become before it loses its value? Increasing worker count may improve throughput, or it may overload the database every worker depends on.

A circuit breaker adds state and another recovery behavior to understand. I would use one for a specific failure pattern and define how traffic resumes, rather than adding one automatically to every dependency. Sometimes clear timeouts, bounded concurrency, and a simpler failure response are enough.

Safe recovery design

I treat recovery as part of the system's behavior. Moving failed work into a dead-letter queue preserves an opportunity to investigate and recover it; it does not establish that replaying the work is safe.

Before a replay, I want to know what changed since the original attempt. Has the underlying failure been addressed? Can the operation repeat a side effect? Are the relevant data and schema still compatible? Can this work be processed out of order, or does it depend on earlier results?

The replay procedure should identify the work to include, establish a rate the dependencies can tolerate, and specify what will be checked before increasing that rate. It also needs stop conditions: unexpected duplicates, a growing downstream backlog, or results that no longer match the assumptions behind the replay.

Idempotency and deduplication belong at the boundaries where effects occur. A duplicate message may be harmless in one part of a pipeline and dangerous at a payment, notification, or external write. I want that distinction to be explicit rather than relying on every caller to remember it.

A runbook should make these decisions usable under pressure: what to inspect, what must be true before proceeding, what action to take, and how to verify the result. I would exercise recovery with failure tests or controlled drills where practical. Those exercises can expose missing signals, unclear ownership, and procedures that are harder to execute than they looked on paper.

Changing an existing system

Modernization needs a path through the current system. I first identify which behavior must remain stable, where the important dependencies are, and what evidence would show that a change helped. A cleaner target architecture is not sufficient when the migration creates an unmanageable period of mixed behavior.

I favor changes that can be evaluated separately: clarify a boundary, make an important failure observable, isolate an expensive operation, or move a workload behind an interface. A larger redesign can still be justified, but its migration and operating costs belong in the decision.

Compatibility and rollback need specific treatment. How will old and new components coexist? What happens to in-flight work? A code rollback may not undo a schema change or an external side effect; those situations may need reconciliation or a forward fix. The point is to establish the recovery options before relying on them.

I carry those decisions into code, tests, rollout checks, and documentation. The team maintaining the system should be able to understand what changed, which assumptions remain, and what evidence would justify the next step.

Related writing and code

Read more about designing for safe recovery and the safe DLQ replay checklist.

My open-source projects explore parts of this failure-handling work in code. More articles are collected in Writing.