You know that sinking feeling. Your app starts timing out. Users are screaming. You check the logs and see a cascade of failures. So you do what any rational developer would do: you add retries. More retries. Exponential backoff. You feel clever. You feel safe.
You just turned your own system into a weapon against itself.
I watched this happen to a friend. His team had a beautiful async service. Clean code. Then a database blipped for three seconds. The retry logic kicked in—every single request retried. And retried again. Within minutes, the database was under a load ten times its normal peak. The circuit breaker? They never installed one. The outage lasted 47 minutes. The postmortem read: ‘distributed denial of service originating from our own application.’
This is the dirty secret of resilience engineering. Most developers add retries thinking they’re building robustness. But without coordinated circuit breaking, you’re not engineering resilience—you’re engineering a localized denial-of-service attack against your own backends.
Let me show you the right way.
The problem starts with how we wrap async functions. We want to abstract away the complexity of network failure. We wrap the function in a retry policy. The wrapper makes the async function look clean—no error handling clutter, no timeout logic, no exponential backoff visible in the business code. It’s beautiful. It’s elegant. And it’s dangerous.
Every retry without a circuit breaker is a love letter to your next outage.
Because the wrapper hides the complexity, you forget the wrapper exists. You add more callers. You scale. The wrapper silently multiplies every failure. The system becomes a black box of hidden interactions. The very abstraction that was supposed to make you safe becomes the source of chaos.
So what’s the fix? It’s not about removing retries. It’s about managing the propagation of failure. Resilience isn’t about preventing failure—it’s about controlling how failure spreads. That means every retry policy must be paired with a circuit breaker, a timeout, and a fallback. And the wrapper must preserve the original function signature, so the caller still sees the same contract. No hidden state. No invisible side effects.
I’ve seen a package that finally gets this right. It’s called Houhou, and it handles resilience policies for TypeScript async functions. The key insight: it preserves the original function signature and integrates AbortSignal natively. That means you can cancel a retry chain from the outside. You can coordinate across services. You can stop the storm before it starts.
But here’s the real lesson: any abstraction that makes failure invisible is a liability. The best resilience tools don’t hide the complexity—they make it explicit, controllable, and testable. They let you see the circuit breaker, the timeout, the retry count. They don’t pretend the network is reliable.
We’ve been conditioned to think that clean code means no error handling visible. That’s a lie. The cleanest code in a distributed system is the code that acknowledges the brutal reality of the network. Every call can fail. Every retry can amplify. Every wrapper can become a weapon.
So the next time you add a retry, ask yourself: am I building a shield, or am I building a bomb? If you can’t see the fuse, you’re probably wiring the wrong one.
FAQ
Q: Isn't retry logic standard practice? Why is it dangerous?
A: Retry logic is standard, but standard practice often ignores circuit breaking. Without it, retries amplify load during failures, turning a single blip into a cascading outage. The danger is that retries make the system <em>more</em> fragile, not less.
Q: What's the practical takeaway for my codebase?
A: Use a resilience library that bundles retry policies with circuit breakers, timeouts, and fallbacks—and make sure it preserves the original function signature. Houhou is one example. More importantly, test your retry logic under failure conditions. Don't assume it works.
Q: Isn't a simpler approach better than adding complex resilience layers?
A: Simplicity is a virtue, but in distributed systems, simplicity without resilience is just failure waiting to happen. The goal isn't complexity—it's <em>controlled</em> complexity. A well-designed resilience layer, like a signature-preserving wrapper, is simpler than debugging a production outage at 3 AM.