You know the feeling. It’s 4 PM on a Friday. You push a commit, the CI pipeline goes red. You rerun the build. It goes green. You roll your eyes, blame the infrastructure, and clock out.
We’ve been trained to hate flaky tests. They are the mosquitoes of the software development lifecycle—annoying, seemingly useless, and universally targeted for extermination. We quarantine them, rewrite them, or just delete them to keep our dashboards green.
But what if I told you that the exact thing making you tear your hair out is the only thing keeping your production environment from catching fire?
Recently, the engineering team at Buildkite encountered a textbook flaky test. It failed intermittently. It would have been incredibly easy to slap a ‘quarantine’ label on it, ignore the noise, and move on to the next sprint. But someone dug deeper.
The crash wasn’t a timing issue. It wasn’t a poorly mocked dependency. It was a use-after-free bug in a Redis client. A rare, latent memory corruption issue that was silently waiting to take down production servers under the right load.
A deterministic test is a lie we tell ourselves to feel safe in a non-deterministic world.
Use-after-free bugs are absolute nightmares. They don’t crash every time. They depend on the exact state of memory allocation at the exact millisecond of execution. Your pristine, perfectly orchestrated unit tests will never catch this. Your expensive static analysis tools will shrug and look the other way.
But a flaky test? A test that runs at slightly different times, with slightly different memory states and garbage collection pauses? It trips the wire. The randomness of the test suite perfectly mirrors the randomness of production traffic.
The noise you are desperately trying to silence might be the only signal warning you of an impending meltdown.
We treat flaky tests as an annoyance, a sign of poor engineering hygiene. We need to flip the script. That unpredictability isn’t a bug in your test suite—it’s a feature. It is a stress test you didn’t know you were running.
If you aggressively hunt down and silence every intermittent failure without understanding the root cause, you aren’t cleaning up your codebase. You are dismantling your early-warning systems. You are blinding yourself to the exact edge cases that cause 3 AM outages.
Next time a test fails on run #4 but passes on run #5, don’t just roll your eyes and hit ‘rerun’. Look closer. The randomness that frustrates you might be the exact stress test your memory management needs.
FAQ
Q: Isn't a flaky test just a sign of bad test design?
A: Often, yes. But if you blindly quarantine every intermittent failure, you're throwing the baby out with the bathwater. The randomness might be exposing a real concurrency or memory bug that a perfectly deterministic test could never catch.
Q: What's the practical implication here?
A: Stop blindly rerunning failed tests. When a test fails intermittently, investigate the memory state, race conditions, and concurrency issues before assuming it's just a flaky test.
Q: What's the contrarian take?
A: Flaky tests are actually better than deterministic ones. They simulate the chaotic reality of production traffic, catching non-deterministic memory bugs that your perfect, predictable unit tests will always miss.