You already know the worst kind of outage. Not the mysterious one. Not the one that takes three days and a Slack thread with 400 messages to diagnose. No — the worst kind is the one where you read the post-mortem, recognize every single step, and realize you’ve been sitting on the fix the entire time.
That’s exactly what happened to a self-hosted Garage S3 operator who recently published a corruption post-mortem that should make every single one of us uncomfortable.
The most dangerous bug in your infrastructure isn’t in the code. It’s the gap between what you documented and what you actually did.
Here’s the story in one breath: LMDB — the embedded database Garage uses for metadata storage — has a known, documented vulnerability to corruption after an unclean shutdown. This isn’t a zero-day. This isn’t a surprise. It’s been written about, warned about, and accounted for by the Garage team themselves. They built a feature specifically for this scenario: metadata_auto_snapshot_interval. One config line. One toggle. That’s all it would have taken.
The operator never enabled it.
They also had Btrfs available — a filesystem with native snapshot support — on the same machine. They didn’t configure snapshots there either.
Two safety nets. Both unused. Both sitting right there in the documentation, waiting to be switched on like a porch light that nobody ever bothered to flip.
Now let’s be honest with ourselves for a second. If you run any stateful service — and I mean anything with a database, anything that writes to disk, anything that holds state you’d cry over — you probably have a similar gap somewhere in your stack. A backup script you wrote but never automated. A replication setting you meant to enable. A snapshot interval you set to ‘later.’ Later hasn’t come yet, has it?
We don’t build fragile systems because we’re careless. We build them because we’re optimistic, and optimism is the most expensive default setting in infrastructure.
The tempting reaction here is to blame the software. ‘LMDB shouldn’t corrupt on unclean shutdown.’ ‘Garage should enable snapshots by default.’ ‘Btrfs should auto-snapshot.’ And sure — in a perfect world, all of that would be true. But we don’t operate in a perfect world. We operate in the world where power cables get kicked, where kernel panics happen at 3 AM, and where the gap between ‘documented best practice’ and ‘what’s actually running on your box’ is where outages are born.
The real failure mode here isn’t technical. It’s informational. The operator had every tool. They had the feature, the filesystem, the knowledge. What they lacked was the operational discipline to close the loop — to take what they knew and translate it into what they did. That’s not a Garage problem. That’s not an LMDB problem. That’s a human problem, and it’s everywhere.
Think about your own infrastructure right now. Not the idealized version you describe in architecture diagrams — the actual running thing. The one with the TODO comment in the config file. The one where you ‘temporarily’ disabled backups three months ago. The one where you know about a vulnerability but haven’t patched because ‘it hasn’t caused problems yet.’
Every ‘I knew about this’ in a post-mortem is a tombstone for a decision you postponed.
The Garage team did their job. They identified a known failure mode, built a mitigation, and documented it. LMDB’s behavior after unclean shutdown is well-characterized. Btrfs snapshots are a solved problem. The technology was never the bottleneck. The bottleneck was the space between reading the docs and editing the config file — that invisible stretch where good intentions go to die.
So here’s the uncomfortable takeaway, and I want you to feel it: your next outage is probably already preventable. The fix is likely already in your documentation, your config template, or a GitHub issue you starred and never revisited. The question isn’t whether you have the tools. You do. The question is whether you’ve actually turned them on.
Go check your snapshot config. Right now. Before the next unclean shutdown finds you at 3 AM with a corrupted metadata store and a sinking feeling that you’ve been here before.
The best post-mortem is the one you never have to write — because you did the boring, unglamorous work of enabling the safety net before you needed it.
FAQ
Q: Isn't this Garage's fault for not enabling snapshots by default?
A: No. Garage documented the feature, built it for exactly this scenario, and named it clearly. Defaulting to snapshots assumes storage capacity and performance tradeoffs that vary per deployment. The operator's job is to read the docs and configure for their risk profile. Blaming the software here is dodging responsibility.
Q: What should I actually do if I run Garage on LMDB?
A: Enable metadata_auto_snapshot_interval in your config. If you're on Btrfs, set up automated snapshots on the metadata partition as a second layer. Test that snapshots actually restore. Do it today, not after your next power loss.
Q: Isn't LMDB fundamentally broken if it corrupts on unclean shutdown?
A: Broken implies unexpected. LMDB's corruption behavior after unclean shutdown is documented and well-known — it's a tradeoff of its design for performance and simplicity. The contrarian take: every embedded database has sharp edges. The mature response isn't to demand they be rounded off, but to wrap them in operational safety nets you actually control.