Stop Buying Expensive Databases. S3 Already Is One.

You’ve probably been told that databases are special. That they need specialized storage, bespoke infrastructure, replicated disk arrays whispering to each other across data centers. You’ve probably also paid handsomely for that belief.

What if I told you the cheapest, dumbest storage on AWS — the one you use to dump log files and cat photos — can be repurposed into a strongly consistent, crash-safe, transactional database?

The database industry has spent two decades convincing you that storage is smart. The truth is, the intelligence was always in the layers above it, and object storage was never as dumb as you thought.

Here’s the setup: S3 is object storage. It has eventual consistency (historically). It doesn’t have transactions. It doesn’t have WAL. It doesn’t have rows. It’s a place where you PUT a blob and GET a blob. That’s it. And yet — a team of engineers built a fully functional key-value database on top of it in Go, using nothing but LSM trees, immutable SSTables, conditional commits, and fencing tokens.

Let me walk you through why this is not a toy project, and why it should make you question every database bill you’ve ever paid.

The Twist: S3’s Limitations Are Actually Features

The conventional wisdom says: to build a database, you need fast local disks, a write-ahead log, and a storage engine that understands pages and blocks. S3 violates every one of these assumptions. It’s a network filesystem with high latency and no guarantee of immediate consistency.

But here’s the twist — when you stop fighting S3’s limitations and start designing around them, those same limitations become architectural guarantees.

S3 is immutable once written? Perfect — that’s exactly what LSM-tree SSTables need. S3 objects are content-addressable? That’s a built-in deduplication layer. S3 has conditional writes now? That’s your compare-and-swap primitive for transactional fencing.

The engineers behind this project didn’t try to make S3 behave like a local disk. They treated S3 as what it actually is: a massively durable, infinitely scalable, eventually consistent append-only log. And then they built the database on top of that abstraction.

How It Actually Works (Without the Hand-Waving)

The architecture is an LSM tree — the same structure behind RocksDB, Cassandra, and HBase. Writes go into an in-memory memtable. When the memtable fills up, it’s flushed as an immutable SSTable to S3. Reads check the memtable first, then walk down through SSTables from newest to oldest.

Compaction merges older SSTables into fewer, larger ones — the same garbage-collection problem every LSM database solves. The difference is that here, the “disk” is S3, and the compaction strategy has to account for S3’s latency profile and per-request cost model.

Every database is just a sorting problem with a durability problem. S3 solves the durability problem so completely that you’re free to obsess over the sorting.

The hard part — the part that makes this real and not a hack — is crash safety. What happens when a write fails mid-flight? What happens when two compactions race? This is where conditional commits and fencing tokens come in. Every SSTable write is guarded by a conditional check: “only write this if the previous version is X.” Fencing tokens ensure that a stale writer can’t clobber a newer write after a crash and recovery.

This is not a trick. This is distributed systems engineering applied to a substrate that most people dismiss as too slow or too dumb.

The Latency Trade-Off (Let’s Be Honest)

Yes, S3 is slower than local NVMe. That’s not a secret, and pretending otherwise would be dishonest. A read that hits multiple SSTables on S3 will have latency in the tens to hundreds of milliseconds, not microseconds.

But here’s the question you should be asking: for your workload, does latency actually matter, or have you been optimizing for a number that nobody cares about?

If you’re building a billing system that needs sub-millisecond reads, this isn’t for you. If you’re building a system that needs 11 nines of durability, horizontal scalability to petabytes, and operational simplicity — where a single S3 GET taking 50ms is perfectly acceptable — then you’ve been overpaying for infrastructure your entire career.

The caching layer handles hot keys. The compaction strategy keeps the SSTable count manageable. The conditional writes handle correctness. Everything else is just S3 being S3 — which is to say, absurdly durable and absurdly cheap.

Why This Matters Beyond S3

Here’s the bigger picture. This project isn’t just about S3. It’s about a principle that applies to every layer of your stack: the line between “storage” and “database” is not a property of the storage layer — it’s a property of the abstractions you build above it.

Most engineers treat infrastructure categories as fixed: this is a database, this is a queue, this is object storage, this is a cache. But the best engineers — the ones who build things that scale without breaking — understand that these categories are conventions, not laws of physics.

A queue can be a database (Kafka streams). A database can be a cache (Redis). And object storage can be a database — if you’re willing to do the work of building the LSM tree, the compaction strategy, and the fencing protocol that makes it safe.

The Go concurrency model makes this particularly elegant. Goroutines handle parallel compaction. Channels coordinate the memtable flush pipeline. The concurrency primitives in Go map almost perfectly onto the concurrency requirements of an LSM database — which is probably why this implementation feels clean rather than tortured.

The Real Question

So here’s what I want you to walk away with: next time you’re evaluating a database, don’t ask “which database should I use?” Ask: “what invariants do I actually need, and what’s the cheapest substrate that can enforce them?”

The most expensive engineering decision is the one you make by default, without questioning it. And the database industry has been counting on you not asking questions.

S3 as a database isn’t the right answer for everything. But it’s proof that the right answer is almost never the expensive default — it’s the one you build when you understand your problem deeply enough to stop paying other people to solve it for you.