You’ve probably been using Kafka compacted topics for years and treating them like a pure streaming mechanism. You push messages in, you read the latest state out, and you move on with your life. But here’s the thing: you’re missing half the picture.
Compacted topics aren’t a streaming pattern — they’re a distributed key-value store wearing a streaming costume.
Kafka’s log compaction is, at its core, a garbage collector. It keeps only the latest value for each key and silently discards everything else. That’s fine if you only care about the current state. But the moment something goes wrong — a config drift, a mysterious state regression, an audit request from compliance — you’re in trouble. You need to know what a key looked like three hours ago. And Kafka has already thrown that away.
Every Kafka engineer has been here. You’re staring at a compacted topic at 2 AM, trying to reconstruct what happened to a user profile or a session state, and your only options are either digging through raw log files or writing a throwaway script that consumes from the beginning and prays the retention window hasn’t expired. It’s miserable. It’s slow. And it’s completely unnecessary.
This is the tension at the heart of compacted topics: they’re designed to forget, but developers desperately need them to remember.
KGazer resolves this contradiction. It continuously consumes messages from your Kafka clusters, stores the full history in PostgreSQL, and gives you a web interface to browse keys, view message history, and compare changes over time. It honors the compacted topic’s semantics — the latest state is still the latest state — but it refuses to throw away the journey that got you there.
Every Kafka engineer has stared into the void of a compacted topic and wondered: what did this key look like yesterday?
Think about what this actually means for your workflow. If you’re using compacted topics for user profiles, configuration stores, session data, or any other application state, you’ve been flying blind whenever something goes wrong. You can see the present, but you can’t see the past. KGazer doesn’t just give you a window into your topics — it gives you a time machine.
The tool exposes something that most developers haven’t fully internalized: compacted topics have a dual nature. On one hand, they’re a streaming system — messages flow through them in real time. On the other hand, they’re a state store — the latest value per key represents the current truth. Most tooling treats them only as the former. KGazer treats them as both, and in doing so, reveals that the real value isn’t the latest state but the evolution of that state over time.
This is why the “just use Kafka” advice falls apart when you need to debug. Kafka gives you the what. It doesn’t give you the when, the who-changed-what, or the why-did-this-key-mutate-three-times-in-ten-minutes. Those questions require history, and history is exactly what compaction destroys.
The real value of state isn’t where it is — it’s how it got there.
What KGazer does isn’t revolutionary technology. It’s a PostgreSQL-backed mirror of your compacted topics with a browsing layer on top. But that simplicity is the point. The problem was never that this was technically hard to build — it’s that nobody had built it in a way that was ready to use. Every team that uses compacted topics at scale has reinvented some version of this ad hoc, with scripts that break, queries that nobody remembers, and dashboards that show the wrong thing.
KGazer replaces all of that with something you can actually use. Browse keys. View history. Compare changes. Done.
If you’re using Kafka compacted topics for any kind of application state and you don’t have a tool like this, you’re accepting a debugging nightmare that you don’t have to live with. The next time something goes wrong at 2 AM, you’ll wish you had the history. KGazer makes sure you already do.
FAQ
Q: Why not just increase Kafka's retention or write my own consumer?
A: You can crank up retention, but that burns storage and still doesn't give you a queryable history. Writing your own consumer means maintaining it, debugging it, and building a UI for it. KGazer already did all of that.
Q: What does this actually change about my day-to-day?
A: When a compacted topic's state goes wrong — a key that shouldn't have changed, a config that drifted — you stop guessing and start querying. You see the full mutation history of any key in seconds, not hours.
Q: Is this just reinventing what Kafka Streams or ksqlDB already do?
A: No. Kafka Streams and ksqlDB are for processing and transforming streams. KGazer is for inspecting and auditing state. It's not a stream processor — it's a magnifying glass for state you already have.