You’ve been told that government data is open. That it’s free. That democracy thrives on transparency. I’m here to tell you that’s a beautiful story we tell ourselves while the actual data sits behind a wall that only coders can climb.
A developer just scraped 12.4 million US business registration records from state open-data portals. Free. All of it. Every state portal, every LLC filing, every corporate registration — sitting right there on government websites, waiting to be downloaded. Except most people can’t download them. Not because there’s a paywall. Not because you need special clearance. Because you need to write code.
The new digital divide isn’t about who can afford data. It’s about who can scrape it.
The scraper? About 300 lines of Python. Standard library. Nothing fancy. But that’s 300 lines between you and a dataset that could reshape your market research, your competitive analysis, your investment thesis. Three hundred lines that most business analysts, policy researchers, and journalists will never write.
Think about what that means. A hedge fund quant with Python skills pulls 12.4 million business registrations over a weekend. A seasoned market researcher with 20 years of experience but no coding background? They get nothing. They don’t even know the data exists, because the portals are designed by engineers for engineers.
Here’s where it gets worse. The developer noted that Delaware — DELAWARE, the state where more than half of Fortune 500 companies are incorporated — offered zero bulk access at any price. Not free, not paid, not at all. So the most important business registry in America is effectively a black box, and nobody’s talking about it.
Open data isn’t open if you need a computer science degree to read it.
This is the paradox that should keep every open-government advocate awake at night. We’ve spent a decade fighting for data transparency, winning that fight, publishing millions of records on public portals — and then quietly accepting that only a tiny technical elite can actually use any of it. We replaced one gatekeeper (the government clerk) with another (the person who knows how to handle rate limits, pagination, and malformed JSON responses).
If you work in business intelligence, this is either terrifying or exhilarating. Terrifying because your competitors might already be scraping these portals while you’re paying six figures for third-party data that’s less complete. Exhilarating because now you know the gold mine exists — you just need someone who can write 300 lines of Python to claim it.
The real bottleneck in the information economy has shifted. It’s no longer access. It’s extraction. The data is free. The ability to get it is not. And that gap — between the promise of open government and the reality of practical inaccessibility — is where a new kind of inequality is being forged.
Every dataset that’s free but unreadable is a quiet transfer of power from the public to the technically literate.
So the next time someone celebrates open data as a democratic victory, ask them one question: how many people in your city can actually use it? The answer will tell you whether transparency is real, or just a very convincing performance.
FAQ
Q: If the data is truly public, isn't this just a skills issue, not an access issue?
A: That's exactly the trap. 'Public' and 'accessible' are not the same thing. A document locked in a filing cabinet in a government building is also public — you just need to drive there during business hours. We wouldn't call that open access. If you need to write a scraper, handle rate limits, and parse inconsistent JSON to get data that's supposedly 'open,' then it's open in name only.
Q: What should businesses actually do with this information?
A: If you're in market research, competitive intelligence, or investment analysis, stop overpaying for third-party business datasets and hire someone who can scrape these portals directly. The raw data is more current, more complete, and costs nothing but engineering time. The ROI on a junior developer who can write scrapers will dwarf your Bloomberg subscription.
Q: Isn't this just how technology works — early adopters always have an edge?
A: No, because this is government data. The whole point of public records is democratic access. When a private company builds a better tool, that's capitalism. When the government publishes data that only coders can use, that's a structural failure. The state has an obligation to make public records usable by the public — not just the subset of the public that knows Python.