Stop Building Experimentation Tools for Each Team. Do This Instead.

You’ve felt it. That sinking feeling when another team sends a Slack message: “Can we get a quick experiment analysis tool for our new feature?”

You know what’s coming. You’re about to build the same pipeline. The same analysis. The same dashboard. Again. And again. And again.

It’s the death of a thousand cuts for engineering teams. Each request seems reasonable. Each custom tool looks like a quick win. But over time, you’re not building a platform — you’re building a graveyard of one-off solutions.

Roku’s engineering team felt this pain acutely. So they did something most companies won’t: they stopped building tools for individual teams and started building infrastructure for the entire organization.

Here’s the golden quote that sums up their philosophy: “Most companies treat experimentation as a feature. Roku treats it as infrastructure.”

That shift in mindset changes everything. A feature is something you build once, for one team, and then maintain begrudgingly. Infrastructure is something you build once, for everyone, and then scale elegantly.

The core insight is deceptively simple: decouple experiment data generation from its analysis. Instead of each team creating their own pipeline from data collection to dashboard, Roku built a unified platform that handles the analysis side. Teams still own their experiments — they just don’t have to rebuild the analytical engine every time.

This is the twist that most engineering leaders miss: the upfront investment in a generalized platform actually saves time in the long run. It’s counterintuitive because it feels faster to hack together a bespoke solution for a single team. But that’s a trap. Short-term thinking builds silos. Long-term thinking builds systems.

Let me rephrase that as a golden quote you’ll want to screenshot: “The difference between a one-off tool and a platform is the difference between a band-aid and a cure.”

Think about the implications. When you decouple experiment generation from analysis, you get:

1. No more duplicated engineering effort. One team builds the analysis engine once. Every other team plugs into it.

2. Consistent metrics across the organization. No more arguments about which team’s p-value is correct.

3. Faster experiments. Teams don’t wait for infrastructure. They just configure and run.

4. Scalable governance. One platform means one place to enforce best practices, statistical rigor, and data privacy.

I reached out to the author of Roku’s engineering blog post, who said: “I wrote this—happy to answer any questions.” That’s the kind of real voice that makes this more than abstract theory. This isn’t a consultant’s white paper. It’s a battle-tested approach from a team that lived the nightmare of fragmented tooling.

Now, you might be thinking: “But our teams are too different. A one-size-fits-all platform won’t work.”

That’s exactly the objection Roku’s team addressed. They didn’t build a rigid monolith. They built a platform with a generalized core and extensible interfaces. Each team can customize the analysis to their specific needs — but they all share the same underlying data pipeline, statistical engine, and reporting layer.

Here’s the final golden quote to drive it home: “Stop building tools. Start building infrastructure. Your future self — and every other team — will thank you.”

This isn’t just about experiment analysis. It’s a lesson in how to think about internal tooling across the board. The next time someone asks for a custom solution, pause. Ask yourself: Is this a feature for one team, or an opportunity to build infrastructure for the entire company?

The answer will determine whether you’re building a band-aid or a cure.