You know the feeling. You’re waiting for a build. The CI pipeline is red. Again. You ping the DevOps Slack channel, but it’s a ghost town. The test suite has been flaky for three weeks. Your deployment script just broke for the third time this month. And your manager? They’re asking why features are late.
No one is paged when the pipeline goes down. No one treats it like a crisis. Because it’s not production, right? It’s just the thing that makes the thing that goes to production.
That’s the lie we’ve all swallowed.
The development pipeline is the factory floor of software. When it breaks, you’re not making anything. And yet, we staff it with zero on-call, zero urgency, and zero budget.
Let me show you what happens when you get this wrong.
I worked at a company where the CI/CD pipeline was down for two days. Not a single alert went out. The engineering team sat around, ran local builds, and tried to work around a broken system. Management didn’t notice because no customer complained. But we lost 48 hours of development time across 30 engineers. That’s 60 person-days of value that never existed. The product slipped. The roadmap burned. And the root cause? A disk failure on a build server that nobody had budgeted to replace.
You’ve probably seen this too. The team that treats its internal tooling as an afterthought, then wonders why they can’t ship. The VP who greenlights a third-party monitoring tool for customer-facing systems but balks at spending $500 on a faster CI runner. The incident post-mortem that blames ‘process’ instead of the broken pipeline.
Here’s the uncomfortable truth: Companies that refuse to treat their development pipeline as a production system are implicitly admitting that their capacity to create value is less important than their capacity to serve it.
Think about it. If your customer-facing site goes down, you have a war room, a pager rotation, and a post-mortem within hours. If your pipeline goes down, you have a Jira ticket with a ‘medium’ priority and a ‘we’ll get to it next sprint’ promise. The pipeline is the engine that generates the product. The customer-facing site is just the exhaust pipe. Yet we obsess over the exhaust pipe and let the engine rust.
This isn’t a technical problem. It’s a cultural one. The mindset that says ‘production equals user-facing’ is a relic from the era when software was shipped on CDs. Today, software is continuously manufactured. The development pipeline is the assembly line. If the assembly line stops, the factory stops. Period.
I know the counterarguments. ‘We can’t page people for every flaky test.’ ‘Pipeline failures are often transient.’ ‘We don’t have the headcount.’ I’ve heard them all. But here’s the thing: you don’t need to treat every hiccup as a P0. You need to treat pipeline breakdowns as incidents. A flaky test is a bug. A broken build that blocks all deployments for an hour is an outage. A disk failure that takes down the artifact repository is a crisis. Distinguish between the two, and build the on-call and escalation paths accordingly.
One team I know did this. They set up a dedicated on-call rotation for their CI/CD infrastructure. They defined SLIs: build success rate, pipeline latency, artifact availability. They created a runbook for common failures. The first month, they had seven pages. Most were false alarms. But they fixed the underlying issues. By month three, they were down to one page per week. The team’s delivery velocity increased by 40%. Not because they worked harder, but because the pipeline stopped being a bottleneck.
Here’s the golden rule: If your pipeline breaks and you don’t wake someone up, you’re not running a software company. You’re running a lottery.
So what do you do? If you’re an engineer, start documenting the pipeline’s failure modes. Quantify the cost of downtime. Present it to your manager as a risk register. If you’re a leader, start treating internal tooling as a first-class production system. Give it a budget, a team, and an on-call rotation. Stop pretending that the factory floor is less important than the showroom.
Because the next time your pipeline goes down, your competitors will be shipping. And they won’t be waiting for a pull request.
FAQ
Q: Isn't this just a semantic argument? A pipeline outage doesn't affect users.
A: It's not semantic. A pipeline outage directly affects the ability to deliver value to users. If you can't deploy, you can't fix bugs, ship features, or respond to market changes. In a continuous delivery world, the pipeline is the production system for software itself.
Q: What practical steps can I take to get my team to treat the pipeline as production?
A: Start by measuring pipeline downtime and its cost in engineering hours. Create a simple incident response process for pipeline failures that block deployments. Assign a rotating on-call engineer for the pipeline, even if it's just during business hours. Build a runbook. Then present the data to leadership alongside the cost of inaction.
Q: Isn't this just a way to get engineers more on-call burden?
A: No. The goal isn't more pager noise — it's fewer pipeline failures. By treating the pipeline as production, you invest in reliability, automation, and tooling that reduces failures. The on-call is a temporary measure to build accountability. Done right, the pipeline becomes more stable and the on-call burden decreases.