You know that sinking feeling. You’ve just pushed a change to your data pipeline — a single field added to a massive JSON object. Now you’re staring at a progress bar that’s crawling at the speed of a wounded snail. Everything is recomputing. Everything. Again.
It’s not your fault. It’s the architecture you inherited. But the engineers at Jane Street have been laughing at this problem for years — and they built a framework called Incremental that makes your compute-heavy workflows feel like a cheat code.
Here’s the core idea: Don’t recalculate what hasn’t changed. Sounds obvious, right? Except every major system I’ve seen — from financial models to recommendation engines — happily recomputes entire graphs on every new input. It’s the software equivalent of rebuilding the entire house every time you want to change a lightbulb.
Incremental flips that. It tracks every node in your computational graph, detects exactly which parts changed, and only recomputes the downstream dependencies that are actually affected. The result is a performance gain that can be measured in orders of magnitude, not percentages.
Here’s where it gets spicy. Jane Street built this in OCaml — a language that famously struggles with runtime introspection. The common wisdom is that you need dynamic languages like Python or JavaScript to do this kind of change-detection magic. But Jane Street’s team proved that the real bottleneck isn’t language flexibility — it’s your ability to statically guarantee system stability at scale. OCaml’s type system becomes a superpower, not a limitation.
I’ve seen firsthand how this plays out in high-frequency trading. A single market data tick can cascade through millions of calculations. Without Incremental, you’re either sacrificing accuracy for speed or building a fortress of caching logic that’s brittle and unmaintainable. With it, you get near-instantaneous propagation with zero wasted compute.
Now, you might be thinking: ‘This sounds like an observable pattern. I’ve seen that before.’ And you’re not wrong. But the devil is in the details. Jane Street’s implementation goes far beyond simple publish-subscribe. It handles complex DAGs, detects cycles, and does it all with a level of performance that makes your average reactive framework look like a toy.
So what can you take away from this? Three things:
First, stop optimizing hot paths until you’ve fixed the cold dead ends. Most teams spend weeks micro-optimizing a single function while ignoring the 90% of their pipeline that’s recomputing identical results over and over.
Second, language choice matters, but not for the reasons you think. OCaml’s static guarantees enable Jane Street to build a system that would be a nightmare to maintain in a dynamically typed language. The lesson is: pick the language that lets you eliminate entire categories of runtime bugs, not the one that lets you prototype fastest.
Third, incremental computation isn’t just a performance trick — it’s a philosophy. It forces you to think about what your system actually needs to know, not what it can compute. When you internalize that, your architecture becomes leaner, faster, and infinitely more maintainable.
Jane Street’s Incremental isn’t just a library. It’s a challenge to every engineer who’s ever accepted ‘it’s a bit slow’ as a fact of life. The next time you see a progress bar crawling, ask yourself: did I really need to recompute all of that? Or did I just need to check my lightbulb?
FAQ
Q: Isn't this just reactive programming or an observable pattern?
A: Superficially, yes. But Incremental goes deeper — it handles complex DAGs with cycles, provides change detection at the node level, and is optimized for the extreme latency requirements of high-frequency trading. Most reactive frameworks are linear or tree-based; Incremental handles arbitrary graphs efficiently.
Q: What's the practical implication for my team's codebase?
A: If you have any data pipeline that recomputes from scratch on every input — even a simple one — you're leaving performance on the table. Start by identifying the parts of your system that are 'pure' (same input always produces same output) and can be memoized or diffed. Even a basic incremental update can slash compute time by 90%.
Q: But isn't OCaml a niche language? Why should I care?
A: The language choice is the provocative part. Jane Street showed that static typing can be a superpower for building reliable incremental systems. The lesson isn't 'use OCaml' — it's that the runtime introspection you think you need (Python's type inspection, JS's proxy objects) isn't free. It comes with complexity and fragility. Sometimes the best tool is the one that forces you to be explicit about what changes.