You’ve probably written a function pointer a thousand times. You pass it around, you dereference it, you call it. You think you know exactly what it is: a memory address pointing to compiled machine code. But if you’re using GNU C and you’ve ever touched a nested function, you’ve been living a lie.
A function pointer in C isn’t just an address. It’s a promise—and sometimes, a trap.
Over on Hacker News, someone recently lamented the state of modern systems programming: “Self-modifying code is cool. It’s a shame we had to disable it for security.” They were talking about nested functions in C. For decades, if you wanted a function inside a function to access the parent’s local variables, the compiler would do something illicit. It would generate executable code directly on the stack at runtime. It was a beautiful, dangerous hack.
Then came NX bit hardening. Modern operating systems looked at executable stacks and said, “Absolutely not.” Memory is either writable or executable, never both. It was a death sentence for the old way. The industry consensus? Security won, and the abstraction died.
We love to blame security for killing our favorite abstractions. But security didn’t pull the trigger; lazy engineering did.
Here is the twist: GCC didn’t kill nested functions. They didn’t throw out closures because the stack became read-only. They went back to the drawing board and redesigned how indirect calls carry the static link.
The old way was a trampoline. The compiler synthesized a tiny piece of executable code on the stack that set up the parent’s frame pointer before jumping to the real function. It was self-modifying code masquerading as a function pointer. When you passed that pointer around, you weren’t just passing an address—you were passing a dynamically generated launchpad. (And as one developer pointed out, reading the assembly to figure this out is a journey in itself—one minute you’re parsing AT&T syntax with register sigils, the next you’re completely disoriented).
When you remove the executable stack, you don’t just break a feature. You silently change the very meaning of a function pointer.
The new GCC approach proves something profound. You can keep the expressive power of nested functions without keeping the vulnerability. By redesigning the calling convention to carry the static link without runtime code generation, they tamed the illicit allure of self-modifying code into a clean, legitimate systems puzzle.
For systems programmers, this is a wake-up call. Very few codebases will ever notice this change until it breaks. You assume your tools are static, but they are shifting beneath your feet. The compiler is constantly negotiating the tension between the high-level abstractions you demand and the brutal machine reality you run on.
The best abstractions aren’t the ones that hide the machine. They’re the ones that force the machine to evolve.
Next time you dereference a function pointer, remember what you’re actually holding. It’s not just an address. It’s a history of compromises, security patches, and brilliant compiler engineering. And it’s still moving.
FAQ
Q: Why should I care about nested functions in C when I have Rust and closures?
A: Because you are still running C. Billions of lines of C code run the world's infrastructure. If the compiler changes how function pointers work under the hood, your assumptions about ABI and memory safety break silently.
Q: What's the practical implication of removing the executable stack?
A: You can finally use nested functions in GNU C on hardened platforms without triggering NX-bit violations. The expressive feature survives, but the massive security hole of runtime code generation is permanently closed.
Q: What's the contrarian take?
A: Nested functions in C were never a good idea. They are a leaky abstraction that complicates the calling convention. GCC's fix is brilliant engineering applied to a feature that should have been left in Pascal.