This is the sort of thing we do in HFT to get single instruction calls and save nanos in places that get called many many thousands of times a second.

I mean, at some point, why not just write a custom compiler?

If you care that much, you use intrinsics, asm in the C file, or asm to create an object you call into.

Most of the code doesn't need nearly that level of optimization, so you write it in a higher level language.

Making your own compiler is a lot of work compared to just doing what I've outlined above. So you don't. I's not worth it, and you'd still end up back where you are.

Writing assembly really isn't that scary, especially in small doses for performance-critical code. That was a very common practice in the 80s/90s, though it's faded as compiler optimizations have improved.