That's pretty depressing to read. x264 was handling the encoding side with sub-frame latency 15 years ago, and sub-frame decoding is significantly easier. "with –tune zerolatency, single-frame VBV, and intra refresh, x264 can achieve end-to-end latency (not including transport) of under 10 milliseconds for an 800×600 video stream"
But for some reason you can't make use of that and have to burn bandwidth instead.
In theory it's a small part. But if you got that many frames of latency difference by changing codec, then it wasn't being a small part.
It's not that you should have gotten a magical 10ms latency glass to glass, it's that you should have been able to get 4 frames latency on h.264. But something prevented that, so I'm sad about it.
(And if you say the bandwidth was fine in your situation I won't argue, but using more than a gigabit extra is not usually thought of as free.)
But for some reason you can't make use of that and have to burn bandwidth instead.