Well considering the 4 bytes were designed in the time of a small research network between defense and universities, it's great foresight that they made it as big as it is. It still runs most of the internet to this day.
But it's not free, after all every packet carries this burden. I know about the annotation but it also makes it very difficult to parse.
That's basically no burden at all. If we cut the address length down to increase throughput, we would get a one-time increase of about 0.8% -- but consider how much faster Internet connections have gotten over the past 30 years. They've improved by about 0.8% per week on average. You're worrying about something that's smaller than one or two weeks of natural progress in Internet connection speeds
Rather than trying to minmax the address length, it makes more sense to sacrifice a few bits per packet to the addresses, wait a week or two on average to get the lost throughput back, and then spend those bits elsewhere to make other things easier. (For example, avoiding NAT is an obvious one, but even just "everything is a /64" removes the need to ever need to think about the size of a network.)
If you can spend a few address bits to make something easier elsewhere, that's a good trade. Maybe start worrying if the addresses were a kilobit long or something, but they aren't even close to that.
But it's not free, after all every packet carries this burden. I know about the annotation but it also makes it very difficult to parse.