Out of curiosity, where does one learn about writing a video codec? Is there a canonical resource for codec implementation?

There's quite a verbose spec for all of the major codecs people talk about, but sadly it's 10k(? number I heard tossed around). ffmpeg is a great place to look, but checking out some stuff on fft is a good way to get some background knowledge.