No, the post is correct about MQA. A KV-cache only caches the key and value heads. The point of MQA is that your KV-cache is 1/heads smaller than usual because of this sharing.
Having multiple query heads does not affect the cache size, which is the limiting factor in MHA decoding for both memory capacity and bandwidth reasons.
>Autoregressive decoder inference is a severe bottleneck for Transformer models due to the memory bandwidth overhead from loading decoder weights and all attention keys and values at every decoding step (Shazeer, 2019; Pope et al., 2022; de Jong et al., 2022). The memory bandwidth from loading keys and values can be sharply reduced through multi-query attention (Shazeer, 2019), which uses
multiple query heads but single key and value heads.
FWIW the original MQA paper is called One Write head is all you need.
Here's the quote from that referencing multiple heads [1]
>We propose a variant called multi-query attention, where
the keys and values are shared across all of the different attention "heads", greatly reducing the size of
these tensors and hence the memory bandwidth requirements of incremental decoding. We verify experimentally that the resulting models can indeed be much faster to decode, and incur only minor quality degradation from the baseline.
Having multiple query heads does not affect the cache size, which is the limiting factor in MHA decoding for both memory capacity and bandwidth reasons.