The main problem with common VMs for multiple existing languages is that often the semantics of the language are heavily coded into the VM. For example, Parrot's core opcodes mimic Perl's behavior of "Type coercion for everyone, the string "0" is false!" There are a lot of other issues as well, and while they are certainly solvable, the resulting VM would be very heavyweight. If you want to design a VM that runs multiple languages, often you have to design the languages around the VM.
Well, at some point you might run out of space in the opcode byte (or multiple bytes), and then you need longer sequences that put up more cache pressure and require more time decoding, right?
Also, the developers time budget and the VMs complexity migth keep you from optimizing all possible paths.