I don’t believe so. AES is symmetric-key encryption; if you do it incorrectly, it doesn’t decrypt at all. The only place you can mess up, then, is in the mode, which I don’t believe is accelerated by most (any?) AES encryption instruction sets; instead, they mostly handle doing an individual round of encryption/decryption.
Public key crypto systems seem like they would be much scarier to have hardware acceleration, though I’m sure if you broke it down to low level enough bits you could make it impossible for the hardware to “break” it (aside from, well, if it decided to just maliciously subsitute your code for its own. But it could do that without extension ISAs.)
I am not a cryptography expert, so please take what I say with a grain of nacl. The parameters you choose for public key crypto systems can weaken the cryptographic properties of the result. For example, nonce reuse can single-handedly destroy your security. Similarly, with RSA, padding is crucial to security. Even if you write the extension ISAs such that they never choose parameters, there’s still probably some room for their to be broken output that seems to work.
It of course depends on the random number generator. That is how you create the keys and sometimes the initialization vector. But AES itself is deterministic so it’s not possible to backdoor that part.
