I'm pretty sure that nobody would have rejected the RSA paper with a note that argued that there was some 32-bit limit. If I've got my history correct, the Diffie-Hellman _New Directions in Cryptography_ paper had already been published, and it already used the same basic mathematical operation (exponentiation modulo a large number). Everyone knew that, sooner or later, a full public-key encryption algorithm would be found.