Stepping back for a moment, while it's obvious how this might be an issue for image algorithm APIs, it's unclear to me whether this is an actual impediment to AI in the real world.
What I mean is: I have a hunch that if you were to gain control of the precise activation of each retinal photoreceptor in my eyes, you could send me into epileptic shock or induce all sorts of terrible physiological conditions. You could create retinal activation maps that, when printed out on a screen, appear like noise or normal objects, but when applied directly to photoreceptors would be interpreted by the brain to be something completely different. So I'm not sure defending against this is necessary for real-world AI, although improvements in this area will probably carry over to improvements in general performance / the theory of how deep learning works.
Your leap from DNNs to brains and "AI" is unjustified. Our brain does not work like a DNN (AFAWK), and we don't know whether DNNs are even the beginning of the path that would lead us to "true" AI some day. So I wouldn't try projecting anything from this result to our brains (which, BTW, are already known to be susceptible to illusions, but we don't know whether that mechanism is the same) or to some hypothetical future AI which we know nothing about.
I have made no such connection, and I am generally skeptical of any attempts to replicate the human brain: https://news.ycombinator.com/item?id=10920890. I'm merely pointing out the possibility this isn't a huge problem to developing robust AI, because my educated guess is humans are similarly vulnerable.
Adding random noise doesn't help a lot, because it's random. Mean 0. This technique relies on changing each pixel in just the right direction to affect its output. Random noise pushes half the pixels back to the original direction and the other half even further in the wrong direction.
It's unclear if viewing the image at different scales or from different angles would help. Likely it would. However NNs are designed and trained to be as invariant to deformation as possible.
A more interesting domain is audio. Audio doesn't have any concept of different rotation or scale. It's guaranteed each bit will be input in the correct order. Its also a recurrent system, which is vulnerable to falling into states of chaotic behavior.
So if humans are vulnerable to these things, it will likely be in the form of audio.
What's new in this paper is that they do not have access to the neural net activations. They need only send it images, and get back the class the nnet predicts.
But argonaut wasn't talking about having access to the neural net activations, but about having access to individual pixels in the image (which corresponds to individual photoreceptor cells in the human retina). Crafting these NN-defeating images seems to depend on tweaking individual pixels.
I'm not sure how far it really does depend on that -- whether, e.g., you can use a more sophisticated version of the same approach to make an image that fools the NN reliably even if it's slightly offset, out of focus, stretched, etc. I wouldn't be astonished if you could. If not, that suggests an obvious defence mechanism: feed the NN not the raw image but several slightly perturbed ones, and take a majority vote or something.
It's also worth observing that there are attacks on the human visual system, found by trying things out and observing their effects. We call them optical illusions, and some of them are very convincing.
If you like that kind of thing I recommend Eclipse Phase (which calls this idea a "Basilisk attack"). In the same general area, Blindsight by Peter Watts involves human perceptual flaws as a plot element.
Yay, good reference, I remember the story so well I knew the one you meant before I followed the link (I didn't remember the title).
However there have been similar ideas in earlier works, as acknowledged by the author himself. And if you asked the authors of those earlier works I'm sure they'd say they first thought of it when reading someone else's work - it's how it goes.
For what it's worth, the idea is similar to the deadliest joke sketch from the Monty Pythons, that predates most (but not all) the authors cited by Langford as influences:
Langford's later short story comp.basilisk FAQ, [1] first published in Nature in December 1999, mentions William Gibson's Neuromancer (1984), Fred Hoyle's The Black Cloud (1957), J.B. Priestley's The Shapes of Sleep (1962), and Piers Anthony's Macroscope (1969) as containing a similar idea. Examples not mentioned include the short story White Cane 7.25 (1985) by Czech writer Ondřej Neff, A. E. van Vogt's War Against the Rull (1959), and John Barnes' Kaleidoscope Century (1996).
What I mean is: I have a hunch that if you were to gain control of the precise activation of each retinal photoreceptor in my eyes, you could send me into epileptic shock or induce all sorts of terrible physiological conditions. You could create retinal activation maps that, when printed out on a screen, appear like noise or normal objects, but when applied directly to photoreceptors would be interpreted by the brain to be something completely different. So I'm not sure defending against this is necessary for real-world AI, although improvements in this area will probably carry over to improvements in general performance / the theory of how deep learning works.