I have been going through this book (https://www.cs.toronto.edu/~graves/preprint.pdf) on sequence labelling by Alex Graves. On page 29, under the section of input representation he states that the performance of neural networks are almost similar over a wide range of input representations. If some of you can put some light on this paragraph from the book and explain why is it that neural networks are relatively robust to input data representations?

Thanks! Image of the paragraph

  • $\begingroup$ Can't post answer because your post is on hold. Neural networks colligate information via several passes (several layers) as it flows through the network. It just so happens that after generalizing such an information again and again, the final layers in the network containing the same intermediate values, even if your input was represented in a somewhat different pattern than the usual one. Assume you teach a child that any time he sees ${x, y, z}$ he should produce $x*y*z$. He will figure out that even if its {z,y,x} it should still be the same rule for the output, since we keep telling him $\endgroup$ – Kari Jun 10 '18 at 11:19