Source: Michael Nielsen blog, Jan 2016
universality tells us that neural networks can compute any function; and empirical evidence suggests that deep networks are the networks best adapted to learn the functions useful in solving many real-world problems.
One of the most striking facts about neural networks is that they can compute any function at all. That is, suppose someone hands you some complicated, wiggly function,:
No matter what the function, there is guaranteed to be a neural network so that for every possible input,, the value (or some close approximation) is output from the network, e.g.: