6
$\begingroup$

I read somewhere on the StackExchange that a neural network can't approximate the Pi number as a function of circles length and radii. Maybe this is an incorrect example or wrong information. Help me to understand.

What about the sum or multiplication of any arbitrary numbers? Are there any other specific functions neural networks can't approximate or not?

$\endgroup$
  • $\begingroup$ It was not in the book or article. I think it was on StackExchange, but now I can't find it. Maybe to say "I know" was wrong. So, if this possible than for me it is also valuable information. $\endgroup$ – CyberDreamer Apr 4 '17 at 11:08
  • 1
    $\begingroup$ I edit my question that don't be misleading $\endgroup$ – CyberDreamer Apr 4 '17 at 11:11
  • $\begingroup$ Above it is said "A neural network can approximate any function, provided it has at least one hidden layer and uses non-linear activations there. This has been proven by the universal approximation theorem." Please be more mathematically precise and accurate. The hype behind NNs has blinded many people. NNs can learn any CONTINUOUS function (barely, with many assumptions!) but they CANNOT learn non-continuous functions, and many (if not most) functions are not continuous $\endgroup$ – Walid Saba Jan 4 '18 at 2:39
14
$\begingroup$

A neural network can approximate any continuous function, provided it has at least one hidden layer and uses non-linear activations there. This has been proven by the universal approximation theorem.

So, there are no exceptions for specific functions. You ask:

I read somewhere on the StackExchange that a neural network can't approximate the Pi number as a function of circles length and radii.

A neural network to approximate $\pi$ is very easy. Possibly what you read is that a neural network cannot generate new digits of $\pi$ that it has not already been shown. More on that later . . .

What about the sum or multiplication of any arbitrary numbers?

Yes, a neural network can approximate that.

Are there any other specific functions neural networks can't approximate or not?

No, there are no specific functions that a neural network cannot approximate.

However, there are some important caveats:

  • Neural networks do not encode the actual functions, only numeric approximations. This means there are practical limits on the ranges of inputs for which you can achieve a good approximation.

  • A neural network being able to approximate a function in theory is not the same thing as you or I being able to construct a neural network that approximates that function. There is no known method to construct a neural network by analysis of a function alone (it can be done for specific simple functions such as xor).

  • The usual way to achieve approximation is to train a neural network by giving example data. The network will approximate to data it has been shown. There is no guarantee that this will generalise to new inputs that it has not been trained on and approximate the correct outputs. In fact for certain types of input/output it cannot possibly do so. For instance, it will not learn how to generate the 4th digit of $pi$ if it has been shown digits 1,2,3,5,6,7,8,9. The best generalisation results occur for functions that have smooth transitions between the training examples.

  • Neural networks do not extrapolate well to inputs outside of the data they have used for training. They "fit" to the training data (imagine a rubber sheet draped over all the points in the training set).

  • Neural networks do not learn to copy algorithms, only functions. So if you take a complex algorithm, such as AES encryption, and attempt to train a neural network to perform this given lots of input examples, has no real chance of working. Now, AES encryption can be considered a function e.g. $output = encrypt( input, key )$. So the NN can approximate it. But it will only do so for the specific inputs and outputs it has been shown. In addition AES does not respond well to approximation - a single bit wrong will cause it to be a bad encryption. So you won't see NNs used to encrypt or decrypt in cryptography.

  • The capability of a neural network to approximate is limited by the number of neurons and connections it has. More complex functions require larger networks. In order to train a larger network on a more complex function takes more time and more training data. You could in theory train a neural network to learn a random number generator function. However, that would take an impossible amount of resources - memory to store the network, and time to train it against the whole output of the RNG.

$\endgroup$
  • $\begingroup$ Thanks for your nice answer, take also this in mind. $\endgroup$ – Media Mar 31 '18 at 19:38
2
$\begingroup$

Not sure what you mean by NN not approximating $pi$. NN can approximate any continuous function (provide it contains enough nodes, one layer suffices). See Universal_approximation_theorem

$\endgroup$
  • $\begingroup$ That should be "one hidden layer" suffices I think, otherwise some nonlinearities cannot be approximated. $\endgroup$ – Neil Slater Apr 4 '17 at 7:08
  • $\begingroup$ Is the Pi function a continuous function and finite function? May be i don't understand mathematics $\endgroup$ – CyberDreamer Apr 4 '17 at 10:56
  • $\begingroup$ $\pi$ is a constant, not a function. What is this "Pi" function? $\endgroup$ – Emre Apr 6 '17 at 7:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.