2
$\begingroup$

Is there a way to train a neural network as $f(x) = {1 \over x}$ precisely?

$\endgroup$
3
  • $\begingroup$ Related: datascience.stackexchange.com/q/56676/64377 $\endgroup$ – Erwan Sep 7 '19 at 19:33
  • $\begingroup$ On what domain? Certainly (using any of the usual activation functions) a neural network's overall function is continuous, so you can't hope to even properly approximate 1/x over an interval containing zero. I'd doubt you can get the function exactly (again using the common activators) on any interval, but I don't know how to prove that negative assertion. $\endgroup$ – Ben Reiniger Sep 8 '19 at 4:10
  • $\begingroup$ The universal approximation theorem says that a neural network can approximate a continuous function on a closed and bounded set of the real line. Therefore, let’s restrict the question to a closed interval. If that interval does not contain 0, then the universal approximation theorem says that we can get arbitrarily close to $y=1/x$. If that interval does contain 0, then the function is not continuous on the whole interval, so the universal approximation theorem does not guarantee the ability to approximate at 0. $\endgroup$ – Dave Feb 7 '20 at 17:18
2
$\begingroup$

Since the activation function itself can be an inverse function, the answer is yes.

$\endgroup$
4
  • $\begingroup$ Is it possible to learn it with sigmoid or relu activation? $\endgroup$ – Nagabhushan S N Oct 8 '20 at 11:14
  • $\begingroup$ @NagabhushanSN Why do you want to learn it with a sigmoid or relu? What is special about those particular non-linear functions compared to any other non-linear function in the world? Given that the only requirement for an activation function is to be non-linear, why sigmoid or relu are revered so much? $\endgroup$ – Vladislav Gladkikh Oct 26 '20 at 0:11
  • $\begingroup$ @VladislavGladkikh because they're the most popular ones. In any network that we build, we usually have ReLU activation by default. I want to know if suppose I get to know the output is inversely dependent on a feature, should I invert the feature manually or can the network learn by itself? I've read that a neural network (even shallow) can approximate any function (theoretically), but I couldn't figure out how it can approximate the inverse function with ReLU activation. $\endgroup$ – Nagabhushan S N Oct 26 '20 at 4:39
  • 1
    $\begingroup$ @NagabhushanSN The Kolmogorov's theorem that you have read about says that any function of $n$ variables can be approximated with a sum of functions of 1 variable. This is how a neural network is constructed. It is a combination (a sum) of neurons (functions of 1 variable $z=\sum w_i x_i$). The theorem doesn't say which functions should be in the neurons: sigmoids, ReLu, or other. I can approximate any function with $1/x$-neurons. If your target function is similar to ReLu, you need just a few ReLu neurons to approx it. If i is very different from ReLu, you need a lot of ReLu neurons. $\endgroup$ – Vladislav Gladkikh Oct 26 '20 at 7:34
0
$\begingroup$

Division by zero is not defined, so at 1/0 its not possible to find the gradient and hence the function cannot be exactly approximated by NN

$\endgroup$
1
  • 1
    $\begingroup$ $x=0$ is not in the domain of $f(x)=1/x$, so you wouldn’t need the gradient at $x=0$, but you may need the gradient arbitrarily close to zero. $\endgroup$ – Joe Jun 7 '20 at 15:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.