3
$\begingroup$

Is there a way to train a neural network as $f(x) = {1 \over x}$ precisely?

Improve this question
$\endgroup$
3
  • $\begingroup$ Related: datascience.stackexchange.com/q/56676/64377 $\endgroup$
    – Erwan
    Sep 7, 2019 at 19:33
  • $\begingroup$ On what domain? Certainly (using any of the usual activation functions) a neural network's overall function is continuous, so you can't hope to even properly approximate 1/x over an interval containing zero. I'd doubt you can get the function exactly (again using the common activators) on any interval, but I don't know how to prove that negative assertion. $\endgroup$
    – Ben Reiniger ♦
    Sep 8, 2019 at 4:10
  • $\begingroup$ The universal approximation theorem says that a neural network can approximate a continuous function on a closed and bounded set of the real line. Therefore, let’s restrict the question to a closed interval. If that interval does not contain 0, then the universal approximation theorem says that we can get arbitrarily close to $y=1/x$. If that interval does contain 0, then the function is not continuous on the whole interval, so the universal approximation theorem does not guarantee the ability to approximate at 0. $\endgroup$
    – Dave
    Feb 7, 2020 at 17:18

2 Answers 2

Reset to default
2
$\begingroup$

Since the activation function itself can be an inverse function, the answer is yes.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Is it possible to learn it with sigmoid or relu activation? $\endgroup$
    – Nagabhushan S N
    Oct 8, 2020 at 11:14
  • $\begingroup$ @NagabhushanSN Why do you want to learn it with a sigmoid or relu? What is special about those particular non-linear functions compared to any other non-linear function in the world? Given that the only requirement for an activation function is to be non-linear, why sigmoid or relu are revered so much? $\endgroup$
    – Vladislav Gladkikh
    Oct 26, 2020 at 0:11
  • $\begingroup$ @VladislavGladkikh because they're the most popular ones. In any network that we build, we usually have ReLU activation by default. I want to know if suppose I get to know the output is inversely dependent on a feature, should I invert the feature manually or can the network learn by itself? I've read that a neural network (even shallow) can approximate any function (theoretically), but I couldn't figure out how it can approximate the inverse function with ReLU activation. $\endgroup$
    – Nagabhushan S N
    Oct 26, 2020 at 4:39
  • 1
    $\begingroup$ @NagabhushanSN The Kolmogorov's theorem that you have read about says that any function of $n$ variables can be approximated with a sum of functions of 1 variable. This is how a neural network is constructed. It is a combination (a sum) of neurons (functions of 1 variable $z=\sum w_i x_i$). The theorem doesn't say which functions should be in the neurons: sigmoids, ReLu, or other. I can approximate any function with $1/x$-neurons. If your target function is similar to ReLu, you need just a few ReLu neurons to approx it. If i is very different from ReLu, you need a lot of ReLu neurons. $\endgroup$
    – Vladislav Gladkikh
    Oct 26, 2020 at 7:34
1
$\begingroup$

Division by zero is not defined, so at 1/0 its not possible to find the gradient and hence the function cannot be exactly approximated by NN

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ $x=0$ is not in the domain of $f(x)=1/x$, so you wouldn’t need the gradient at $x=0$, but you may need the gradient arbitrarily close to zero. $\endgroup$
    – Joe
    Jun 7, 2020 at 15:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.