# Can we learn f(x)=1/x using a neural network exactly?

Is there a way to train a neural network as $$f(x) = {1 \over x}$$ precisely?

• Sep 7, 2019 at 19:33
• On what domain? Certainly (using any of the usual activation functions) a neural network's overall function is continuous, so you can't hope to even properly approximate 1/x over an interval containing zero. I'd doubt you can get the function exactly (again using the common activators) on any interval, but I don't know how to prove that negative assertion. Sep 8, 2019 at 4:10
• The universal approximation theorem says that a neural network can approximate a continuous function on a closed and bounded set of the real line. Therefore, let’s restrict the question to a closed interval. If that interval does not contain 0, then the universal approximation theorem says that we can get arbitrarily close to $y=1/x$. If that interval does contain 0, then the function is not continuous on the whole interval, so the universal approximation theorem does not guarantee the ability to approximate at 0.
– Dave
Feb 7, 2020 at 17:18

Since the activation function itself can be an inverse function, the answer is yes.

• Is it possible to learn it with sigmoid or relu activation? Oct 8, 2020 at 11:14
• @NagabhushanSN Why do you want to learn it with a sigmoid or relu? What is special about those particular non-linear functions compared to any other non-linear function in the world? Given that the only requirement for an activation function is to be non-linear, why sigmoid or relu are revered so much? Oct 26, 2020 at 0:11
• @VladislavGladkikh because they're the most popular ones. In any network that we build, we usually have ReLU activation by default. I want to know if suppose I get to know the output is inversely dependent on a feature, should I invert the feature manually or can the network learn by itself? I've read that a neural network (even shallow) can approximate any function (theoretically), but I couldn't figure out how it can approximate the inverse function with ReLU activation. Oct 26, 2020 at 4:39
• @NagabhushanSN The Kolmogorov's theorem that you have read about says that any function of $n$ variables can be approximated with a sum of functions of 1 variable. This is how a neural network is constructed. It is a combination (a sum) of neurons (functions of 1 variable $z=\sum w_i x_i$). The theorem doesn't say which functions should be in the neurons: sigmoids, ReLu, or other. I can approximate any function with $1/x$-neurons. If your target function is similar to ReLu, you need just a few ReLu neurons to approx it. If i is very different from ReLu, you need a lot of ReLu neurons. Oct 26, 2020 at 7:34

Division by zero is not defined, so at 1/0 its not possible to find the gradient and hence the function cannot be exactly approximated by NN

• $x=0$ is not in the domain of $f(x)=1/x$, so you wouldn’t need the gradient at $x=0$, but you may need the gradient arbitrarily close to zero.
– Joe
Jun 7, 2020 at 15:38