3
$\begingroup$

How do I capture $y = x_1 x_2$ using a simple neural network with commonly used activation functions? I assume that I need at least one hidden layer. What mix of commonly activation functions should I use?

So far, I have used $max(0,x)$ and $tanh$ activation for the hidden layer, but the gradient descent diverges very quickly.

Some thoughts which may be useful:

$$ (x_1 y_1)^2 = \exp(\log(x_1^2) + \log(x_2^2)) $$

Improve this question
$\endgroup$
2
  • $\begingroup$ Welcome to DataScience.SE! Is this a homework question? $\endgroup$
    – Emre
    Jul 18, 2016 at 19:34
  • $\begingroup$ no. i'm just playing around. I'm a NN newb, so I'm not sure if this sort of thing is in standard texts. $\endgroup$
    – nbren12
    Jul 18, 2016 at 19:49

2 Answers 2

Reset to default
3
$\begingroup$

Probably you need to do one or more of:

  • Decrease learning rate. Diverging loss is often a symptom of learning rate too high.

  • Increase number of hidden neurons. The output function will be the combination of many "patches" each created by a neuron that has learnt a different bias. The shape and quality of each patch is determined by the activation functions, but almost any nonlinear activation function used in an NN library should work to make a universal function approximator.

  • Normalise inputs. If you are training with high values for $x_1$ or $x_2$, this could make it harder to train unless you normalise your training data.

For your purposes, it might be an idea to skip the need for normalisation by training with $x_1$ and $x_2$ in range -2.0 to 2.0. It doesn't change your goal much to do this, and removes one potential problem.

You should note that a network trained against an open-ended function like this (where there are no logical bounds on the input params) will not learn to extrapolate. It never "learns" the function itself, but an approximation of it close to the supplied examples. When you supply a $(x_1, x_2)$ input far from the training examples, the output is likely to completely mismatch your original function.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thanks for your answer. I assume the "activation function" for the output layer just be the identity? $\endgroup$
    – nbren12
    Jul 18, 2016 at 21:22
  • $\begingroup$ @nbren12: Yes for regression you would normally just have linear output layer, no activation function. $\endgroup$
    – Neil Slater
    Jul 18, 2016 at 21:23
  • $\begingroup$ all your points are well taken, thank you. Is there any good literature on using NNs for regression rather than classification? most of the tutorials online are about classifying MNIST digits or doing super simple linear regression. For instance, when can a multilayer network with nonlinear activation functions do a decent job fitting a linear function? $\endgroup$
    – nbren12
    Jul 18, 2016 at 22:52
  • $\begingroup$ @nbren12: If you have a linear function to fit, you would probably use simple linear regression to fit it - or you could use a single layer network with no activation function, which is equivalent. NNs work OK at regression, just use linear output layer and suitable objective function such as mean square error. Other aspects of any tutorial would remain the same. You still need to worry about over-fitting for example, and search for good architecture and other meta-params, and there is a bewildering array of clever designs/choices to learn about. $\endgroup$
    – Neil Slater
    Jul 19, 2016 at 6:34
  • $\begingroup$ I was just thinking that the most basic test of any regression platform is linear regression, so any fancy nonlinear regression model (e.g. neural network) should, at least, be able to handle linear functions. thanks for all your comments. $\endgroup$
    – nbren12
    Jul 19, 2016 at 16:06
1
$\begingroup$

To add onto the above answer, a simple feedforward network doesn't learn the function itself. But recently, Neural Turing Machines claim to learn the algorithm. Worth a shot!

Improve this answer
$\endgroup$
4
  • $\begingroup$ There is no time (or steps) in this "algorithm". It is a simple algebraic expression, so the paper is inapplicable. But thanks for the reference! The paper was fun to read! $\endgroup$
    – Ricardo Magalhães Cruz
    Jul 19, 2016 at 8:22
  • $\begingroup$ is this just an arxiv paper? $\endgroup$
    – nbren12
    Jul 19, 2016 at 16:07
  • $\begingroup$ yes, there is an arxiv paper by Alex Graves et al $\endgroup$
    – Hima Varsha
    Jul 19, 2016 at 16:08
  • $\begingroup$ interesting paper. i do have to agree with ricardo. i'm just trying to learn a function of known inputs, there is no dynamical behavior, so the paper isn't applicable. $\endgroup$
    – nbren12
    Jul 19, 2016 at 16:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.