4
$\begingroup$

I posted this question on Cross Validated before I realized that this existed. I think it is better suited here and got no answers over there so I have deleted other post. I have reproduced the question below:

I have been playing around with the neural network toolbox in MATLAB to develop an intuition for how the architectural requirements scale with feature dimension.

I put together a simple example, and the results have surprised me. I am hoping someone can point to either (a) an unrealistic expectation of mine, or (b) a mistake/misuse of the neural network toolbox.

The example is as follows: I have a simple un-normalized one-dimensional Gaussian that I am trying to learn. I do the following:

x = -5:0.2:5;
y = exp(-x.^2/2);

net = feedforwardnet(2);
net = configure(net, x, y); 
net = train(net, x, y); 
y2 = net(x); 

plot(x, y, 'o', x, y2);
legend('Data', 'NN');

This gives me good results. I get the plot below.

ANN 1D

Now, I try to extend this to 2 dimensions and this is where I run into trouble. I don't think I'm asking too much. My data is not noisy, or is it sparse. I figure if I double the number of neurons that should be sufficient for an increase in dimensionality. Here's my code:

x1 = -5:0.2:5;
x2 = -5:0.2:5;
[x1g, x2g] = meshgrid(x1, x2);
xv = [x1g(:)'; x2g(:)'];
yv = exp(-dot(xv,xv)/2);

net = feedforwardnet(4); 
net = configure(net, xv, yv);
net = train(net, xv, yv);
y2v = net(xv);  
plot3(xv(1,:), xv(2,:), yv, xv(1,:), xv(2,:), y2v, 'o');
legend('Data', 'NN');

The plot I get is this:

NNet 2D

This is pretty poor. Perhaps I need more neurons? Maybe if I double the number of dimensions, I need to quadruple the number of neurons. I get this for 8 neurons:

8 neurons

Maybe with 8 neurons I have a lot of weights to fit, so let me try training with regularization. I get the plot below with trainbr:

8 neurons with regularization

It's only at around 16 neurons that I start getting something I would consider reasonable.

16 neurons

However, there are still oscillations which I don't like. Now I know I'm using it out of the box in a naïve manner, or perhaps I'm expecting too much. But this simple example resembles the real problem I want to tackle. I have the following questions:

  • Why is it that an increase from 1 to 2 dimensions increases the number of neurons required to get a decent fit considerably?
  • Even when I go to a larger number of neurons, I get oscillations that are going to be a problem in my real world application. How can I get rid of that?
  • Most resources on NN that I've read indicate a substantially lower number of neurons. They usually state something like "equal to or less than the number of input variables". Why is that? Is a multidimensional Gaussian a pathological case?
  • If I need to be more intelligent with how I treat my network for a given number of neurons, what do I need to do? I tried retraining the network to see if it was a local minima issue, but I generally get a similar fit.
  • Anything else that may be remotely useful to this issue is appreciated!
$\endgroup$
  • $\begingroup$ I think ncases has mentioned good points, one thing I want to add is that you often don't want your NN to fit your data perfectly, because of generalization. You want your NN to learn the patterns, not, to bring in a school analogy, memorize the formular without knowing why and how. $\endgroup$ – RyanMcFlames Jan 29 '18 at 7:22
5
$\begingroup$

Some things to take into account:

  • Try to apply appropriate input space transformations, e.g. convert to polar coordinates.
  • Despite the fact that a single hidden layer feedforward network can be a universal approximator, there is no guarantees about the number of neurons needed to approximate an arbitrarily complex function. Instead of having a single hidden layer and making it increasingly wide, try stacking more layers. This enables the network to model non linearities in an easier way.
  • Do not verify the performance of your model based on visual inspection (at least not only), but based on error measurements (e.g. MSE).
| improve this answer | |
$\endgroup$
  • $\begingroup$ - I could but I would generally not know that the data has some spherical symmetry so this is a non-starter. - I will give this a shot. Much of the literature I've read has suggested going wide as opposed to going deep. Do you have any references you recommend. How does this wide/deep ratio scale with input dimension? - I have checked MSE, gradient, etc.. and also normalized MSE to penalize the central region and edges similarly. $\endgroup$ – pragmatist1 Dec 16 '16 at 23:49
  • $\begingroup$ Please check the NN faq, and this question on quora and this question on cross validated. There you can find both the intuition behind it and references to books and articles. $\endgroup$ – ncasas Dec 18 '16 at 9:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.