Performance and architecture of neural network for increased dimensions

Question

I posted this question on Cross Validated before I realized that this existed. I think it is better suited here and got no answers over there so I have deleted other post. I have reproduced the question below:

I have been playing around with the neural network toolbox in MATLAB to develop an intuition for how the architectural requirements scale with feature dimension.

I put together a simple example, and the results have surprised me. I am hoping someone can point to either (a) an unrealistic expectation of mine, or (b) a mistake/misuse of the neural network toolbox.

The example is as follows: I have a simple un-normalized one-dimensional Gaussian that I am trying to learn. I do the following:

x = -5:0.2:5;
y = exp(-x.^2/2);

net = feedforwardnet(2);
net = configure(net, x, y); 
net = train(net, x, y); 
y2 = net(x); 

plot(x, y, 'o', x, y2);
legend('Data', 'NN');

This gives me good results. I get the plot below.

Now, I try to extend this to 2 dimensions and this is where I run into trouble. I don't think I'm asking too much. My data is not noisy, or is it sparse. I figure if I double the number of neurons that should be sufficient for an increase in dimensionality. Here's my code:

x1 = -5:0.2:5;
x2 = -5:0.2:5;
[x1g, x2g] = meshgrid(x1, x2);
xv = [x1g(:)'; x2g(:)'];
yv = exp(-dot(xv,xv)/2);

net = feedforwardnet(4); 
net = configure(net, xv, yv);
net = train(net, xv, yv);
y2v = net(xv);  
plot3(xv(1,:), xv(2,:), yv, xv(1,:), xv(2,:), y2v, 'o');
legend('Data', 'NN');

The plot I get is this:

This is pretty poor. Perhaps I need more neurons? Maybe if I double the number of dimensions, I need to quadruple the number of neurons. I get this for 8 neurons:

Maybe with 8 neurons I have a lot of weights to fit, so let me try training with regularization. I get the plot below with trainbr:

It's only at around 16 neurons that I start getting something I would consider reasonable.

However, there are still oscillations which I don't like. Now I know I'm using it out of the box in a naïve manner, or perhaps I'm expecting too much. But this simple example resembles the real problem I want to tackle. I have the following questions:

• Why is it that an increase from 1 to 2 dimensions increases the number of neurons required to get a decent fit considerably?
• Even when I go to a larger number of neurons, I get oscillations that are going to be a problem in my real world application. How can I get rid of that?
• Most resources on NN that I've read indicate a substantially lower number of neurons. They usually state something like "equal to or less than the number of input variables". Why is that? Is a multidimensional Gaussian a pathological case?
• If I need to be more intelligent with how I treat my network for a given number of neurons, what do I need to do? I tried retraining the network to see if it was a local minima issue, but I generally get a similar fit.
• Anything else that may be remotely useful to this issue is appreciated!

Answer 1

Some things to take into account:

• Try to apply appropriate input space transformations, e.g. convert to polar coordinates.
• Despite the fact that a single hidden layer feedforward network can be a universal approximator, there is no guarantees about the number of neurons needed to approximate an arbitrarily complex function. Instead of having a single hidden layer and making it increasingly wide, try stacking more layers. This enables the network to model non linearities in an easier way.
• Do not verify the performance of your model based on visual inspection (at least not only), but based on error measurements (e.g. MSE).