1
$\begingroup$

I am training a neural network with 1 sigmoid hidden layer and a linear output layer. The network simply approximates a cosine function. The weights are initiliazed according to Nguyen-Widrow initialization and the biases are initialized to 1. I am using MATLAB as a platform.

Running the network a number of times without changing any parameters, I am getting results (mean squared error) which range from 0.5 to 0.5*10^-6. I cannot understand how the results can even vary that much, I'd imagine there would at least be a narrower and more consistent window of errors.

What could be causing such a large variance?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

In general, there is no guarantee that ANNs such as a multi-layer Perceptron network will converge to the global minimum squared error (MSE) solution. The final state of the network can be heavily dependent on how the network weights are initialized. Since most initialization schemes (including Nguyen-Widrow) use random numbers to generate the initial weights, it is quite possible that some initial states will result in convergence on local minima, whereas others will converge on the MSE solution.

Improve this answer
$\endgroup$
5
  • $\begingroup$ I agree. I primarily tested with the weights initialized to 1 (I have my reasons) and it's how I first noticed the large variance. Then I initialized the weights using NW to check if the results would be better. There was some improvement but the large variance was still present. $\endgroup$
    – mesllo
    Commented Mar 12, 2015 at 13:28
  • $\begingroup$ If you repeat training with the same set of initial weights, I would expect the same result (unless you have some sort of asynchronous processing going on). One other point: the difference in MSE between 0.5 and 0.5*10^-6 is only about 0.5, which isn't necessarily a large difference, depending on your training set size, number of outputs, and initial MSE. $\endgroup$
    – bogatron
    Commented Mar 12, 2015 at 15:00
  • $\begingroup$ The variance was still being observed with weights and biases all initialized to 1, which is why I felt confused when this happened. The data set is also consistent and it contains 10,000 values. There is less variance now even with such initialization, however everynow and then it tends to happen and it seems very strange. Is it possible that the error surface being created is different everytime and hence the network sometimes gets stuck in local minimum by chance? $\endgroup$
    – mesllo
    Commented Mar 12, 2015 at 15:44
  • $\begingroup$ What makes you say that 0.5 vs. 0.5*10-6 is a large variance? What is the MSE at the start of training? Also, I'm suspicious about all weights being initialized to 1. If all the weights in an MLP are initialized to the same value, then I would expect all final weights for a given layer to converge to a common value. $\endgroup$
    – bogatron
    Commented Mar 12, 2015 at 15:53
  • $\begingroup$ I have one output since it's approximating a one dimensional cosine function, 10,000 values for training set size. The initial MSE (after the first epoch using Levenberg Marquardt BP) is 14.6268. The neural network has one hidden layer with 4 neurons and it is approximating cos(Pi). All the weights and biases are initialized to 1. It might be worth noting that the data division is happening at random (I'm calling dividerand in MATLAB) every time. Again, the dataset is consistent, and each point is evenly spaced out (that is the cosine function is being plotted with a fixed step) $\endgroup$
    – mesllo
    Commented Mar 12, 2015 at 19:12
0
$\begingroup$

I asked a similar question:

https://stats.stackexchange.com/q/140406/70282

What I ended up doing is having a while loop repeatably create NNs of a small size looking for the best root mean square error.

Essentially: For hidden layer size of 1 to 10 do For trials of 1 to 10: Create NN and test for rmse with test data. Remember NN and best rmse for particular hidden layer size. Done for. Done for. Review results choosing the smallest hidden layer size which had a small rmse.

By separating your data into training and tedtinh data, you don't pick an over fit NN.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.