I have trained 2 neural networks with SGD and then I have taken a linear path between their weights. Say W_0 and W_1 are the weight matrices of network 1 and network 2, respectively.

Then I compute W_new = (1-t)W_0+tW_1 for t ranging from -1 to 2, with small increments within this range. I then compute the loss for W_new at each t value. I get the following loss (it displays the cross-entropy and the accuracy), shown in the image below enter image description here.

As you can see, the local minima of N_0 and N_1 happen at t=0 and t=1 as expected. However, there seem to be 2 local minima at t=-0.5 and t=0.5. I find this very strange and it has occurred over many pairs of neural networks.

This is a simple toy network with one hidden layer with 512 neurons and a relu activation.

Any idea as to why this is happening?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.