1
$\begingroup$

Trying to calibrate a relatively vanilla NN, setting the hyper-parameter tuning aside*, it appears that weight initialisation has a lot of impact on the model output. Ie. Models calibrated with differents weight initialisations, despite similar overall performance, can yield very different individual prediction.

What could be the causes of such a behavior ? What would be the solutions ?

*The hyper-parameter tuning can't really be set aside, as regularisation may help getting the same result by reducing the number of variables. However, I observe similar behavior if I use a value significantly above / below the optimal hyper-parameter value.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
-1
$\begingroup$

As you have noticed weight initialization is extremely important, If for instance, you initialize your network with zeros, then it's guaranteed you will not learn from data. there are many ways to initialize your data:

Most simple approach (beginner approach) : just use random initialization

Xavier Initialization: this might be a more intermediate approach if you are interested on this check the following link: https://towardsdatascience.com/weight-initialization-in-neural-networks-a-journey-from-the-basics-to-kaiming-954fb9b47c79

I'd say that you gotta keep in mind when your network is "learning" it is basically navigating in a hyper dimension. most people do not actually standardize data. Standardizing data is essential since you make learning easier for your algorithm.

So short answer: standardize your data, your algorithm will converge faster into a minimum (of course not always global).

Improve this answer
$\endgroup$
5
  • $\begingroup$ I am familiar whit that, the question is different here. Given a random initialisation technique, taht give good preformance overall. How do you handle the fact that individual prediction depends on the initialisation ? $\endgroup$
    – lcrmorin
    Jun 11, 2020 at 10:40
  • $\begingroup$ Hmm no offense but your question shows that you are not familiar with the mathematical concepts of deep learning. you can't handle the fact that different predictions depend on different initial conditions due to the fact that you are not implementing a closed solution. The only way to "handle" it would be the following: with higher dimensions the probability of finding a local minimum goes down, it's more probable to find a saddle point, so your algorithm will converge to a global minimum, but if it encounters saddle points it will be slow. $\endgroup$
    – Eduardo Lozano
    Jun 11, 2020 at 20:36
  • $\begingroup$ I actually have a very advanced background in optimisation... I am more than familiar with the concepts you mention. I am not asking about how to perform initialisation (your answer) nor about standardisation (which I already perform...), but rather what to do if I can't reach a global minimum (very big parameter space) but multiple and 'very different' local minima. $\endgroup$
    – lcrmorin
    Jun 12, 2020 at 11:39
  • $\begingroup$ have you tried to add momentum? $\endgroup$
    – Eduardo Lozano
    Jun 16, 2020 at 8:07
  • $\begingroup$ Though computing the Hessian Might computational expensive. $\endgroup$
    – Eduardo Lozano
    Jun 16, 2020 at 8:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.