Trying to calibrate a relatively vanilla NN, setting the hyper-parameter tuning aside*, it appears that weight initialisation has a lot of impact on the model output. Ie. Models calibrated with differents weight initialisations, despite similar overall performance, can yield very different individual prediction.
What could be the causes of such a behavior ? What would be the solutions ?
*The hyper-parameter tuning can't really be set aside, as regularisation may help getting the same result by reducing the number of variables. However, I observe similar behavior if I use a value significantly above / below the optimal hyper-parameter value.