I am making an ensemble of deep models for solving a classification problem. The initial weights follow the default distribution of keras layers. Each time I run the model(train the 'n' models, make predictions and then get their mode), I get a different result. What could be the possible reasons(rather errors) for this? And how shall I handle them?
$\begingroup$
$\endgroup$
2
-
$\begingroup$ What do you mean by different result? In what sense are they different? $\endgroup$– matthiaw91Commented Oct 23, 2019 at 12:42
-
$\begingroup$ The F1 score, precision, recall etc vary on the predictions. $\endgroup$– yamini goelCommented Oct 23, 2019 at 14:48
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
3
Getting varying results at the level of F1 score is possible but should be high, as your are essentially taking the average over many values (assuming a reasonably sized dataset).
There are however many sources of random behaviour that could be creeping in. From most obvious to most obscure:
- Are the definitions and parameters of each of your models identical each time?
- Do you use the same train/validation/test splits for each session?
- Are your training hyper-parameters (number of epochs, learning rate, early stopping) the same each time?
- Do you set the random seed in Numpy/Keras/Tensorflow?
- Did you set any Nvidia CUDA flags for determinism?
Regarding the final point, GPU determinism - this is a very new thing in the context of deep learning and many people believed it was impossible, but there is good progress being made. Check out these video/slides from Nvidia GTC conference this year.
These are really just the components of each individual building block i.e. each deep model. You could also introduce random behaviour perhaps in the way you chain these models together to create your ensemble.
-
$\begingroup$ I understand your points but don't know if fixing seed values would be a good idea. Point 1,2 and 3 are not my cases. Now in such a case what shall be the right choice in terms of reporting the results? Fixing seeds? $\endgroup$ Commented Oct 24, 2019 at 0:43
-
$\begingroup$ I would usually report some results with a level of confidence. Fixing seeds will be necessary if you want exact result every time, otherwise most libraries/algorithms will introduce some random choice somewhere - a simple example is shuffling the data or weight initialisation. $\endgroup$– n1k31t4Commented Oct 24, 2019 at 13:22
-
$\begingroup$ Just an afterthought @n1k31t4 if I said the seed values, will this not mean that all models in the ensemble would have same initial weights i.e making them identical in my case? $\endgroup$ Commented Oct 29, 2019 at 15:29