I'm trying to evaluate classifiers comparison by running the sample script that can be found here.
What I noticed is that in some cases the classifier is not reset. In fact, duplicating some of those (with no parameter change) the score and the countour change between the two.
This can be seen simply replacing
AdaBoostClassifier() in the classifier list with another
I guess that at every cicle of the for loop the classifier should be reset in order to make a fair comparison among the different models, and this case should behave the same I think.
In particular, differences are noticed duplicating the MPL (Neural Net) and the Random Forest, while there is no change duplicating KNN or RBF SVM.
I also tried to
clone the classifier, and even
del clf in the loop, but the behaviour stays the same.
How can I make the evaluation replicable and not influenced by the previous run? I want to be sure that when I use the same model, and only change the parameters the result is correct, and this will be possible only if two identical model yield the same result.