Why the one validation score is lower than the other sections of cross validation

Question

I was working on RandomForestClassifier and doing hyperparameter tuning. But something caught my attention. I always get a lower validation value in the 2nd part of Cross Validation. Here is the code:

model6 = RandomForestClassifier(n_estimators = 200,
                            max_depth = 10,
                            min_samples_leaf = 2,
                            min_samples_split = 2)



 cross_val_score(model6,
            X_train,
            y_train,
            cv = 5,
            scoring = 'accuracy')

 array([0.87755102, 0.83673469, 0.85416667, 0.91666667, 0.875     ])

As you see the second section is lower than the others. What could be the reason for this?

Answer 1

This problem also might be related to the seeding of the random number generator (RNG). Typically, your RNG spits out a sequence of pseudo-random numbers, meaning, that they look random but there might be, e.g., some repetition at some point. The random number seed tells the RNG where to start in this very long sequence of (pseudo) random numbers. In other words, if you always "seed" the RNG with the same integer value, then you always get a reproducible sequence of random numbers. THis is useful for reproducing of results and for debugging.

E.g. in numpy, if you set the seed as

import numpy as np

np.random.seed(10)
print(np.random.rand())

np.random.seed(10)
print(np.random.rand())

you will get two times the same number. If you use np.random.seed(None) (or np.random.seed()) you always will get different random numbers.

It is important to note that the RNGs of numpy and, e.g., pytorch need both to be initialized/seeded.

Answer 2

Maybe data is not shuffled properly. Try to shuffle it explicitly before cross_val_score() is called. What is your dataset?

Answer 3

That is the code:

X_train, X_test, y_train, y_test = train_test_split(X,
                                                    y,
                                                    test_size = 0.2)

model = RandomForestClassifier(n_estimators = 200,
                               max_depth = 10,
                               min_samples_leaf = 2,
                               min_samples_split = 2,
                               random_state = 8)

cross_val_score(model,
                X_train,
                y_train,
                cv = 5,
                scoring = 'accuracy').mean()