0
$\begingroup$

I was working on RandomForestClassifier and doing hyperparameter tuning. But something caught my attention. I always get a lower validation value in the 2nd part of Cross Validation. Here is the code:

model6 = RandomForestClassifier(n_estimators = 200,
                            max_depth = 10,
                            min_samples_leaf = 2,
                            min_samples_split = 2)



 cross_val_score(model6,
            X_train,
            y_train,
            cv = 5,
            scoring = 'accuracy')

 array([0.87755102, 0.83673469, 0.85416667, 0.91666667, 0.875     ])

As you see the second section is lower than the others. What could be the reason for this?

$\endgroup$

3 Answers 3

1
$\begingroup$

This problem also might be related to the seeding of the random number generator (RNG). Typically, your RNG spits out a sequence of pseudo-random numbers, meaning, that they look random but there might be, e.g., some repetition at some point. The random number seed tells the RNG where to start in this very long sequence of (pseudo) random numbers. In other words, if you always "seed" the RNG with the same integer value, then you always get a reproducible sequence of random numbers. THis is useful for reproducing of results and for debugging.

E.g. in numpy, if you set the seed as

import numpy as np

np.random.seed(10)
print(np.random.rand())

np.random.seed(10)
print(np.random.rand())

you will get two times the same number. If you use np.random.seed(None) (or np.random.seed()) you always will get different random numbers.

It is important to note that the RNGs of numpy and, e.g., pytorch need both to be initialized/seeded.

$\endgroup$
4
  • $\begingroup$ After your comment i splitted data with random_state from 1 until 10. And the highest validation score mean has come with random_state = 8. array([0.95918367, 0.81632653, 0.91666667, 0.91666667, 0.85416667]) But still second section is the lower one. $\endgroup$ Nov 10, 2023 at 15:02
  • $\begingroup$ I think it ain't about modelling it's about data processing. Or data's structure. $\endgroup$ Nov 10, 2023 at 15:08
  • $\begingroup$ I think it would tremendously increase your chance for getting a helpful reply if you would post a piece of code in the spirit of a minimal working example... $\endgroup$
    – BanDoP
    Nov 10, 2023 at 15:45
  • $\begingroup$ I posted the code as a answer. $\endgroup$ Nov 10, 2023 at 15:49
0
$\begingroup$

Maybe data is not shuffled properly. Try to shuffle it explicitly before cross_val_score() is called. What is your dataset?

$\endgroup$
4
  • $\begingroup$ I shuffled data with data.sample(frac = 1). Also my data is heart disease dataset. $\endgroup$ Nov 10, 2023 at 10:35
  • $\begingroup$ Also, make sure that you didn't fix the seeding of the random number generator. A piece of complete but minimal code would be quite helpful to give some proper advice. $\endgroup$
    – BanDoP
    Nov 10, 2023 at 11:04
  • $\begingroup$ That is the another subject which i don't understand. Just because of it i don't use any random seed for generators. Because when i use i always take another validation score. Higher or lower. $\endgroup$ Nov 10, 2023 at 11:20
  • $\begingroup$ since the explanation is a bit longer and needed code formatting I added an answer which should explain these things. Hope that helps. $\endgroup$
    – BanDoP
    Nov 10, 2023 at 13:56
0
$\begingroup$

That is the code:

X_train, X_test, y_train, y_test = train_test_split(X,
                                                    y,
                                                    test_size = 0.2)

model = RandomForestClassifier(n_estimators = 200,
                               max_depth = 10,
                               min_samples_leaf = 2,
                               min_samples_split = 2,
                               random_state = 8)

cross_val_score(model,
                X_train,
                y_train,
                cv = 5,
                scoring = 'accuracy').mean()
$\endgroup$
1
  • $\begingroup$ Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. $\endgroup$
    – Community Bot
    Nov 11, 2023 at 2:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.