I am new to AI and ML and I am learning how does random forest work. I implemented a small experiment. I have got a dataset with 1.6M samples and about 120 features. It is a classification problem, the output, which I am trying to predict, is a binary value. I am using RandomForestClassifier from sklearn in python. I am aiming to maximize accuracy calculated by accuracy_score function. At first attempt there was a big difference between train and test set accuracy, e.g. 100% train and 50% test, so I came to conclusion that my forest is overfitting. I did hyper-params tuning and managed to reduce the difference. Eventually I ended up with the following set:

model = RandomForestClassifier(
  n_estimators = 200,
  max_features = 11,
  max_depth = 30,
  min_samples_leaf = 30,
  n_jobs = 12,
  verbose = 1)

Then I played around with the number of samples and I got the following results: the more samples I use, the lower accuracy I get. Here are results for 2'500, 10'000 and 100'000 samples, on x axis the number of steps ahead I am trying to predict, on y axis accuracy, red is the train set, blue is the test set. It further decreases with more samples.

enter image description here enter image description here enter image description here

I find it counterintuitive, since I believe more data should improve quality, so I would like to first understand why is it the way like that. The only reason I can come up with is, since some of the features used clearly show a trend and are not stationary, the algorithm performs well on a subset of data, which is "more stationary", than on the whole set, which exhibits more changeability. Would it be correct reasoning?

If so, how can I improve it? I can thing of a couple of ideas.

  1. De-trend features, which are not stationary. Seems to be against the general rule, which says decision trees do not require data preprocessing.
  2. Just use a subset of the most recent data. Again intuitively the more data the better, so it sounds awkward.
  3. Accept the fact that with these features this is the best I can get and look for different/more features.

Thanks in advance.

  • $\begingroup$ Have you tried doing hyperparameter tuning for each of the different dataset sizes? I can image that as the number of samples increases so does the variability within each of the classes. A specific hyperparameter might therefore work well for a smaller dataset and not very well for a large dataset. An example of this could be the max_depth value, where I would expect a larger number would be beneficial for a larger dataset. $\endgroup$
    – Oxbowerce
    Jun 18, 2021 at 11:15

2 Answers 2

  1. For all the three samples the training data accuracy saturates around 85℅.
  2. However, with more training data, the gap of training and test data widens.
  3. This is a clear sign of overfit. The hyper-parameter tuning has not alleviated overfiting yet.
  4. Try hyper-parameter tuning with k-fold cross validation.
  5. Here is an article on the same: https://towardsdatascience.com/hyperparameter-tuning-the-random-forest-in-python-using-scikit-learn-28d2aa77dd74
  • $\begingroup$ Thanks for the suggestion, further hyper params tuning with k-fold cross validation helped with overfitting. $\endgroup$
    – 970541804
    Jun 29, 2021 at 16:43

Error may increase with number of samples because of non correlated features.

Have you done a correlation heatmap in order to know if there are non correlated features?

Note: non correlated are close to 0, but you should keep the anti-correlated (<0) because they have a kind of correlation.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.