Random forest and the number of samples

Question

I am new to AI and ML and I am learning how does random forest work. I implemented a small experiment. I have got a dataset with 1.6M samples and about 120 features. It is a classification problem, the output, which I am trying to predict, is a binary value. I am using RandomForestClassifier from sklearn in python. I am aiming to maximize accuracy calculated by accuracy_score function. At first attempt there was a big difference between train and test set accuracy, e.g. 100% train and 50% test, so I came to conclusion that my forest is overfitting. I did hyper-params tuning and managed to reduce the difference. Eventually I ended up with the following set:

model = RandomForestClassifier(
  n_estimators = 200,
  max_features = 11,
  max_depth = 30,
  min_samples_leaf = 30,
  n_jobs = 12,
  verbose = 1)

Then I played around with the number of samples and I got the following results: the more samples I use, the lower accuracy I get. Here are results for 2'500, 10'000 and 100'000 samples, on x axis the number of steps ahead I am trying to predict, on y axis accuracy, red is the train set, blue is the test set. It further decreases with more samples.

I find it counterintuitive, since I believe more data should improve quality, so I would like to first understand why is it the way like that. The only reason I can come up with is, since some of the features used clearly show a trend and are not stationary, the algorithm performs well on a subset of data, which is "more stationary", than on the whole set, which exhibits more changeability. Would it be correct reasoning?

If so, how can I improve it? I can thing of a couple of ideas.

1. De-trend features, which are not stationary. Seems to be against the general rule, which says decision trees do not require data preprocessing.
2. Just use a subset of the most recent data. Again intuitively the more data the better, so it sounds awkward.
3. Accept the fact that with these features this is the best I can get and look for different/more features.

Thanks in advance.

Answer 1

1. For all the three samples the training data accuracy saturates around 85℅.
2. However, with more training data, the gap of training and test data widens.
3. This is a clear sign of overfit. The hyper-parameter tuning has not alleviated overfiting yet.
4. Try hyper-parameter tuning with k-fold cross validation.
5. Here is an article on the same: https://towardsdatascience.com/hyperparameter-tuning-the-random-forest-in-python-using-scikit-learn-28d2aa77dd74

Answer 2

Error may increase with number of samples because of non correlated features.

Have you done a correlation heatmap in order to know if there are non correlated features?

Note: non correlated are close to 0, but you should keep the anti-correlated (<0) because they have a kind of correlation.

https://medium.com/@szabo.bibor/how-to-create-a-seaborn-correlation-heatmap-in-python-834c0686b88e