I am new to AI and ML and I am learning how does random forest work. I implemented a small experiment. I have got a dataset with 1.6M samples and about 120 features. It is a classification problem, the output, which I am trying to predict, is a binary value. I am using RandomForestClassifier from sklearn in python. I am aiming to maximize accuracy calculated by accuracy_score function. At first attempt there was a big difference between train and test set accuracy, e.g. 100% train and 50% test, so I came to conclusion that my forest is overfitting. I did hyper-params tuning and managed to reduce the difference. Eventually I ended up with the following set:
model = RandomForestClassifier( n_estimators = 200, max_features = 11, max_depth = 30, min_samples_leaf = 30, n_jobs = 12, verbose = 1)
Then I played around with the number of samples and I got the following results: the more samples I use, the lower accuracy I get. Here are results for 2'500, 10'000 and 100'000 samples, on x axis the number of steps ahead I am trying to predict, on y axis accuracy, red is the train set, blue is the test set. It further decreases with more samples.
I find it counterintuitive, since I believe more data should improve quality, so I would like to first understand why is it the way like that. The only reason I can come up with is, since some of the features used clearly show a trend and are not stationary, the algorithm performs well on a subset of data, which is "more stationary", than on the whole set, which exhibits more changeability. Would it be correct reasoning?
If so, how can I improve it? I can thing of a couple of ideas.
- De-trend features, which are not stationary. Seems to be against the general rule, which says decision trees do not require data preprocessing.
- Just use a subset of the most recent data. Again intuitively the more data the better, so it sounds awkward.
- Accept the fact that with these features this is the best I can get and look for different/more features.
Thanks in advance.