here is a time series, at each time $i,$ we have several features $(a_1, a_2, ... a_k)$ and a binary label $y.$ Now I use a window
$$(a_1[i], a_1[i-1],..., a_1[i-s],..., a_k[i], a_k[i-1],..., a_k[i-s])$$
to predict $y[i+1].$ The model is random forest of classification and below is my parameter:
RandomForestClassifier(max_depth=20, max_features='sqrt', min_samples_leaf=4, min_samples_split=5, n_estimators=1400, n_jobs=50),
I happened to a strange result that as I change the selection of feature, length of window, e.t.c, the accuracy on training set is between 70%-90%. However the accuracy on test set is always close to the distribution of labels on test set. e.g. 60% 1 and 40% 0 for test set, then the accuracy on test set is about 60%. It seems a naive prediction namely using mode of test set as a constant prediction. But I checked output of prediction, it is not a constant. Could I ask what happened?
