here is a time series, at each time $i,$ we have several features $(a_1, a_2, ... a_k)$ and a binary label $y.$ Now I use a window

$$(a_1[i], a_1[i-1],..., a_1[i-s],..., a_k[i], a_k[i-1],..., a_k[i-s])$$

to predict $y[i+1].$ The model is random forest of classification and below is my parameter:

RandomForestClassifier(max_depth=20, max_features='sqrt', min_samples_leaf=4,  min_samples_split=5, n_estimators=1400, n_jobs=50),

I happened to a strange result that as I change the selection of feature, length of window, e.t.c, the accuracy on training set is between 70%-90%. However the accuracy on test set is always close to the distribution of labels on test set. e.g. 60% 1 and 40% 0 for test set, then the accuracy on test set is about 60%. It seems a naive prediction namely using mode of test set as a constant prediction. But I checked output of prediction, it is not a constant. Could I ask what happened?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.