1
$\begingroup$

I have been looking for some methods to improve my prediction model, but could not find any so far. I have data, including 10 numerical features, that I use for prediction. I used a random forest regression model and it works perfectly on the train set (some part of perfection comes from over fitting).

However, the model is not doing a good job for data points that some of their features are not within the range of the sample data. For example, parameter x is between 0 and 2000 in the sample but the new data point that we want to predict has a value of 3000 for x.

I know this is due to data limitation, but I wonder if there is any way that I can generate samples or improve the prediction?

I wanted to build a simple model excluding this parameter, but the problem is that this is the most significant parameter in my model.

Any hints are appreciated.

$\endgroup$
  • $\begingroup$ Any issues with overfitting will be greatly magnified with out of sample parameter data. $\endgroup$ – Paul Sep 15 '17 at 19:08
1
$\begingroup$

I have encountered case in which potent tree learners acted like Nearest Neighbor variants: They would learn to divide the search space so that only examples remained that where close on some meaningful axis (in my case lat and long :)). This could still generalize to examples in the test set that shared those features but not to genuinely unseen examples. Or to put it in another way: The method is potent enough to find rules that work well, but do not generalize in the way you want it to (aka over fitting, but the maybe in a specific way). What helped me were two things: First of all I tested this characteristic of the domain by just using KNN,only feeding it subsets of the feature space (and sure enough KNN over lat, lng worked like a charm). This helped me understand that examples that were in the same space time coords shared certain history, so the second action was to eliminate those examples in the train space and evaluate on an a set of examples totally outside of the coordinates (space and time) of the train set. It is hard to say where this behavior could lie in your case without having a peek at the code, but perhaps my anecdote helps.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.