Pulling out Training Data from another Dataframe

Question

I'm trying to do a Random Forest Regression on a geographical dataset. I'm hoping I'm doing things right, and if anyone can see an issue with this please let me know!

Problem: I have an area, with a bunch of features and Northing and Easting information, and a target variable (DISTANCE). My area_df has all the data, except the DISTANCE.

I have written a script that scans over that training df with a moving window, and creates several .csv's that I intend to use as training sets over the entire area.

I iterate through those files, one by one, and run a Random Forest Regression on them. Ending up with this:

I intend to get a consensus through average, or sum, or something, of all models. However, the serious problem is training data leaking into the model. You can see from the first 5 instances, and the first training model, it predicts very accurately. Of course, because the area_df has all the data.

I feel like there's 100 ways to do this, but I can't even think of one. I intend to keep working on this, but hopefully someone can help me out as well (and maybe highlight things where I might be going wrong and not realizing it).

Thanks!

Answer 1

Have you try to use train_test_split from scikit-learn (https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html)?

After this, you can use the cross_val_score (https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_score.html) to evaluate your model.

Other option is ensemble the models into an unique model (https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.VotingRegressor.html#sklearn.ensemble.VotingRegressor).