I have a collected data from 50 unique blocks, and then merged data from 49 blocks into one data set, and saved the data from 1 block for testing purpose.
I then split the merged data set from 49 blocks using train_test_split(sklearn). Then used training data to train a random forest regressor using cross validation and get a good model score(R^2 score from sklearn random forest regressor model) on train (0.99) and test set(0.94). But when I use the trained model on the reserved data from the 1 block, the performance is very bad (-1.0).
If I merge the data from all 50 blocks, then use train test split and keep 60% data as training set, 20% as test set, and 20% (reserved set), I get good scores from all three sets. Training set score(0.98), test set(0.93) and the reserved set(0.96).
Any intuition regarding what could be causing this? And any suggestions on how to improve the model score for the 1 block of unseen data?