1
$\begingroup$

I have a collected data from 50 unique blocks, and then merged data from 49 blocks into one data set, and saved the data from 1 block for testing purpose.

I then split the merged data set from 49 blocks using train_test_split(sklearn). Then used training data to train a random forest regressor using cross validation and get a good model score(R^2 score from sklearn random forest regressor model) on train (0.99) and test set(0.94). But when I use the trained model on the reserved data from the 1 block, the performance is very bad (-1.0).

If I merge the data from all 50 blocks, then use train test split and keep 60% data as training set, 20% as test set, and 20% (reserved set), I get good scores from all three sets. Training set score(0.98), test set(0.93) and the reserved set(0.96).

Any intuition regarding what could be causing this? And any suggestions on how to improve the model score for the 1 block of unseen data?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Can you elaborate what kind of data you have on hand what these "blocks" are? $\endgroup$
    – Jonathan
    Jul 15, 2020 at 5:13
  • 1
    $\begingroup$ It could be that the 1 block you held out included some outlier data that was not found in the other 49 blocks. You can verify this by repeating your experiment and hold out different single blocks and see if the performance is still bad each time. Otherwise, I would look at methods on how to prevent overfitting next. $\endgroup$
    – Donald S
    Jul 15, 2020 at 5:32
  • $\begingroup$ @Sammy, The blocks are layout design data from VLSI chip design. $\endgroup$
    – Deepika Kalra
    Jul 15, 2020 at 18:14
  • $\begingroup$ @DonaldS, I did try the experiment, holding out on a different block, and the result on this new block was also bad. $\endgroup$
    – Deepika Kalra
    Jul 15, 2020 at 18:15

1 Answer 1

Reset to default
1
$\begingroup$

Welcome to the community!

There are points coming to my mind:

  • Check the amount of data in each block and then their distribution. First experience might be due to the lack of enough data in reserved block (i.e. you literally just trained but did not validated resulting to a complete overfitting) or having a totally different distribution (i.e. the difference between blocks is NOT random).
  • Be sure about wht you called "Cross Validation". If you splitted 49 blocks into a training and a testing set, then it is minimal version of Cross Validation. 49 blocks should be splitted to $k$ different set, each time one of them plays the test set role and then the final error is averaged over all $k$ runs. That is statistically more robust (first experience shows problem in robustness of statistical inference)
  • Second experiment uses 100 blocks instead of 50 (where were those other 50 in first experiment?) so first, you have more data, and second, you did not split according to "blocks" (whatever they are) but merged all and randomly chose train, validation and test. This procedure reduces the danger of distributional bias within train/valid/test and also might be an indicator of the fact that statistically those "blocks" are different (their distribution)

hope it helped. Please share your results so everyone can learn from it.

Good Luck!

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for the feedback. I will look at the distribution for each block. For 'Cross Validation', I used RandomizedSearchCV, param_distribs = { 'max_leaf_nodes': randint(low=25, high=75), 'n_estimators': randint(low=70, high=120), 'max_depth': randint(low=5, high=10), } model_rand_for = RandomForestRegressor() rnd_search = RandomizedSearchCV(model_rand_for, param_distributions=param_distribs, n_iter=10, cv=5, random_state=42). For the 2nd experiment, all 50 blocks were used. fixed it in the original question. $\endgroup$
    – Deepika Kalra
    Jul 15, 2020 at 18:11
  • $\begingroup$ 1. I plotted the learning curve and realized that I was over fitting the model. So, I tried to add regularization(With hyper parameter tuning) to avoid over fitting, even the the performance wasn’t very good. 2. Then I looked at the feature correlation with target and plotted it using pairplot. Then I realized my features are not representative of the target. The features were random and no correlation to the target except for one feature. So now, I am back to square one to figure out the right feature set first. $\endgroup$
    – Deepika Kalra
    Jul 22, 2020 at 0:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.