0
$\begingroup$

I used Random Forest and hypertuned the parameters for a binary classification problem on a dataset (dataset A). I got a F1 score of 0.78. I then used a second dataset (dataset B). It was very similar to dataset A (same variables and the distribution of classes in the target variable). I again built and trained a different Random Forest algorithm for dataset B. I expected the f1 score to be around 0.78, but the f1 score for dataset B was 0.50.

Why could there be such a large difference between the f1 scores of the 2 datasets?

Both datasets (A & B) are very similar to each other and I trained separate models on both of them.

Improve this question
$\endgroup$
5
  • $\begingroup$ How have you measured the similarity of two datasets? meaning their distribution? By variable do you mean the features? if so, these two datasets are capable of being aggregated, no? $\endgroup$
    – Fatemeh Asgarinejad
    May 31, 2019 at 23:55
  • $\begingroup$ Yes.Most of the variables (features) are categorical.The distribution of the categories is slightly different but not a lot.Yes they can be aggregated.But I don't understand why the metrics are so different. $\endgroup$
    – data_analyst
    May 31, 2019 at 23:56
  • $\begingroup$ Several things could be going on, ranging from overfitting to just a poor model selection for said data, to possibly insufficient data preprocessing. Try to look into these issues first. If the problem persists; could you please share your code with us? We'd be able to help you more that way. $\endgroup$
    – Nicolas Essipova
    Jun 1, 2019 at 0:18
  • 1
    $\begingroup$ How many instances and how many features? Too few instances and/or too many features could cause overfitting $\endgroup$
    – Erwan
    Jun 1, 2019 at 0:57
  • $\begingroup$ I A had 200 features (almost all categorical) and Dataset B had 215 features ( almost all categorical).Now that I think about it, most of the features in Dataset B had more categories. As I hot encoded the categorical variables, may be I made the dataset very sparse. So although dataset B had more features, the dataset was sparse..I will look more into feature reduction. I used cross validation with randomized search CV.I thought that controlled for over fitting with random forests..am I right in assuming that or do you recommend any other steps to check for over fitting? Thanks! $\endgroup$
    – data_analyst
    Jun 2, 2019 at 7:44

1 Answer 1

Reset to default
1
$\begingroup$

Simple answer: the datasets are different. Apparently, your approach worked well for A but not for B. From this you might infer, that B and A are not so similar after all. I suspect that B has a more complicated relation between the X and targets y. So maybe try boosting on B?!

Improve this answer
$\endgroup$
2
  • $\begingroup$ Boosting B makes sense...but I am torn between investing my time in boosting the model or looking at data processing or feature selection. When should one stop looking at data processing and just focus on model boosting?I feel I am doing something wrong with data processing...just can't put my finger on it. $\endgroup$
    – data_analyst
    Jun 2, 2019 at 7:48
  • $\begingroup$ I never do (too) much manual feature selection, since at some point you just don‘t make good progress. If you can afford, I would just try (a little) boosting with L1 penalty. So there is „automatic“ feature selection by shrinking. After that you can have a look at feature importance. $\endgroup$
    – Peter
    Jun 2, 2019 at 9:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.