I am working on titanic dataset, I achieved 92% accuracy using random forest. However, the accuracy score dropped to 89% after I tuned it using Gridsearch. Now, I was wondering if it caused by imbalanced dataset, since it has only 342 out of 891 passenger survived the disaster. Would appreciate the clarification.

  • $\begingroup$ was the 92% accuracy on the train set or validation set or crossvalidation (or god forbid the test set)? and what about the 89%? $\endgroup$ – A Kareem May 4 at 15:11

Welcome! You haven't given us enough information to be able to diagnose this issue completely, but you should check your grid search code to see how each cross-validated model is being trained and note which parameters are different from those used with the 92% model.

If it has something to do with the unbalanced data, it's because you're not stratifying your sampling of the dataset with respect to the labels (making sure training and validation sets have the same ratio of classes (in this case, 38% survive, 62% don't survive.))

I would guess that something is going on with your cross validation process. If I had to guess specifics (and again, we can't say for sure given what you've posted here) , I would say that something you're doing in the CV probably results in those models not using as much training data as the 92% model.

| improve this answer | |
  • $\begingroup$ To add to this, I'd also state that a 3% difference in a dataset that seems to only have 1000 samples is small enough that it's hard to state with certainty that something wrong is going wrong here and not just random noise. $\endgroup$ – A Kareem May 4 at 15:18
  • $\begingroup$ Right on. The 92% could definitely be a lucky draw from a train-test split (or any number of other things, as you say). $\endgroup$ – Matthew May 4 at 15:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.