I'm using sklearn/pandas/numpy.

I have a labeled data set, where the potential outcomes are either True or False. However, the data set has a much higher proportion of True entries. When running through classifiers with k-fold (n=5) cross validation, this appears to bias the classifier towards just saying True.

Using weights, I was able to adjust the sample data set I'm using to have a proportion closer to 1:1, like so (using a pandas csv):

results = csv[['result']]
weights = np.where(results.as_matrix() == True,0.25,1).ravel()
csv_sample = csv.sample(n=60000, weights=weights)

And the results are much more promising! However, I'm wondering if there's a way for me to do cross validation where the TRAINING set is adjusted in this manner, but the TEST set is closer to the actual proportion of data.

  • $\begingroup$ Have you tried sample_weight=balanced or auto in your model? This takes care of unbalanced datasets rather nicely. $\endgroup$
    – Diego
    Mar 19 '16 at 0:01
  • $\begingroup$ @Diego I'm pretty new to sklearn, so I did not know about this feature! Will definitely look into it, cause it seems promising, thank you $\endgroup$ Mar 19 '16 at 0:16
  • $\begingroup$ Yes, go ahead then. It was a blessing for what I was trying to do. I have added this as an answer so in case this works for you don't forget to accept it. $\endgroup$
    – Diego
    Mar 19 '16 at 0:27

Try to use predictor option class_weight='balanced' or auto. It worked really well for me for SGDClassifier in a similar situation.

  • $\begingroup$ To clarify, the class_weight value is dependent on the docs for the classifier, the one I was looking at only had "balanced" and "balanced_subsample", and when digging through the code, I noticed that, at least in this particular instance, 'auto' was being deprecated just through I'd note for anyone else passing this question/answer! $\endgroup$ Mar 28 '16 at 19:52
  • $\begingroup$ Yes, I mentioned the "auto" just in case an older version of sklearn has to be used. $\endgroup$
    – Diego
    Mar 29 '16 at 11:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.