0
$\begingroup$

I have an extremely unbalanced data set: around 200 positive samples and 70,000 negative samples. To overcome this problem I have tried to over-sample the minority class as suggested in previous questions here.

Since training the classifier (in my case I am using SVM) takes a long time on my computer I first trained a classifier on a small subset of my data picked randomly, the classifier gave solid results when tested on the test set, yet when I trained a classifier using the full training data the classifier produced very poor results. Does anyone have insight into why this is happening and what should i do? I have also tried using weights but it did not produce a different result.

Improve this question
$\endgroup$
2
  • $\begingroup$ So, performance on the same holdout (test) set was worse training on the full set vs. a subset or was the test set different? And the distribution of +/- records were the same? If the majority records are easily classifiable you can try sub-sampling this class and retaining all minority records to even out the distribution a bit. $\endgroup$
    – user13684
    Nov 29, 2015 at 19:51
  • 1
    $\begingroup$ What metric are you using? The smaller dataset has a chance of containing almost no positive samples. In that case you'd learn to basically always guess the negative class, which will perform very well in terms of accuracy. $\endgroup$
    – jamesmf
    Nov 30, 2015 at 13:34

1 Answer 1

Reset to default
1
$\begingroup$

For the 200 positive samples, are there many different "types"? In other words, will future positive samples look nothing like the current positive samples? If this is the case, you may try anomaly detection in machine learning.

From a lecture by Andrew Ng in Coursera, if you have small number of positive samples (e.g., 20), large number of negative samples (e.g., 10000), and it's hard for any algorithm to learn from the positive samples, you should use anomaly detection, otherwise, you can use supervised learning.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I think you nailed it: it's not a classification problem but a case of anomaly detection. $\endgroup$
    – Eddy
    Jan 31, 2016 at 9:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.