Sometimes we come across datasets where classes are imbalanced. For example, class A may have 2000 instances, but class B has only 200. How can we train a classifier for such datasets?
$\begingroup$
$\endgroup$
1
-
$\begingroup$ cross validation $\endgroup$– Sreejithc321Jun 26, 2015 at 4:16
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
3
Well, you just train it on unbalanced dataset, it is not a problem. I don't think you need to apply some special techniques.
The only case where you may want to do something special is if your classes are skewed by their nature (if the skewness is the property of class itself, not of just your specific dataset). For example, if you build a classifier which will tell if someone has a malignant tumor in X-ray image or not. By nature of the task absolute majority of patients will not have a malignant tumor. In such case you may want to tune your classifier algorithm a bit, for example introduce a weighting into your SVM or smth like it.
A special case is if you have EXTREMELY unbalanced classes. For example 100000 of positive examples and 20 negatives. In such case you will want to go away from classification task to the approach called 'anomaly detection'.
-
-
$\begingroup$ Support vector machine. Just as an example, this algorithm offers an ability to assign different weights to classes manually $\endgroup$ Jun 28, 2015 at 15:49
-
$\begingroup$ Are you trying to say: add a little bit of bias so your variance will decrease and your model can predict better on unseen data?! $\endgroup$– user58746Jan 12, 2021 at 21:24
$\begingroup$
$\endgroup$
1
There are a few of other approaches you can take to try to balance your class distribution.
.
Subsample Majority Class
You can balance the class distributions by subsampling the majority class.
Oversample Minority Class
Sampling with replacement can be used to increase your minority class proportion.
Adding Noise
A more sophisticated scheme is to add Gaussian, or other suitable, noise to the existing instances of the minority class in an effort to create a greater number of representative, but diverse, instances.
SMOTE
A popular method to synthesise minority class instances with greater complexity than pure noise addition is SMOTE ( Synthetic Minority Oversampling TEchnique ). This uses a K-member neighbourhood in feature space to impute new instances.
WEKA has a filter for this.
Though there is some evidence that this technique is not overly beneficial with high dimensional data here.
-
$\begingroup$ Can you please cite some references on the "adding noise" solution you are suggesting for example in the context of time series? $\endgroup$– tagomaJul 9, 2017 at 20:52
$\begingroup$
$\endgroup$
General options:
Group your features into groups by generating new labels that represent multiple features
Use dimensionality reduction, e.g. PCA, Autoencoder, etc. A lot of these are implemented in Sklearn and the downside is that your features become confusing to analyze once it converts them to pure mathematically representations that can have a relation meaning that the algorithm 'learned'
Algorithm dependent options:
Normalization, which in Neural Networks defines the contributions of each feature in a specific layer to the final classification, regression, etc
Dropout, this is similar to my comment, tells your model to randomly ignore a certain percentage of inputs between your Neural Network layers
Observation: I described Neural Networks for the last two options, but you can also apply them in a few different algorithms, to give you an example I studied Trees a lot and I saw that you can apply it in Decision Tree, Random Forest, and in Extreme Gradient Boosting versions of trees like Ada Boost, Cat Boost, etc
Generally I would see the data information, if you're using pandas
info,describe,plot(works for each feature of your dataset),isnull().values.any(), etc; and mainly the visual plot to see its balance. In a few problems, I didn't know much about these and it played a huge role on the later decisions!