39

I would say the answer depends on your use case. Based on my experience: If you're trying to build a representative model -- one that describes the data rather than necessarily predicts -- then I would suggest using a representative sample of your data. If you want to build a predictive model, particularly one that performs well by measure of AUC or rank-...


20

Max Kuhn covers this well in Ch16 of Applied Predictive Modeling. As mentioned in the linked thread, imbalanced data is essentially a cost sensitive training problem. Thus any cost sensitive approach is applicable to imbalanced data. There are a large number of such approaches. Not all implemented in R: C50, weighted SVMs are options. Jous-boost. ...


19

scale_pos_weight is used for binary classification as you stated. It is a more generalized solution to handle imbalanced classes. A good approach when assigning a value to scale_pos_weight is: sum(negative instances) / sum(positive instances) For your specific case, there is another option in order to weight individual data points and take their weights ...


16

Undersampling the majority class is usually the way to go in such situations. If you think that you have too few instances of the positive class, you may perform oversampling, for example, sample 5n instances with replacement from the dataset of size n. Caveats: Some methods may be sensitive to changes in the class distribution, e.g. for Naive Bayes - it ...


14

When use any sampling technique ( specifically synthetic) you divide your data first and then apply synthetic sampling on the training data only. After you train you use the testing set ( which contains only original samples) to evaluate. The risk if you use your strategy is to have the original sample in training ( testing) and the synthetic sample ( that ...


11

Yes, it's problematic. If you oversample the minority, you risk overfitting. If you undersample the majority, you risk missing aspects of the majority class. Stratified sampling, btw, is the equivalent to assigning non-uniform misclassification costs. Alternatives: (1) Independently sampling several subsets from the majority class and making multiple ...


11

I would recommend training on more balanced subsets of your data. Training random forest on sets of randomly selected positive example with a similar number of negative samples. In particular if the discriminative features exhibit a lot of variance this will be fairly effective and avoid over-fitting. However in stratification it is important to find balance ...


11

Gradient boosting is also a good choice here. You can use the gradient boosting classifier in sci-kit learn for example. Gradient boosting is a principled method of dealing with class imbalance by constructing successive training sets based on incorrectly classified examples.


11

Actually NLP is one of the most common areas in which resampling of data is needed as there are many text classification tasks dealing with imbalanced problem (think of spam filtering, insulting comment detection, article classification, etc.). But SMOTE seem to be problematic here for some reasons: SMOTE works in feature space. It means that the output of ...


10

Maybe try to encode your target values as binary. Then, this class_weight={0:1,1:2} should do the job. Now, class 0 has weight 1 and class 1 has weight 2.


10

This answer by @KeremT is correct. I provide an example for those who still have problems with the exact implementation. weight parameter in XGBoost is per instance not per class. Therefore, we need to assign the weight of each class to its instances, which is the same thing. For example, if we have three imbalanced classes with ratios class A = 10% class ...


9

Class imbalance is a very common problem. You can either oversample the positive class (or undersample the negative) or add class weights. Another thing to remember in this case is that accuracy is not a very useful metric here. You might consider AUC or F1 score. Changing your decision threshold may seem appealing, but will obviously lead to (in this ...


9

The Kappa is Cohen's Kappa score for inter-rater agreement. It's a commonly-used metric for evaluating the performance of machine learning algorithms and human annotaters, particularly when dealing with text/linguistics. What it does is compare the level of agreement between the output of the (human or algorithmic) annotater and the ground truth labels, to ...


9

I think it always depends on the scenario. Using a representative data set is not always the solution. Assume that your training set has 1000 negative examples and 20 positive examples. Without any modification of the classifier, your algorithm will tend to classify all new examples as negative. In some scenarios this is O.K. But in many cases the costs of ...


8

If you are looking for just an alternative loss function: Focal Loss has been shown on imagenet to help with this problem indeed. Focal loss adds a modulating factor to cross entropy loss ensuring that the negative/majority class/easy decisions not over whelm the loss due to the minority/hard classes. I would look into using that is it seems to be ...


8

The error here seems to be because you want train and test data (so two data sets), meaning that each class must be present in each of the data sets. This would mean that each class must have at least two samples. It is a design choice of whoever implemented train_test_split. I guess it might not technically be stratified otherwise. You can see where it is ...


7

Just assign each entry of your train data its class weight. First get the class weights with class_weight.compute_class_weight of sklearn then assign each row of the train data its appropriate weight. I assume here that the train data have the column class containing the class number. I assumed also that there are nb_classes that are from 1 to nb_classes. ...


7

You can use Multi-label data stratification in skmultilearn library http://scikit.ml/stratification.html


7

Class weights do help with the imbalance problem ("resolve" seems too much), but upsampling has a certain advantage on it. If you think about it, downsampling/upsampling the number of samples in each class to balance the dataset is almost exactly the same as using class weights. For example, say you have a dataset containing 3 samples divided into 2 ...


6

There always is the solution to try both approaches and keep the one that maximizes the expected performances. In your case, I would assume you prefer minimizing false negatives at the cost of some false positive, so you want to bias your classifier against the strong negative prior, and address the imbalance by reducing the number of negative examples in ...


6

I suspect the reason is that the class balance in your test set is different from the class balance in your training set. That will throw everything off. The fundamental assumption made by statistical machine learning methods (including logistic regression) is that the distribution of data in the test set matches the distribution of data in the training ...


6

Imagine that your data is not easily separable. Your classifier isn't able to do a very good job at distinguishing between positive and negative examples, so it usually predicts the majority class for any example. In the unbalanced case, it will get 100 examples correct and 20 wrong, resulting in a 100/120 = 83% accuracy. But after balancing the classes, the ...


5

Separate the operational and the training scenarios. The operational scenario is the one in which your classifier will be measure on. This is where you should perform well. Use should have a dataset that is representative of this scenario. The training scenario is whatever you are doing in order to build a classifier that will perform well on the ...


5

Everyone stumbles upon this question when dealing with unbalanced multiclass classification problem using XGBoost in R. I did too! I was looking for an example to better understand how to apply it. Invested almost an hour to find the link mentioned below. For all those who are looking for an example, here goes. Thanks wacax


5

Actually what they mentioned is right. The idea of oversampling is right and is one of, in general, Resampling methods to cope with such problem. Resampling can be done through oversampling the minorities or undersampling the majorities. You may have a look at SMOTE algorithm as a well-stablished method of resampling. But about your main question: No it's ...


5

When choosing the validation set and the test set, it is important that it reflects the actual "production environment" of your problem. Since you have "out of time" validation sets, I assume you have some time structure in your data that you need to address when making predictions. If you are developing your model, not taking this time aspect into ...


5

If you can change the Loss function of the algorithm, It will be very helpful. There are many useful metrics which were introduced for evaluating the performance of classification methods for imbalanced data-sets. Some of them are Kappa, CEN, MCEN, MCC, and DP. If you use python, PyCM module can help you to find out these metrics. Here is a simple code to ...


5

SMOTE is an algorithm for generating as many minority samples as you like. Thus, you can generate as many samples as you want.


4

A fast, easy an often effective way to approach this imbalance would be to randomly subsample the bigger class (which in your case is the negative class), run the classification N number of times with members from the two classes (one full and the other subsampled) and report the average metric values, the average being computed over N (say 1000) iterations. ...


4

you need to distinguish between these cases: Data Imbalance Data Imbalance + Very few number of samples (minority class) Severe Data Imbalance + Very few number of samples (minority class) 20:60 vs. 10:20 vs. 100:1000 vs. 10:100 and these cases: similarities between different classes. wide variations within the same class. You need to understand ...


Only top voted, non community-wiki answers of a minimum length are eligible