40

I would say the answer depends on your use case. Based on my experience: If you're trying to build a representative model -- one that describes the data rather than necessarily predicts -- then I would suggest using a representative sample of your data. If you want to build a predictive model, particularly one that performs well by measure of AUC or rank-...


24

Max Kuhn covers this well in Ch16 of Applied Predictive Modeling. As mentioned in the linked thread, imbalanced data is essentially a cost sensitive training problem. Thus any cost sensitive approach is applicable to imbalanced data. There are a large number of such approaches. Not all implemented in R: C50, weighted SVMs are options. Jous-boost. ...


22

scale_pos_weight is used for binary classification as you stated. It is a more generalized solution to handle imbalanced classes. A good approach when assigning a value to scale_pos_weight is: sum(negative instances) / sum(positive instances) For your specific case, there is another option in order to weight individual data points and take their weights ...


20

Undersampling the majority class is usually the way to go in such situations. If you think that you have too few instances of the positive class, you may perform oversampling, for example, sample 5n instances with replacement from the dataset of size n. Caveats: Some methods may be sensitive to changes in the class distribution, e.g. for Naive Bayes - it ...


20

When use any sampling technique ( specifically synthetic) you divide your data first and then apply synthetic sampling on the training data only. After you train you use the testing set ( which contains only original samples) to evaluate. The risk if you use your strategy is to have the original sample in training ( testing) and the synthetic sample ( that ...


13

Gradient boosting is also a good choice here. You can use the gradient boosting classifier in sci-kit learn for example. Gradient boosting is a principled method of dealing with class imbalance by constructing successive training sets based on incorrectly classified examples.


13

This answer by @KeremT is correct. I provide an example for those who still have problems with the exact implementation. weight parameter in XGBoost is per instance not per class. Therefore, we need to assign the weight of each class to its instances, which is the same thing. For example, if we have three imbalanced classes with ratios class A = 10% class ...


13

Actually NLP is one of the most common areas in which resampling of data is needed as there are many text classification tasks dealing with imbalanced problem (think of spam filtering, insulting comment detection, article classification, etc.). But SMOTE seem to be problematic here for some reasons: SMOTE works in feature space. It means that the output of ...


13

Class weights do help with the imbalance problem ("resolve" seems too much), but upsampling has a certain advantage on it. If you think about it, downsampling/upsampling the number of samples in each class to balance the dataset is almost exactly the same as using class weights. For example, say you have a dataset containing 3 samples divided into 2 ...


13

You should always do your evaluation of model performance on data that has not been over/undersampled. You can setup a pipeline with scikit-learn to perform your undersampling on the training set and then evaluate on the non-undersampled fold of data for each iteration as you described.


11

Yes, it's problematic. If you oversample the minority, you risk overfitting. If you undersample the majority, you risk missing aspects of the majority class. Stratified sampling, btw, is the equivalent to assigning non-uniform misclassification costs. Alternatives: (1) Independently sampling several subsets from the majority class and making multiple ...


11

I would recommend training on more balanced subsets of your data. Training random forest on sets of randomly selected positive example with a similar number of negative samples. In particular if the discriminative features exhibit a lot of variance this will be fairly effective and avoid over-fitting. However in stratification it is important to find balance ...


11

Just assign each entry of your train data its class weight. First get the class weights with class_weight.compute_class_weight of sklearn then assign each row of the train data its appropriate weight. I assume here that the train data has the column class containing the class number. I assumed also that there are nb_classes that are from 1 to nb_classes. ...


11

If you are looking for just an alternative loss function: Focal Loss has been shown on imagenet to help with this problem indeed. Focal loss adds a modulating factor to cross entropy loss ensuring that the negative/majority class/easy decisions not over whelm the loss due to the minority/hard classes. I would look into using that is it seems to be ...


11

The error here seems to be because you want train and test data (so two data sets), meaning that each class must be present in each of the data sets. This would mean that each class must have at least two samples. It is a design choice of whoever implemented train_test_split. I guess it might not technically be stratified otherwise. You can see where it is ...


11

The choice of a metric depends on how you rank the importance of your classes and what you value from a classifier. Let's look at your example: For example if we have a data set with 90%-10% class distribution then a baseline classifier can achieve 90% accuracy by assigning the majority class label. One minor correction is that this way you can achieve a ...


10

Maybe try to encode your target values as binary. Then, this class_weight={0:1,1:2} should do the job. Now, class 0 has weight 1 and class 1 has weight 2.


10

I think subsampling (downsampling) is a popular method to control class imbalance at the base level, meaning it fixes the root of the problem. So for all of your examples, randomly selecting 1,000 of the majority of the class each time would work. You could even play around with making 10 models (10 folds of 1,000 majority vs the 1,000 minority) so you ...


10

Is this approach better than the mere augmentation or just the use of class weights ? Note that data augmentation is the process of changing the training samples (e.g. for images, flipping them, changing their luminosity, adding noise, etc.) and adding them back into the set. It is used for enriching the diversity of training samples, thus, in this aspect ...


9

Class imbalance is a very common problem. You can either oversample the positive class (or undersample the negative) or add class weights. Another thing to remember in this case is that accuracy is not a very useful metric here. You might consider AUC or F1 score. Changing your decision threshold may seem appealing, but will obviously lead to (in this ...


9

The Kappa is Cohen's Kappa score for inter-rater agreement. It's a commonly-used metric for evaluating the performance of machine learning algorithms and human annotaters, particularly when dealing with text/linguistics. What it does is compare the level of agreement between the output of the (human or algorithmic) annotater and the ground truth labels, to ...


9

I think it always depends on the scenario. Using a representative data set is not always the solution. Assume that your training set has 1000 negative examples and 20 positive examples. Without any modification of the classifier, your algorithm will tend to classify all new examples as negative. In some scenarios this is O.K. But in many cases the costs of ...


9

You should not expect class_weight parameters and SMOTE to give the exact same results because they are different methods. Class weights directly modify the loss function by giving more (or less) penalty to the classes with more (or less) weight. In effect, one is basically sacrificing some ability to predict the lower weight class (the majority class for ...


8

You need to deal with imbalanced data set when the value of finding the minority class is much higher than that of finding the majority. Let say that 1% of the population have that rare disease. Suppose that you assign the same cost to saying that a healthy man is sick or saying that a sick man is healthy. Provide a model that say that everybody are healthy,...


8

You can use Multi-label data stratification in skmultilearn library


8

First of all, just to be clear, you shouldn't evaluate the performance of your models on the balanced data set. What you should do is to split your dataset into a train and a test set with ideally the same degree of imbalance. The evaluation should be performed exclusively on the test set, while the balancing on the training set. As for your question, any ...


8

Your data set is unbalanced since 28432 out of 28481 examples belong to class 0 (that is 99.8%). Therefore, your predictor almost always predicts any given sample as belonging to class 0 and thereby achieves very high scores like precision and recall for class 0 and very low scores for class 1. In the case of weighted average the performance metrics are ...


7

You should use the testing set without any change, as answered by others. But it is very important to understand the difference between average accuracy and overall accuracy. In overall accuracy you find ( number of samples predicted correctly/ total number of samples) in average accuracy, you find the overall accuracy per class and then you find the ...


7

To see clearly why the procedure of upsampling before CV is mistaken and it leads to data leakage and other undesired consequences, it is useful to imagine first the simpler "baseline" case, where we simply upsample (i.e. create duplicate samples) without SMOTE. The first reason why such a procedure is invalid is that, this way, some of the ...


6

There always is the solution to try both approaches and keep the one that maximizes the expected performances. In your case, I would assume you prefer minimizing false negatives at the cost of some false positive, so you want to bias your classifier against the strong negative prior, and address the imbalance by reducing the number of negative examples in ...


Only top voted, non community-wiki answers of a minimum length are eligible