Questions tagged [class-imbalance]

Questions referring to classifiers or classifying problems where some of the classes in the data are under-represented.

Filter by
Sorted by
Tagged with
4
votes
2answers
418 views

How to choose best classifier for Low positive to negative class ratio in data (training, validation and real time)?

Positive class is ~4%. The class weight methods will not work as even if I balance the data while training by scaling positive class samples, in real time (or in test data), the distribution is ...
2
votes
0answers
736 views

class imbalance - applied SMOTE - next steps

I am new to ML and learnt a lot from your valuable posts. I need your advise with the following situation and guidance on if the steps make sense. I have a binary classification problem, my dataset ...
0
votes
1answer
77 views

Subset of training set produces good results while full training set produces poor results

I have an extremely unbalanced data set: around 200 positive samples and 70,000 negative samples. To overcome this problem I have tried to over-sample the minority class as suggested in previous ...
4
votes
4answers
3k views

oversampling plus down sampling using smote not working on random forests

I am trying to solve a classification problem on a highly imbalanced data set. I am using SMOTE to over sample the minority samples and down sample the majority ones. After creating a balanced data ...
12
votes
3answers
18k views

Unbalanced classes -- How to minimize false negatives?

I have a dataset that has a binary class attribute. There are 623 instances with class +1 (cancer positive) and 101,671 instances with class -1 (cancer negative). I've tried various algorithms (Naive ...
3
votes
1answer
2k views

Which accuracy metric of a ML classifier can maximize map@K of a recommender system for an unbalanced dataset?

I have to build a recommender system & it will be evaluated using map@10 criteria. I have rolled up the data/rows at user-item level & is using Gradient Boosting in scikit learn to build the ...
3
votes
2answers
10k views

Ratio of positive to negative sample in data set for best classification

Suppose I have 100 positive samples. How many negative samples do I need to have in order to make the classifier work the best. In many papers, I have noticed that they take 4 times or 5 times the ...
2
votes
2answers
2k views

Tune hyperparameters for cost-sensitive classification

I have an unbalanced data set with about 8% of negative examples. The goal is to minimize false negatives given a cost matrix. It seems like SVM (with radial kernel) and random forest work best. How ...
7
votes
4answers
1k views

Training and testing AdaBoost for low probability classification

I have a dataset that I want to classify as fraud/not fraud and I have many weak learners. My concern is that there is much more fraud than not fraud, so my weak learners perform better than average, ...
3
votes
1answer
99 views

What are the possible ways to handle class unbalance in a large scale image recognition problem with Deep Neural Nets?

I have 22 classes of objects but they have very skewed distributions where max class has 100.000 images and the min class has 1600 images. In that setting I would like to hear some possible solutions ...
3
votes
4answers
2k views

How to learn a classifier from a dataset with high imbalance

What are the most useful techniques for learning a binary classifier from a dataset with a high degree of imbalance (i.e., a dataset with the "target" class being much rarer than the "background" ...
3
votes
1answer
207 views

Modeling when the response variable has too many 0's and few continuous values?

For problems where the data represents online fraud or insurance (where each row represents a transaction), it is typical for the response variable to denote the value of fraud committed in dollars. ...
6
votes
2answers
4k views

Kappa near to 60% in unbalanced (1:10) data set

As mentioned before, I have a classification problem and unbalanced data set. The majority class contains 88% of all samples. I have trained a Generalized Boosted Regression model using ...
34
votes
4answers
15k views

Quick guide into training highly imbalanced data sets

I have a classification problem with approximately 1000 positive and 10000 negative samples in training set. So this data set is quite unbalanced. Plain random forest is just trying to mark all test ...
53
votes
6answers
14k views

Should I go for a 'balanced' dataset or a 'representative' dataset?

My 'machine learning' task is of separating benign Internet traffic from malicious traffic. In the real world scenario, most (say 90% or more) of Internet traffic is benign. Thus I felt that I should ...
15
votes
4answers
2k views

What are the implications for training a Tree Ensemble with highly biased datasets?

I have a highly biased binary dataset - I have 1000x more examples of the negative class than the positive class. I would like to train a Tree Ensemble (like Extra Random Trees or a Random Forest) on ...

1
6 7 8 9
10