Questions tagged [imbalanced-data]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1 vote
0 answers
9 views

Model a classification problem with multiple categorical varialbes as input features only. Diff Model performance

I'm having an input data with 100k rows, 8 input features, I'm trying to predict y (binary 1/0). But all the X are categorical variables(strictly nominal variables, not ordinal). Some with 8 levels, ...
user avatar
  • 11
4 votes
1 answer
61 views

How to tackle imbalanced regression?

I've recently encountered a problem where I want to fit a regression model on data that's target variable is like 75% zeroes, and the rest is a continuous variable. This makes it a regression problem, ...
user avatar
  • 1,248
0 votes
0 answers
33 views

Complex balanced dataloading from multiple imbalanced datasets?

The Setting Let's suppose that I have an imbalanced dataset. For training purposes, I want to implement a dataloading scheme that samples from this dataset in a more balanced way. I want to leverage ...
user avatar
2 votes
3 answers
79 views

Measuring performance of customer purchase predictions

My goal is to develop a model that predicts next customer purchases in USD (Update: During the time period of the dataset, if no purchase was made by the customer, the next purchase label is set to ...
user avatar
1 vote
1 answer
20 views

How to define minority/majority class in a multi-classification task

I am studying classification in imbalanced datasets and I am learning under/over sampling strategies as a way to address the issue. While the literature agrees one needs to oversample 'minority' ...
user avatar
  • 111
1 vote
1 answer
28 views

Class imbalance: Will transforming multi-label (aka multi-task) to multi-class problem help?

I noticed this and this questions, but my problem is more about class imbalance. So now I have, say, 1000 targets and some input samples (with some feature vectors). Each input sample can have label ...
user avatar
0 votes
0 answers
8 views

Should I use "sample_weights" on a calibrator if I already used them while training the model (imbalanced dataset)?

I was wondering what is the right way to proceed when you are dealing with an imbalanced dataset and you want to use a calibrator. When I work with a single model and imbalanced datasets I usually ...
user avatar
0 votes
0 answers
5 views

Can dataset with numeric (cardinal) dependent variable be unbalanced?

can a dataset be unbalanced if the dependent variable is numerical (cardinal scale)? Or does the question whether a dataset is (un)balanced only matter for datasets with a categorical dependent ...
user avatar
  • 1
1 vote
2 answers
72 views

Why does class_weight usually outperform SMOTE?

I'm trying to figure out what exactly class_weight from sklearn does. When working with imbalanced datasets, I'm always using ...
user avatar
  • 151
1 vote
1 answer
19 views

Over-sampling when predicting a contionuous variable

Lets say i am predicting house selling prices (continuous) and therefore have multiple independent variables (numerical and categorical). Is it common practice to balance the dataset when the ...
user avatar
  • 11
0 votes
2 answers
18 views

Rough ideas of expected performance boost from over-sampling techniques?

I am trying to train a classifier for a multi class classification task. However, the dataset is very imbalanced. About half of the around 160 unique labels are such that there are only 10 or less ...
user avatar
  • 103
1 vote
0 answers
50 views

Can an Imbalanced Datset be an oportunity for Transfer Learning with Neural Networks?

While solving classification tasks on imbalanced datasets with Neural Networks(NN) there are two general ways of handling imbalanced data: A. Resample the data, either with over or undersampling ...
user avatar
2 votes
2 answers
163 views

Influence of imbalanced feature on prediction

I want to use XGB regression. the dataframe is coneptually similar to this table: ...
user avatar
  • 339
1 vote
1 answer
126 views

Improving text classification & labeling in imbalanced dataset [closed]

I am trying to classify text titles (NLP) in categories. Let us say I have 6K titles that should fall into four categories. My questions: I do not understand why in some ML techniques categories are ...
user avatar
  • 113
2 votes
2 answers
278 views

Determining if a dataset is balanced

I'm learning about training sets and I have been provided with a set of labelled customer data that segments customers into one of two classes: A or B. The dataset also contains gender, age and ...
user avatar
0 votes
1 answer
25 views

Aren't balanced data sets important in regression?

Why is it that the necessity for balanced data sets is (almost) always exclusively mentioned in the context of classification but not of regression?
user avatar
  • 175
1 vote
1 answer
89 views

How to increase the accuracy of an imbalanced dataset (not precision)?

There's an imbalanced dataset in a Kaggle competition I'm trying. The target variable of the dataset is binary and it is biased towards 0. 0 - 70% 1 - 30% I tried several machine learning algorithms ...
user avatar
0 votes
1 answer
96 views

My semantic segmentation model classifies everything as background

So, I am working on a semantic segmentation task using U-Net. The dataset is very unbalanced, with the background being by far the most common, and the last class being very scarce. First I trained it ...
user avatar
  • 1
0 votes
1 answer
74 views

Why is the f1 score of my imbalanced data for a multiclass problem so low?

I am dealing with a multi-class problem and imbalanced data. I am trying to find an algorithm that can predict well each class with python (sklearn and pandas). My dataset contains: 620 rows, 12 ...
user avatar
  • 1
0 votes
0 answers
30 views

What do you suggest to increase the sensitivity rates in conventional ML model?

I have an imbalanced data problem (prop. rate: 0.8571429 0.1428571) and for this reason, our sensitivity and PPV rates are very low. What do you recommend to fix this problem in R or in general? See ...
user avatar
0 votes
0 answers
69 views

How to apply class weights for imbalanced data and given misclassification costs?

Say I have a dataset for binary classification with 1000 samples in the minority class and 100000 samples in the majority class and use XGBoost to output probabilities. From an expert I know that ...
user avatar
  • 1
0 votes
1 answer
78 views

What is the best practice to normalize/standardize imbalanced data for outlier detection or binary classification task?

I'm researching Anomaly/outlier/fraud detection, and I'm looking for the best practice to pre-process the synthetic data for imbalanced data. I have checked all methodology for normalizing/...
user avatar
  • 369
2 votes
1 answer
68 views

On what threshold we should resample the data?

When working with churn datasets, we usually find imbalanced datasets. My question is how to decide on what basis we should resample the data. For example: while splitting the data before training we ...
user avatar
2 votes
0 answers
39 views

Is balancing class data for imbalanced problems helpful or just folklore when considering thresholds?

Caveat: I'm aware that imbalanced data questions are a dead horse, but I haven't found an answer to this flavor of it directly. When working with highly imbalanced data (e.g. binary class cases), ...
user avatar
  • 141
1 vote
1 answer
118 views

Can the attention mask hold values between 0 and 1?

I am new to attention-based models and wanted to understand more about the attention mask in NLP models. attention_mask: an optional torch.LongTensor of shape [...
user avatar
  • 157
1 vote
1 answer
43 views

Dealing with high frequency tokens during masked Language modelling?

Suppose I am working with a Masked Language Model to pre-train on a specific dataset. In that dataset, most sequences have a particular token of a high frequency ...
user avatar
  • 157
5 votes
2 answers
129 views

Doesn't over(/under)sampling an imbalanced dataset cause issues?

I'm reading a lot about how to use different metrics specifically for imbalanced datasets (e.g. two classes present, but 80% of the data is one class) and how to tackle the issue of imbalanced ...
user avatar
  • 1,248
0 votes
2 answers
53 views

Do we have any method which handles the imbalanced classification with sample weighting instead of class weighting?

I am looking for methods that use sample weighting instead of class weighting for the imbalanced classification. I think sample weighting is more precise than weighting all the samples from one class ...
user avatar
8 votes
1 answer
276 views

Categorization of approaches to deal with imbalanced classes

What is the best way to categorize the approaches which have been developed to deal with imbalance class problem? This article categorizes them into: Preprocessing: includes oversampling, ...
user avatar
  • 1,205