1
$\begingroup$

I have created two simulated random dataset of 3 classes. Only difference between the dataset is that frequency of the classes.

Dataset A: (Class 0 = 300, Class 1 =200, Class 2 = 500)
Dataset B: (Class 0 = 500, Class 1 =500, Class 2 = 500)

Both are random dataset so I should expect from Logistic regresssion model to confuse between each class with equal frequencies. That means in the normalized confusion matrix I should expect equal fraction of confusion among all three classes.

Confusion matrix of Dataset A

enter image description here

Confusion matrix of Dataset B 

enter image description here

My expectation from dataset A is same as dataset B. But I am not able to achieve that. why? I am using the following command in python to run the logistic regression model.

log_reg_model = LogisticRegression(C=1,penalty='l1',multi_class='ovr',class_weight='balanced',solver='liblinear')
pipe=Pipeline([('StandardScaler',StandardScaler()), ('logistic_regression',log_reg_model)])

Edit: I am uploading the both dataset in following dropbox link. https://www.dropbox.com/sh/pkiapvqy3k3f12v/AADpeBJ0XTWA2v9MCjALBcexa?dl=0 First column is index, second column is class id, third to fifth are class features.

$\endgroup$
2
  • $\begingroup$ Can you shed some light on the dataset? Are both exactly identical? I generally work mostly with R, but can try to understand what is happening here and help if possible. $\endgroup$ Sep 22, 2021 at 13:09
  • $\begingroup$ @Shibaprasadb I uploaded the both dataset in above dropbox link. If you have any answer of this imbalance behaviour then please let me know. $\endgroup$ Sep 22, 2021 at 13:59

3 Answers 3

2
$\begingroup$

If your datasets are random (with no real connection between the class and predictive variables), then "the right" model is a constant one: in (A), the predicted probabilities should be roughly $0.3, 0.2, 0.5$, whereas in (B) they should be $0.33, 0.33, 0.33$. When making the hard classifier then, in (A) the maximum probability will nearly always be the third class, whereas in (B) each class should be predicted roughly equally, at random. Of course there will be some deviation, because your model will try to learn some pattern in its randomly selected training set, but you should expect something very much like what you've shown.

In an imbalanced dataset, you may very well not want to just choose the most-likely class. If you had a more predictive set of features that might be OK, because the model may be able to learn enough to overcome the prior probabilities, but here with unpredictive features, nothing new can be learned.

$\endgroup$
2
  • $\begingroup$ Thanks for the explaination. I got your point. So my model is correct for the imbalanced dataset or do I need to play with something? $\endgroup$ Sep 22, 2021 at 15:10
  • $\begingroup$ If it were a real dataset, you'd probably want to see if you could coax more information into your model, and you should generally consider costs of the different types of misclassification when determining actions based on predicted probabilities. $\endgroup$
    – Ben Reiniger
    Sep 22, 2021 at 15:45
0
$\begingroup$

I am assuming, your focus here is the prediction accuracy and not interpretability?

So, as there is a class imbalance, you can do two things:

  1. As suggested by the other user, you can use SMOTE or any technique.
  2. Use a non-parametric method that is more robust in handling the class imbalance.

I tried to use Random Forest on your data, and the classification result was already promising without any parameter tuning. A R code which I used:

library(randomForest)

data_A$V2 = factor(data_A$V2, levels = c(0, 1,2))

set.seed(4)
classifier_random <- randomForest(V2~V3+V4+V5, data=data_A, ntree=500)
pred_forest <- predict(classifier_random, data_A[,c('V3','V4','V5')])

table(data_A$V2, pred_forest)

pred_forest
      0   1   2
  0 297   0   3
  1  13 180   7
  2   9   7 484

What if I divide it into Training and Testing set with an 80:20 ratio?

smp_size <- floor(0.80 * nrow(data_A))

set.seed(4)
train_ind <- sample(seq_len(nrow(data_A)), size = smp_size)

train <- data_A[train_ind, ]
test <- data_A[-train_ind, ]

classifier_random <- randomForest(V2~V3+V4+V5, data=train, ntree=500)
pred_forest <- predict(classifier_random, test[,c('V3','V4','V5')])

table(test$V2, pred_forest)

pred_forest
     0  1  2
  0 17 11 33
  1 14  5 24
  2 16 13 67

Misclassification is more but still, it is working, that too without any tuning.

The code is in R which might not be totally beneficial for you, but I hope you get the point that I am trying to convey here.

$\endgroup$
-1
$\begingroup$

You can look into SMOTE & ADAYSN techniques. This will help you in reducing the imbalance in the dataset by creating synthetic data

https://medium.com/coinmonks/smote-and-adasyn-handling-imbalanced-data-set-34f5223e167

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.