0
$\begingroup$

Say if you have a balanced dataset, with two classes, if the classification model that we’re training doesn’t learn anything ( suppose the data is random ), the model’s output would be 50% first class and 50% the second class. So by that result I would know my model is predicting randomly

But what if my classes are imbalanced? Let’s say 20% for label A and 80% for label B.

How can I check if my model is predicting randomly? If we will predict only class A we will get 80% accuracy but it doesn’t mean the output is random. How should the output’s percentages look like? All in all, what is a random prediction for class imbalanced data?

But mainly I want to train a model on imbalanced MULTIcass data (53 labels ), and I want to make sure that my model’s predictions are completely random and it features doesn’t provide any information about the label. What does a random output result look like in this case?

$\endgroup$

2 Answers 2

1
$\begingroup$

In a multiclass classification problem with 53 labels, let's say your model is making completely random predictions, it should assign approximately equal probability to each class. This means the output probabilities for each class would be around 1/53 (or about 1.89%) for every class.

To check if your model is predicting randomly or not, then you might try using cross-validation technique. If there's high variance in accuracy scores across different folds of data, it might indicate that your model is not stable and its predictions could be highly dependent on specific characteristics of the training data, which can sometimes happen when a model is effectively making random guesses rather than learning meaningful patterns from features. For e.g.if one round of cross-validation gives an accuracy score of 80%, but another gives only 50%, this inconsistency could suggest that your model isn't reliably learning from your features.

Another way to check is to use confusion matrix or classification report. If you get low precision and recall score over most of the classes, then it means that your model might not be reliable.

$\endgroup$
3
  • $\begingroup$ Thanks for your answer:) for the first part, what if my data is imbalanced? What percentages should I expect? $\endgroup$ Aug 17, 2023 at 10:56
  • $\begingroup$ And for the CV part, it’s true if I use that it can tell me, my model is unreliable. But I want to know it hasn’t learned any patterns. Not being reliable doesn’t really show that my model’s output is completely random. I just want to make sure it hasn’t learned any patterns $\endgroup$ Aug 17, 2023 at 10:58
  • $\begingroup$ it's for the inbalanced dataset $\endgroup$ Aug 17, 2023 at 11:08
1
$\begingroup$

... I want to make sure that my model’s predictions are completely random and it features doesn’t provide any information about the label.

This is a rare request, hopefully I don't get it wrong.

The most obvious, straight-forward way is a dummy classifier, which looks like a normal model but is in fact producing random predictions without looking at features.

Similarly, an easy test to ensure a model giving random prediction is to ask it to predict for a same sample multiple times. A normal model should give a consistent answer, while a random model would give different 'predictions' from time to time.

$\endgroup$
3
  • $\begingroup$ Thanks for the answer:) yea if I try it for a specific class and it gives me diff Outputs it shows the model’s prediction is probably random. But what percentages should I get when I test the model? To be sure it’s completely random. My goal is to make sure the model is not learning any patterns. $\endgroup$ Aug 17, 2023 at 11:00
  • $\begingroup$ Actually you do not use the accuracy to check for randomness, just observing the predictions alone is sufficient (think about how we test any variable for randomness). Apply the model on the same data set multiple times, and if you see both 1) different runs on same sample gives different result; 2) the predictions follow a uniform distribution, then it is random. Formally, you will use standard statistical test just as we test whether a dice is fair. $\endgroup$
    – lpounng
    Aug 18, 2023 at 1:35
  • $\begingroup$ One note: it is essential that the predictions follow a uniform distribution i.e. each possible class gets same chance to come up. Though a random variable with a prior is also random, in your case the predictions should not know anything about the prior i.e. how the labels are distributed. $\endgroup$
    – lpounng
    Aug 18, 2023 at 1:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.