1
$\begingroup$

I am trying to use a multiclass classification using python. For that I used few algorthims like Random Forest, Xgboost, Logitic regression.

My problem is simple, I have users, Images, and people ratings on those images. I devided the ratings into 3 classes:

class 1: good marks

class 2: bad marks

class 3: medium evaluations

At first I got these results

I have 70% of bad marks (class2) 18% of good marks (class1) 10% of bad marks (class 3)

so all was good I did the classification and I got a good accuracy (75%).

I have collected more data and more (good marks) and the accuracy kept decreasing.

So I understood that the accuracy was good only because the algorithms where predicting that almost all the marks are bad, so basically only one class- class 2, and when I got more data in class 1 and class 3 the accuracy decreased.

This is example of the confusion matrix I was having

Predicted Marks     1    2  3
Actual Marks               
1                  48   85  3
2                  17  250  4
3                  10   89  1

I understood that I was getting this problem because I have unbalanced data, so I was predicting bascially the highest frequency class.

I did some researches and I found that there are option called `

class_weight='balanced' So I used it in the classification algorithms.

And I got this result

Predicted Marks   1    2   3
Actual Marks                
1                  53   61  29
2                  66  161  53
3                  24   40  17

So my question is:

  • First I know that the option balanced try to requilibrate the data but I don't understand how. I found in the explanations that it"considers each class as important as the other"` but still don't understand how. Does it duplicate rows of the minor categories ?

  • When I used that option, it was obvious that the classifier were predicting other Mark classes more frequently, but there are more number of right predictions than the wrong ones. ( example in the second confusion matrix the class 1 was predicted as class 2 66 times , but in the confusion matrix 1 only 17 times)

Am I analysing the problem right? and what are your suggestions for such a problem?

Sorry for making this long, any help will be appreciated !!

$\endgroup$
  • $\begingroup$ For all algorithm you specified are you getting output as single class? $\endgroup$ – AkshayNevrekar Aug 23 '18 at 10:20
  • $\begingroup$ yes , same behavior for all the algorithms $\endgroup$ – ch.E Aug 23 '18 at 12:09
0
$\begingroup$

I found in the explanations that it "considers each class as important as the other" but still don't understand how. Does it duplicate rows of the minor categories?

No, it adds a weight to each example, depending on its class. The majority class will be assigned a small weight while the minority ones will be assigned larger weights. This weight are considered during the training phase of your model, so that each example of a minority class impacts the parameter updates more than one from the majority class.

For example, suppose class 1 has a weight of $0.5$, class 2 has a weight of $1.2$ and class 3 has a weight of $1.5$. Each example from class 3 will impact the parameter updates 3 times more than an example from class 1, etc.

You can see what weights scikit-learn has selected through this function. You can also select your own weights (if you feel that some class should be more important than what scikit-learn has selected) by modifying the dictionary.

When I used that option, it was obvious that the classifier were predicting other Mark classes more frequently, but there are more number of right predictions than the wrong ones.

I'm not sure what the question is, but if you are asking "why do I have more misses now than before?".

Because you used that option, your model has learned to classify examples while treating the classes of equal importance. But the classes don't have the same frequencies. For example if you miss-classify 1 out of 4 examples, you would miss more predictions from the majority class than from the minority ones.

Is this good or bad?

Depending on your goal.

If you want to simply maximize your algorithm's correct predictions, then you might get a better result without balancing your classes. In this case you should use a metric like accuracy.

If your goal something else, for instance to have the same accuracy in each class, you should select a more appropriate metric (e.g. a macro-averaged one).

Is there an alternative to class_weights?

Yes, in most cases it is preferable to balance the number of samples in your classes. This is done by either over-sampling the minority classes, under-sampling the majority ones, or a combination of both.

If you want to try out this approach, I'd suggest using imbalanced-learn.

$\endgroup$
  • 1
    $\begingroup$ Thank you so much for your answer, your response was very clear. $\endgroup$ – ch.E Aug 23 '18 at 13:54
  • 1
    $\begingroup$ I tried the SMOTE oversampling it gave me a really good results on training set, but not good on the test set. I caluclated also the f1_score for each class, and their macro-averege. In fact my final purpose is to see the impact of some features on the predection accuracy. to be me more precise i have the age and gender of the users and I want to see if they have on impact. Then I want to recommand for each user an image. $\endgroup$ – ch.E Aug 23 '18 at 14:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.