using sklearn class weight to increase number of positive guesses in extremely unbalanced data set?

Question

Hi I have a poorly correlated and unbalanced data set I have to work with. The set is 2 classes, 0 has 96,000 values and 1 has about 200. When I run random forest or other methods I get an output like:

    precision    recall  f1-score   support

          0       1.00      1.00      1.00     38300
          1       1.00      0.01      0.02        90

avg / total       1.00      1.00      1.00     38390

Precision is very high but it only classified one row as positive?

I tried using {class_weight = 'balanced'} in the random forest parameters and it provides:

   micro avg       1.00      1.00      1.00     38390
   macro avg       1.00      0.51      0.51     38390
weighted avg       1.00      1.00      1.00     38390

But still not many positive guesses? Should I look into oversampling?

Answer 1

You can try to compute class weights and assign these values to model via weight classes function. One more reminder about weights; probably major classes weight will be less than 1 so you need to round it to 1 otherwise model won't learn major class this time.

Answer 2

If you can change the Loss function of the algorithm, It will be very helpful. There are many useful metrics which were introduced for evaluating the performance of classification methods for imbalanced data-sets. Some of them are Kappa, CEN, MCEN, MCC, and DP.

If you use python, PyCM module can help you to find out these metrics.

Here is a simple code to get the recommended parameters from this module:

>>> from pycm import *

>>> cm = ConfusionMatrix(matrix={"Class1": {"Class1": 1, "Class2":2}, "Class2": {"Class1": 0, "Class2": 5}})  

>>> print(cm.recommended_list)
["Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN", "MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC", "AUCI", "G", "DP", "DPI", "GI"]

After that, each of these parameters you want to use as the loss function can be used as follows:

>>> y_pred = model.predict      #the prediction of the implemented model

>>> y_actu = data.target        #data labels

>>> cm = ConfusionMatrix(y_actu, y_pred)

>>> loss = cm.Kappa             #or any other parameter (Example: cm.SOA1)

Answer 3

You could try to do some resampling to your data in order to create a balanced training set for the training your classification model. This can be done in python with SMOTE - you can find some implementation example here.

Also, have a look to this question and answers, discussing ways to train and validate classification models from unbalanced datasets, eliminating the bias as much as possible.