1
$\begingroup$

I tried to do multi-class classification problem. The goal is to predict whether the match will be won by HomeTeam, AwayTeam or Draw. I did feature engineering from the attributes and finally came up with final data to train a classifier. I make sure that the data is balanced for all the 3 class.

To train a classifier I did XGB Classifier, Logistic Regression, SGD Classifier and Normal DNN(Tensorflow Estimator). I checked the metrics for all the classifiers and I am picking out the best one from the classifier.

Linear SGD Classifier Performance on Validation Set

     Class, Precision, Recall,    spe,       f1,      geo,      iba,      sup

      A       0.58      0.69      0.79      0.63      0.74      0.54       275
      D       0.51      0.61      0.66      0.55      0.63      0.40       338
      H       0.81      0.50      0.94      0.62      0.69      0.45       315
   Avg/mean   0.63      0.60      0.79      0.60      0.68      0.46       928        

Model Performance for Test Dataset

              pre       rec       spe        f1       geo       iba       sup

      A       0.87      0.55      0.97      0.67      0.73      0.51        84
      D       0.43      0.69      0.66      0.53      0.67      0.45        83
      H       0.80      0.69      0.86      0.74      0.77      0.58       139

We can see this model is stable over the class A and H but the precision is so poor for class D. I think because of a lack of feature the model is not performing well for class D. Though, I did several EDA and Feature Engineering to increase the recall for class D.

My question is, Is this model is considered stable?

$\endgroup$

2 Answers 2

0
$\begingroup$

A consideration: I dont think A is stable since it has a huge difference between validation and test results.

Some questions before answer:

  1. Are you using cross validating with how many folds? The best result are the mean between folds results? What is the stddev? If stddev is big, it can tell us a history.

  2. Your folds are shuffled? I see lot of people folding when using scikit learn without shuffle since its not the default behavior.

  3. Is your test data unbalanced? if so, its normal the difference between test and validation results.

  4. How many examples are you using for test and how many fot test? Sometimes the stability of the model requires more examples.

$\endgroup$
1
  • $\begingroup$ Yes, test data is unbalanced $\endgroup$
    – Madhi
    Jan 19, 2019 at 9:41
0
$\begingroup$

Looking at the f1_score of all the classes, I would say A and D are fine but the issue lies in H.The validation score for H and test score are miles apart and suggest the model is overfitting on the H class.

I would suggest the following:

  1. Increase the dataset size. Get as much data as you can, specially for the H class.
  2. Use RepeatedStratifiedKFold present in the sklearn library. Since it is a multiclass classification and that too imbalanced, it is very important that you use stratified cross validation in order to get accurate results.
  3. While splitting the data into train and test, use the stratified parameters as that will ensure that both the splits get even counts of all the classes.

Let me know if any of the above worked for you!

Cheers!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.