Hyperparameter Optimization for a Machine Learning Algorithm

I have a question regarding Hyperparameter Optimization for a Machine Learning Algorithm.

I try to fit a Support Vector Classifier and use Hyperparameter-Tuning (but it could be also another classifier).

My classes are highly imbalanced (20% of one class, lets call it “red” and 80% of the other, lets call it “black”).

Now, the objective of my Hyperparameter-Optimization is the cross-validation loss.

If, say, 20% of the observations are “reds” and 80% are “black”, then a really bad classifier would just label every case as “black” and achieve on average a cross-validation loss of 20%.

Now my question: If I see that the objective of my Hyperparamer-Optimization does not fall significantly below 20%, I could infer that it is useless without further analyzing. Correct? Or is there something I do not understand correctly?

This happens in all optimizations I am running now. I could conclude that my features are not informative.

I guess I have to change the objective for the hyperparameter optimization routine? (If my package allows for that)

It is likely the case that your relationship in data/features is not simple, so you need to allow your SVM model more flexibility and/or train for longer. This will of course bring in the danger of overfitting, but should improve things... you'll need to try out a few different things.

If you are using SciKit Learn, this might equate to using a radial basis function with a high value of argument C (giving much more flexibility to the model) and also try a higher value for the argument gamma, which will reduce the radius of influence of each individual data point. Check out this example for further explanation.

Think of the classic example of flipping an unbiased coin: we would expect to get 50% heads and 50% tails. So in this binary prediction, any time you make a prediction, you have 50-50 odds and so a model with more than 50% error is worse than a random guess!

If you are getting 20% error, it sounds like your model is just predicting black. A simple way to see and understand this - and why it really makes sense perhaps - would be to plot the data.

If in your feature speace, the points all look like a big mixture, a cloud, where the majority of points are black, then even a human would likely just predict black:

So two options:

1. Allow the model higher flexibility (as described above), which will let it (over-)fit a line that cleanly cuts out the sparse red points from the black
2. Get new features, or preprocess your features in such a way that a plot might end up looking something more like thi the image below, which will allow a simpler model to classify the red from the black with e.g. a straight line.

You are right, what has to be changed is the objective. You are currently using accuracy as a measure of how good your classifier is. Accuracy is not a good measure when you have class imbalance. For this reason, other objectives have to be used. AUC is a better measure, and so is the log-loss, although not that interpretable. However, both AUC and log-loss need you method to output probabilities, not just predictions of classes. For this reason, it is required that your SVM implementation has a predict probability method.

• Thanks a lot, unfortunately I can "accept" only one answer here on stackexchange Jun 15 '18 at 9:02
• So, my hyperparameter-search function will need AUC or log-loss as an objective. So maybe I have to adjust the code of my package accordingly? I will look whether Python/scikit has this option. Matlab may not have this... Jun 15 '18 at 9:06
• I am sure sklearn has this option on GridSearchCV. Jun 15 '18 at 9:10

Measuring the AUC of the ROC curve provides a good measure because it allows you to evaluate the probability that a model correct classifies input vs the probability that it incorrectly classifies, as opposed to simply evaluating the number of items that were correctly classified in the training data.

There are a few tools that can help resolve this issue while searching for a good model, including one in the imblearn library found here, which allows you to randomly undersample the dominant class so the disparity isn't so high (i.e. 80%).