# Evaluating performance of classifier on lopsided dataset

I have a binary classifier that I would like to evaluate the performance of. It's been both trained and tested on a data set where the ratio of true to false labels is lopsided. This means that while it's quite poor at correctly guessing true, its overall performance on the test set looks very good when using a metric such as right_guesses/total.

What is a better metric to use? Preferably, one where the true false labels account for the same percentage of the score although their numbers are unequal.

sklearn has weighted accuracy score which works just fine:

sklearn.metrics.balanced_accuracy_score()


It's better to use the F-score(F-measure) in this case to evaluate youre model. To calculate the F-score you can use the following equation:

$$\textrm{ F score} = \frac{(2 * Precision * Recall) }{ (Precision + Recall)} = \frac{tp}{tp+\frac{1}{2}(fp+fn)}$$

where:

Dont use accuracy as you may have a high accuracy without correctly classifying the majority of a class if you have an imbalanced dataset.

This is an article if you'd like to read more on imbalanced data.

If you are working on a real application, then the way in which to evaluate performance depends on what is important for the operational use of the model. For many practical problems the misclassification costs for false-positive and false-negative errors are different (e.g. for a medical screening test a false-positive is likely to be a much less serious error than a false-negative as a false-negative may result in someone not getting treated when they really need it, whereas a false-positive just results in wasting the cost of a more expensive test). So if you know the operational misclassification costs, then I would use the expected loss as a performance measure.

"Preferably, one where the true false labels account for the same percentage of the score although their numbers are unequal."

The balanced error rate (suggested by @Mikkel_brunn) is the correct solution to this question as posed, but the question is why is this important (given that it is effectively treating false-positives as more important than false-negatives)?

Any single accuracy-based statistic (e.g. F-score) is making assumptions about the importance of false-positives and false-negatives. Another approach is to consider the Receiver Operating Characteristic, and especially the Area Under the ROC (AUROC), which effectively gives the probability of a randomly chosen positive pattern being ranked more highly by the classifier than a randomly chosen negative pattern. This gives a measure of the skill of the classifier in ranking the patterns (we can tune the threshold depending on the relative importance of the classes later).

Lastly, I find probabilistic classifiers to be useful for problems with class imbalance or for problems with disparate training set and operational class frequencies, as it is easy to tune the classifier after training. A measure of the calibration of the probability estimates is also useful (e.g. negative log-likelihood - but that can be brittle).

So in short, rather than have just one performance measure, I'd go for an accuracy based statistic, a ranking based statistic, and for probabilistic classifier, a probability calibration statistic, as all can be informative.

For basic accuracy, you could measure the improvement in accuracy over that of a classifier that just classifies everything as belonging to the majority class and then normalise to lie in the range 0-1.