1
$\begingroup$

Considering a balanced training set, I noticed that the results of a classification primarily depend on the class imbalance of the test set.

As shown in this article, unless the classes are perfectly separable, the performance (precision & recall) of a model for a given class will always decrease based on the imbalance of the class. Ie: the more the test set is imbalanced, the less the model is capable of classifying the minority class.

This means that for any given model, the classification performance will always primarily depend on the balance of the data you are testing it with.

How can the imbalance of a test set define the predictive capabilities of my model once it is already trained? Does the performance of a classifier always depend on the class balance of the target population? What is the mathematical reasoning behind this?

$\endgroup$
1
  • $\begingroup$ It's Bayes' theorem. For an example of distinguishing photos of dogs from photos of cats: $P(\text{dog}\vert\text{photo}) = \dfrac{P(\text{photo}\vert\text{dog})P(\text{dog})}{P(\text{photo)}}$. The class proportion is the prior distribution, $P(\text{dog})$. $\endgroup$
    – Dave
    Aug 11 at 18:22
0
$\begingroup$

There is a confusion between the "true" performance of the classifier, which is indeed fixed once the classifier is trained, and the observed performance on a particular test set.

The "true" performance can only be estimated, and it should be estimated using a random sample which follows the "true" distribution of the data. Supervised learning always assumes a "true population" and both the training set and test set are supposed to be subsets of this true population.

If one uses a the test set with a different distribution then there's no guarantee that the performance will be the same as the true performance. This can be relevant in some experiments, but it's not a proper evaluation of the classifier itself.

Intuitively this can be compared to a test given to some students after they have studied some exercises:

  • If the test questions are similar to the questions they studied in the exercises then their mark reflects their true performance, i.e. how well they learned from the exercises.
  • If the professor makes the test with questions which are not seen or rarely seen in the exercises, then the mark is likely to be lower even for the good students.

Edit: study of the specific case of balanced training set vs. imbalanced test set (asked by OP in comments).

Re-edited after fixing mistake found by OP

This is an interesting case to study, thanks for asking :)

Using your code as a basis I tested the following code:

import matplotlib.pyplot as plt
from sklearn.datasets import  make_classification
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
#from scikitplot.metrics import plot_roc
#from scikitplot.metrics import plot_precision_recall
from sklearn.metrics import accuracy_score
from sklearn.metrics import confusion_matrix
from sklearn.metrics import f1_score
from sklearn.metrics import classification_report
import numpy as np
from collections import Counter
from imblearn.under_sampling import RandomUnderSampler
import statistics as s
from collections import defaultdict
import random
from sklearn import tree

N_RUNS = 20
OPT_KNN = True

def fit_and_apply(X_train, y_train, X_test, y_test):
    #training and testing on balanced data
    if OPT_KNN:
        clf = KNeighborsClassifier(n_neighbors=5)
        clf = clf.fit(X_train, y_train)
    else:
        clf = tree.DecisionTreeClassifier().fit(X_train, y_train)
    y_pred = clf.predict(X_test)
    y_pred_tr = clf.predict(X_train)
    #    print('train acc. : ',accuracy_score(y_train, y_pred_tr))
    #    print('test acc. : ',accuracy_score(y_test, y_pred))
    # print('confusion matrix: \n',confusion_matrix(y_test, y_pred))
    #    print(classification_report(y_test, y_pred))
    conf_mat_train = confusion_matrix(y_train, y_pred_tr)
    conf_mat_test = confusion_matrix(y_test, y_pred)
    report_train = classification_report(y_train, y_pred_tr,output_dict=True)
    report_test = classification_report(y_test, y_pred,output_dict=True)
    return conf_mat_train, conf_mat_test, report_train, report_test

def print_results(proportions, perf,summary=True):
    print("")
    for k,v in proportions.items():
        print("Prop. ",k,"=",v)
    for t,d0 in perf.items():
        for c,d1 in d0.items():
            if summary:
                print(t,"class",c,"P,R,F:\t",end='')
            for m,values in d1.items():
                if m != "support":
                    if summary:
                        print("%.3f" % (s.mean(values)),end="\t")
                    else:
                        print(t,"class",c,m,":"," ".join([ "%.3f" % (p) for p in values ]), ". MEAN:",s.mean(values))
            if summary:
                print("")

def accu(conf_mat):
    correct = conf_mat[0][0]+conf_mat[1][1]
    incorrect=  conf_mat[0][1]+conf_mat[1][0]
    return correct/(correct+incorrect)
    
# perf[train|test][class][measure] = list of values
perf = defaultdict(lambda: defaultdict(lambda: defaultdict(list)))
proportions = {}
avg_conf_mat = defaultdict(lambda: [[0,0],[0,0]])
print("*** BALANCED -",end='')
for i in range(N_RUNS):
    print(i,end=' ',flush=True)

    #creating balanced dataset
    X, y = make_classification(n_samples=10000, n_features=5, n_informative=5, n_redundant=0, n_classes=2, n_clusters_per_class=2, weights=None, flip_y=0, class_sep=0.5, hypercube=True, shift=0.0, scale=1.0, shuffle=True, random_state=None)

    proportions["data"] = Counter(y)

    #splitting data
    X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.3, random_state=None, stratify=y)

    proportions["train"] = Counter(y_train)
    proportions["test"] = Counter(y_test)

    conf_mat_train,conf_mat_test,report_train, report_test = fit_and_apply(X_train, y_train, X_test, y_test)
    for c in range(2):
        for m,v in report_train[str(c)].items():
            perf["train"][c][m].append(report_train[str(c)][m])
            perf["test"][c][m].append(report_test[str(c)][m])
    for i in range(2):
        for j in range(2):
            avg_conf_mat["train"][i][j] += conf_mat_train[i][j] / N_RUNS
            avg_conf_mat["test"][i][j] += conf_mat_test[i][j] / N_RUNS

print_results(proportions, perf)
print("avg confusion matrix train: ",avg_conf_mat["train"]," avg accuracy=",accu(avg_conf_mat["train"]))
print("avg confusion matrix test: ",avg_conf_mat["test"]," avg accuracy=",accu(avg_conf_mat["test"]))
print("")

perf = defaultdict(lambda: defaultdict(lambda: defaultdict(list)))
proportions = {}
avg_conf_mat = defaultdict(lambda: [[0,0],[0,0]])
print("*** IMBALANCED A -",end='')
for i in range(N_RUNS):
    print(i,end=' ',flush=True)

    #making imbalanced data set (80%-20%)
    imbalance = (0.8,0.2)
    X, y = make_classification(n_samples=10000, weights=imbalance, n_features=5, n_informative=5, n_redundant=0, n_classes=2, n_clusters_per_class=2, flip_y=0, class_sep=0.5, hypercube=True, shift=0.0, scale=1.0, shuffle=True, random_state=None)

#    print(Counter(y))
    proportions["data"] = Counter(y)

    #splitting data
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=None, stratify=y)

    #undersampling majority class to obtain balanced training set
    res = RandomUnderSampler()
    X_train_res, y_train_res  = res.fit_resample(X_train, y_train)

#    print("y_train_res:",Counter(y_train_res))
#    print("y_test:",Counter(y_test))
    proportions["train"] = Counter(y_train_res)
    proportions["test"] = Counter(y_test)
    
    conf_mat_train,conf_mat_test,report_train, report_test = fit_and_apply(X_train, y_train, X_test, y_test)
    for c in range(2):
        for m,v in report_train[str(c)].items():
            perf["train"][c][m].append(report_train[str(c)][m])
            perf["test"][c][m].append(report_test[str(c)][m])
    for i in range(2):
        for j in range(2):
            avg_conf_mat["train"][i][j] += conf_mat_train[i][j] / N_RUNS
            avg_conf_mat["test"][i][j] += conf_mat_test[i][j] / N_RUNS

print_results(proportions, perf)
print("avg confusion matrix train: ",avg_conf_mat["train"]," avg accuracy=",accu(avg_conf_mat["train"]))
print("avg confusion matrix test: ",avg_conf_mat["test"]," avg accuracy=",accu(avg_conf_mat["test"]))
print("")

perf = defaultdict(lambda: defaultdict(lambda: defaultdict(list)))
proportions = {}
avg_conf_mat = defaultdict(lambda: [[0,0],[0,0]])
print("*** IMBALANCED B -",end='')
for i in range(N_RUNS):
    print(i,end=' ',flush=True)

    #creating balanced dataset
    X, y = make_classification(n_samples=10000, n_features=5, n_informative=5, n_redundant=0, n_classes=2, n_clusters_per_class=2, weights=None, flip_y=0, class_sep=0.5, hypercube=True, shift=0.0, scale=1.0, shuffle=True, random_state=None)

#    print(Counter(y))
    proportions["data"] = Counter(y)

    #splitting data
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=None, stratify=y)

    #undersampling class 1 to obtain imbalanced test set
    X_test_res = []
    y_test_res = []
    for i,c in enumerate(y_test):
        # pick value in [0,1]
        p = random.uniform(0,1)
        if c == 0 or p<0.1:
            X_test_res.append(X_test[i])
            y_test_res.append(y_test[i])

    proportions["train"] = Counter(y_train_res)
    proportions["test"] = Counter(y_test_res)
    
    conf_mat_train,conf_mat_test,report_train, report_test = fit_and_apply(X_train, y_train, X_test_res, y_test_res)
    for c in range(2):
        for m,v in report_train[str(c)].items():
            perf["train"][c][m].append(report_train[str(c)][m])
            perf["test"][c][m].append(report_test[str(c)][m])
    for i in range(2):
        for j in range(2):
            avg_conf_mat["train"][i][j] += conf_mat_train[i][j] / N_RUNS
            avg_conf_mat["test"][i][j] += conf_mat_test[i][j] / N_RUNS

print_results(proportions, perf)
print("avg confusion matrix train: ",avg_conf_mat["train"]," avg accuracy=",accu(avg_conf_mat["train"]))
print("avg confusion matrix test: ",avg_conf_mat["test"]," avg accuracy=",accu(avg_conf_mat["test"]))
print("")

The two main modifications are:

  • Running every experiment N_RUNS times in order to get a good estimate of the performance in every case. This is the same principle as cross-validation except that the generation of the data is included. Also I set random_state to None everywhere to avoid any bias.
  • Your version of the imbalanced experiment is included as "imbalanced A". I added another version "B" where the imbalance is produced directly in the test set. In fact the results with version A show that the method of undersampling a class from an imbalanced dataset is not equivalent to training on a balanced training set, certainly because of differences in the features as generated by make_classification (I don't know the details). This is visible in the fact that the 2 classes perform differently on the training set, something which is not supposed to happen if the training data is balanced.

I think your version (called A in my code) is an interesting illustration of the point that I was making above: the "true performance" can only be found if both the training set and the test set follow the "true distribution" of the data. Btw there's an ambiguity when we talk about "distribution of the data", people often assume that it's only the distribution of the classes, but in general it's about the distribution of the full instances (features+class) because otherwise the statistical relation between features and classes is potentially lost. In the case of version A the training set doesn't follow the "true distribution" of the data whereas the test set does.

[edited] Now if we compare the performance obtained with the imbalanced test set in option B to the performance obtained with the balanced test set, the performance still differs in F1-score. Let's see what happens in detail:

  • For both classes, the recall values are practically the same as for the balanced test set. This is because the proportion of correctly identified instances among true instances of this class stays the same, even though the number of instances in class 1 is lower.
  • Importantly this implies that the accuracy is (nearly) identical, since the global proportion of correctly identified instances is directly based on these two proportions.
  • However the precision values for the two classes differ, thus causing the F1-scores to differ as well. This is worth a detailed explanation by looking at the confusion matrices:
    • From the point of view of class 0, the precision increases because there are less possibilities of FP errors since there are less instances of class 1.
    • From the point of view of class 1, the precision decreases because there are proportionally more possibilities of FP errors: the number of TP instances for class 1 has decreased, so the precision value is weighted down by the comparatively high number of class 0 instances, mechanically causing a higher proportion of FPs for class 1.

What this means is that the difference in precision (and F1-score) compared to the balanced case is an artifact of the new distribution of classes: while the model has exactly the same chance to correctly recognize an instance of either class, its F1-score performance is lower for class 1 and higher for class 0. Btw this is a good example of the difficulty to choose a global performance metric: accuracy (or equivalently micro f1-score) is identical as in the balanced case, but macro f1-score differs. In this case I would consider that the performance is the same in reality, but strictly speaking it can be seen as different indeed.

$\endgroup$
6
  • $\begingroup$ Could you explain me why, even when the training set is balanced, the more a test set is imbalanced, the worst the performance of the classifier is for the minority class? Considering the students example, it is like if you train your students on a balanced set of exercise and, during the test, their capacity to solve a certain exercise depends on how often such exercise appears (proportionally) on the test. $\endgroup$
    – giogioia
    Aug 12 at 18:12
  • $\begingroup$ @giogioia in this scenario your conclusion is not correct actually: if the training data was perfectly balanced, then the test set imbalance would not affect performance because the model gives equal importance to the classes (of course the performance can vary a bit by chance). The model is influenced to favour the majority class only if there is a majority class in the training data, and in this case a different distribution in the test set will affect performance: if there is more majority instances then the performance improves and conversely. $\endgroup$
    – Erwan
    Aug 12 at 20:19
  • $\begingroup$ I am most probably missing something; however, on my university project, I noticed that even with a balanced training set, the test set imbalance is still the main discriminant of the classifier's performance for the minority class. I made a small colab [link] (colab.research.google.com/drive/…) to prove my point, I would immensely appreciate if you could check it and tell me where's my mistake. Thanks! $\endgroup$
    – giogioia
    Aug 13 at 22:08
  • $\begingroup$ @giogioia that was an interesting case, took me some time to figure it out. It's way too long for a comment so I edited my answer, see above. $\endgroup$
    – Erwan
    Aug 14 at 14:58
  • $\begingroup$ Thanks a lot for the answer! However, if I'm not wrong, in "IMBALANCED B", it looks like you actually tested the model on the balanced test set. I made a new colab colab link with the new results of your adjusted code and we can see that the issue of the minority class' performance still being inversely proportional to the class balance. $\endgroup$
    – giogioia
    Aug 14 at 19:12
2
$\begingroup$

Most classification algorithms define a decision boundary between classes. Class imbalances will cause cause the learned decision boundary to have a preference for the majority class. This is preference exists because most loss functions try to minimize average error (this is best done by maximizing performance on the majority class).

Then when the test data set is classified, the minority class will continue to perform worse because the decision boundary is designed to maximize majority class performance.

$\endgroup$
1
  • 1
    $\begingroup$ In other words, the prior probability of minority class membership being low is reflected in the posterior probability (as it should be) via Bayes' rule, and that results in the posterior probability being on the "wrong" side of the decision boundary, typically a probability of $0.5$. $\endgroup$
    – Dave
    Aug 11 at 19:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.