There is a confusion between the "true" performance of the classifier, which is indeed fixed once the classifier is trained, and the observed performance on a particular test set.
The "true" performance can only be estimated, and it should be estimated using a random sample which follows the "true" distribution of the data. Supervised learning always assumes a "true population" and both the training set and test set are supposed to be subsets of this true population.
If one uses a the test set with a different distribution then there's no guarantee that the performance will be the same as the true performance. This can be relevant in some experiments, but it's not a proper evaluation of the classifier itself.
Intuitively this can be compared to a test given to some students after they have studied some exercises:
- If the test questions are similar to the questions they studied in the exercises then their mark reflects their true performance, i.e. how well they learned from the exercises.
- If the professor makes the test with questions which are not seen or rarely seen in the exercises, then the mark is likely to be lower even for the good students.
Edit: study of the specific case of balanced training set vs. imbalanced test set (asked by OP in comments).
Re-edited after fixing mistake found by OP
This is an interesting case to study, thanks for asking :)
Using your code as a basis I tested the following code:
import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
#from scikitplot.metrics import plot_roc
#from scikitplot.metrics import plot_precision_recall
from sklearn.metrics import accuracy_score
from sklearn.metrics import confusion_matrix
from sklearn.metrics import f1_score
from sklearn.metrics import classification_report
import numpy as np
from collections import Counter
from imblearn.under_sampling import RandomUnderSampler
import statistics as s
from collections import defaultdict
import random
from sklearn import tree
N_RUNS = 20
OPT_KNN = True
def fit_and_apply(X_train, y_train, X_test, y_test):
#training and testing on balanced data
if OPT_KNN:
clf = KNeighborsClassifier(n_neighbors=5)
clf = clf.fit(X_train, y_train)
else:
clf = tree.DecisionTreeClassifier().fit(X_train, y_train)
y_pred = clf.predict(X_test)
y_pred_tr = clf.predict(X_train)
# print('train acc. : ',accuracy_score(y_train, y_pred_tr))
# print('test acc. : ',accuracy_score(y_test, y_pred))
# print('confusion matrix: \n',confusion_matrix(y_test, y_pred))
# print(classification_report(y_test, y_pred))
conf_mat_train = confusion_matrix(y_train, y_pred_tr)
conf_mat_test = confusion_matrix(y_test, y_pred)
report_train = classification_report(y_train, y_pred_tr,output_dict=True)
report_test = classification_report(y_test, y_pred,output_dict=True)
return conf_mat_train, conf_mat_test, report_train, report_test
def print_results(proportions, perf,summary=True):
print("")
for k,v in proportions.items():
print("Prop. ",k,"=",v)
for t,d0 in perf.items():
for c,d1 in d0.items():
if summary:
print(t,"class",c,"P,R,F:\t",end='')
for m,values in d1.items():
if m != "support":
if summary:
print("%.3f" % (s.mean(values)),end="\t")
else:
print(t,"class",c,m,":"," ".join([ "%.3f" % (p) for p in values ]), ". MEAN:",s.mean(values))
if summary:
print("")
def accu(conf_mat):
correct = conf_mat[0][0]+conf_mat[1][1]
incorrect= conf_mat[0][1]+conf_mat[1][0]
return correct/(correct+incorrect)
# perf[train|test][class][measure] = list of values
perf = defaultdict(lambda: defaultdict(lambda: defaultdict(list)))
proportions = {}
avg_conf_mat = defaultdict(lambda: [[0,0],[0,0]])
print("*** BALANCED -",end='')
for i in range(N_RUNS):
print(i,end=' ',flush=True)
#creating balanced dataset
X, y = make_classification(n_samples=10000, n_features=5, n_informative=5, n_redundant=0, n_classes=2, n_clusters_per_class=2, weights=None, flip_y=0, class_sep=0.5, hypercube=True, shift=0.0, scale=1.0, shuffle=True, random_state=None)
proportions["data"] = Counter(y)
#splitting data
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.3, random_state=None, stratify=y)
proportions["train"] = Counter(y_train)
proportions["test"] = Counter(y_test)
conf_mat_train,conf_mat_test,report_train, report_test = fit_and_apply(X_train, y_train, X_test, y_test)
for c in range(2):
for m,v in report_train[str(c)].items():
perf["train"][c][m].append(report_train[str(c)][m])
perf["test"][c][m].append(report_test[str(c)][m])
for i in range(2):
for j in range(2):
avg_conf_mat["train"][i][j] += conf_mat_train[i][j] / N_RUNS
avg_conf_mat["test"][i][j] += conf_mat_test[i][j] / N_RUNS
print_results(proportions, perf)
print("avg confusion matrix train: ",avg_conf_mat["train"]," avg accuracy=",accu(avg_conf_mat["train"]))
print("avg confusion matrix test: ",avg_conf_mat["test"]," avg accuracy=",accu(avg_conf_mat["test"]))
print("")
perf = defaultdict(lambda: defaultdict(lambda: defaultdict(list)))
proportions = {}
avg_conf_mat = defaultdict(lambda: [[0,0],[0,0]])
print("*** IMBALANCED A -",end='')
for i in range(N_RUNS):
print(i,end=' ',flush=True)
#making imbalanced data set (80%-20%)
imbalance = (0.8,0.2)
X, y = make_classification(n_samples=10000, weights=imbalance, n_features=5, n_informative=5, n_redundant=0, n_classes=2, n_clusters_per_class=2, flip_y=0, class_sep=0.5, hypercube=True, shift=0.0, scale=1.0, shuffle=True, random_state=None)
# print(Counter(y))
proportions["data"] = Counter(y)
#splitting data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=None, stratify=y)
#undersampling majority class to obtain balanced training set
res = RandomUnderSampler()
X_train_res, y_train_res = res.fit_resample(X_train, y_train)
# print("y_train_res:",Counter(y_train_res))
# print("y_test:",Counter(y_test))
proportions["train"] = Counter(y_train_res)
proportions["test"] = Counter(y_test)
conf_mat_train,conf_mat_test,report_train, report_test = fit_and_apply(X_train, y_train, X_test, y_test)
for c in range(2):
for m,v in report_train[str(c)].items():
perf["train"][c][m].append(report_train[str(c)][m])
perf["test"][c][m].append(report_test[str(c)][m])
for i in range(2):
for j in range(2):
avg_conf_mat["train"][i][j] += conf_mat_train[i][j] / N_RUNS
avg_conf_mat["test"][i][j] += conf_mat_test[i][j] / N_RUNS
print_results(proportions, perf)
print("avg confusion matrix train: ",avg_conf_mat["train"]," avg accuracy=",accu(avg_conf_mat["train"]))
print("avg confusion matrix test: ",avg_conf_mat["test"]," avg accuracy=",accu(avg_conf_mat["test"]))
print("")
perf = defaultdict(lambda: defaultdict(lambda: defaultdict(list)))
proportions = {}
avg_conf_mat = defaultdict(lambda: [[0,0],[0,0]])
print("*** IMBALANCED B -",end='')
for i in range(N_RUNS):
print(i,end=' ',flush=True)
#creating balanced dataset
X, y = make_classification(n_samples=10000, n_features=5, n_informative=5, n_redundant=0, n_classes=2, n_clusters_per_class=2, weights=None, flip_y=0, class_sep=0.5, hypercube=True, shift=0.0, scale=1.0, shuffle=True, random_state=None)
# print(Counter(y))
proportions["data"] = Counter(y)
#splitting data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=None, stratify=y)
#undersampling class 1 to obtain imbalanced test set
X_test_res = []
y_test_res = []
for i,c in enumerate(y_test):
# pick value in [0,1]
p = random.uniform(0,1)
if c == 0 or p<0.1:
X_test_res.append(X_test[i])
y_test_res.append(y_test[i])
proportions["train"] = Counter(y_train_res)
proportions["test"] = Counter(y_test_res)
conf_mat_train,conf_mat_test,report_train, report_test = fit_and_apply(X_train, y_train, X_test_res, y_test_res)
for c in range(2):
for m,v in report_train[str(c)].items():
perf["train"][c][m].append(report_train[str(c)][m])
perf["test"][c][m].append(report_test[str(c)][m])
for i in range(2):
for j in range(2):
avg_conf_mat["train"][i][j] += conf_mat_train[i][j] / N_RUNS
avg_conf_mat["test"][i][j] += conf_mat_test[i][j] / N_RUNS
print_results(proportions, perf)
print("avg confusion matrix train: ",avg_conf_mat["train"]," avg accuracy=",accu(avg_conf_mat["train"]))
print("avg confusion matrix test: ",avg_conf_mat["test"]," avg accuracy=",accu(avg_conf_mat["test"]))
print("")
The two main modifications are:
- Running every experiment
N_RUNS times in order to get a good estimate of the performance in every case. This is the same principle as cross-validation except that the generation of the data is included. Also I set random_state to None everywhere to avoid any bias.
- Your version of the imbalanced experiment is included as "imbalanced A". I added another version "B" where the imbalance is produced directly in the test set. In fact the results with version A show that the method of undersampling a class from an imbalanced dataset is not equivalent to training on a balanced training set, certainly because of differences in the features as generated by
make_classification (I don't know the details). This is visible in the fact that the 2 classes perform differently on the training set, something which is not supposed to happen if the training data is balanced.
I think your version (called A in my code) is an interesting illustration of the point that I was making above: the "true performance" can only be found if both the training set and the test set follow the "true distribution" of the data. Btw there's an ambiguity when we talk about "distribution of the data", people often assume that it's only the distribution of the classes, but in general it's about the distribution of the full instances (features+class) because otherwise the statistical relation between features and classes is potentially lost. In the case of version A the training set doesn't follow the "true distribution" of the data whereas the test set does.
[edited]
Now if we compare the performance obtained with the imbalanced test set in option B to the performance obtained with the balanced test set, the performance still differs in F1-score. Let's see what happens in detail:
- For both classes, the recall values are practically the same as for the balanced test set. This is because the proportion of correctly identified instances among true instances of this class stays the same, even though the number of instances in class 1 is lower.
- Importantly this implies that the accuracy is (nearly) identical, since the global proportion of correctly identified instances is directly based on these two proportions.
- However the precision values for the two classes differ, thus causing the F1-scores to differ as well. This is worth a detailed explanation by looking at the confusion matrices:
- From the point of view of class 0, the precision increases because there are less possibilities of FP errors since there are less instances of class 1.
- From the point of view of class 1, the precision decreases because there are proportionally more possibilities of FP errors: the number of TP instances for class 1 has decreased, so the precision value is weighted down by the comparatively high number of class 0 instances, mechanically causing a higher proportion of FPs for class 1.
What this means is that the difference in precision (and F1-score) compared to the balanced case is an artifact of the new distribution of classes: while the model has exactly the same chance to correctly recognize an instance of either class, its F1-score performance is lower for class 1 and higher for class 0. Btw this is a good example of the difficulty to choose a global performance metric: accuracy (or equivalently micro f1-score) is identical as in the balanced case, but macro f1-score differs. In this case I would consider that the performance is the same in reality, but strictly speaking it can be seen as different indeed.
