Suppose that I have the well known
iris dataset and I want to perform error analysis on the misclassified examples, more specifically for a specific class.
I don't really care about fine-tuning or selecting another model, stratification when splitting etc, which would improve precision and recall for each class, rather than how to manually focus on the features of the misclassified examples and understand the logic of misclassification (if you could please avoid being too technical e.g. using packages as
Below I provide a MWE to make it more clear.
import matplotlib.pyplot as plt import numpy as np import pandas as pd import seaborn as sns from sklearn import datasets from sklearn.model_selection import train_test_split from sklearn.metrics import classification_report, confusion_matrix from sklearn.linear_model import SGDClassifier
Get the data and train a model.
iris = datasets.load_iris() X = iris.data y = iris.target_names[iris.target] X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.3, random_state=42 ) sgd = SGDClassifier(random_state=42) sgd.fit(X_train, y_train) y_pred = sgd.predict(X_train) conf_matrix = confusion_matrix(y_true=y_train, y_pred=y_pred) conf_matrix
The confusion matrix will look like:
array([[31, 0, 0], [ 1, 17, 19], [ 0, 0, 37]], dtype=int64)
and the classification report will be like:
precision recall f1-score support setosa 0.97 1.00 0.98 31 versicolor 1.00 0.46 0.63 37 virginica 0.66 1.00 0.80 37 accuracy 0.81 105 macro avg 0.88 0.82 0.80 105 weighted avg 0.87 0.81 0.79 105
Looking at the classification report we see that the recall of the
versicolor class is really low and actually we have more false negatives (in favor of the
virginica class), rather than true positives (skewed classes is not the case here). Also, notice that within the code, we test the performance in the training data.
My feeling says that the examples where we have misclassification
y_pred=virginica are more "similar" to the examples where we have
y_true=virginica, but I am not quite sure how to measure this similarity.
Would this error analysis be different if we chose a different model, e.g. a
In reality, we would have potentially a lot of features, like one-hot encoded ones, text along with numerical ones, but I gave this MWE as a start on how to proceed.