1
$\begingroup$

Suppose that I have the well known iris dataset and I want to perform error analysis on the misclassified examples, more specifically for a specific class.

I don't really care about fine-tuning or selecting another model, stratification when splitting etc, which would improve precision and recall for each class, rather than how to manually focus on the features of the misclassified examples and understand the logic of misclassification (if you could please avoid being too technical e.g. using packages as lime or shap).

Below I provide a MWE to make it more clear.

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns

from sklearn import datasets

from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report, confusion_matrix

from sklearn.linear_model import SGDClassifier

Get the data and train a model.

iris = datasets.load_iris()

X = iris.data
y = iris.target_names[iris.target]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42
)

sgd = SGDClassifier(random_state=42)
sgd.fit(X_train, y_train)
y_pred = sgd.predict(X_train)

conf_matrix = confusion_matrix(y_true=y_train, y_pred=y_pred)
conf_matrix

The confusion matrix will look like:

array([[31,  0,  0],
       [ 1, 17, 19],
       [ 0,  0, 37]], dtype=int64)

and the classification report will be like:

              precision    recall  f1-score   support

      setosa       0.97      1.00      0.98        31
  versicolor       1.00      0.46      0.63        37
   virginica       0.66      1.00      0.80        37

    accuracy                           0.81       105
   macro avg       0.88      0.82      0.80       105
weighted avg       0.87      0.81      0.79       105

Looking at the classification report we see that the recall of the versicolor class is really low and actually we have more false negatives (in favor of the virginica class), rather than true positives (skewed classes is not the case here). Also, notice that within the code, we test the performance in the training data.

My feeling says that the examples where we have misclassification y_true=versicolor and y_pred=virginica are more "similar" to the examples where we have y_true=virginica, but I am not quite sure how to measure this similarity.

Would this error analysis be different if we chose a different model, e.g. a RandomForest?

In reality, we would have potentially a lot of features, like one-hot encoded ones, text along with numerical ones, but I gave this MWE as a start on how to proceed.

Thanks!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.