ROC_AUC score is higher before tuning n _neighbors for KNN

Question

This is for multiclass classification. Before tuning the n_neighbors for KNN, these were the results:

    Train Accuracy:  99.54%
    Test Accuracy:  99.58%
    ROC AUC Score: 99.86%

After finding the optimum n_neighbors, these were the results:

    Train Accuracy:  99.64%
    Test Accuracy:  99.67%
    ROC AUC Score: 99.82%

My recall score went from 0.996 to 0.997. As we can see, the results improved without overfitting. But why did my ROC AUC score went down by 0.04? I thought the ROC AUC score increases when the model improves? My confusion matrices also improved:

Before tuning:

After tuning:

Answer 1

I'll answer first for two-class classification.

Your accuracy and recall values are being shown for a single threshold. ROC AUC is considering all thresholds. The threshold here is the decision of how to use the numeric output from your model to choose which of two classes.

So what your results show is that for one threshold your results improved. But, as AUC went down, for other thresholds it must have got worse. AUC is Area Under Curve, and it is often a good idea to plot the curves.

AUC is not used so much for multiclass classification. I believe it is normally done as one class vs. all other classes. So in your case you would end up with 8 curves, rather than a single curve. And then the AUC scores of those curves are averaged.

Again, plotting the curves is the best way to understand what is changing in your before and after models.