2
$\begingroup$

I am studying machine learning. After some research I understood that a typical workflow for a classification problem (after having prepared the data) is the following:

  1. Split data in test, train and validation sets
  2. Train the model
  3. Generate the confusion matrix
  4. Analyze the metrics: accuracy, precision, recall and f1
  5. Tune hyper-parameters based on the metric I have decided to optimize.

My question is: why do we ever need the confusion matrix? Shouldn't we already know what metric we need to optimize given the type of problem we are trying to solve?

I am asking this because, as far as I understand, if we have enough computational power, we could basically group steps 2 and 5 by applying a grid-search (which consists basically in a cross-validation for each tuning parameter) which takes as an input the metric to be measured. This would imply that you need to know the metric beforehand and furthermore you cannot get the confusion matrix.

Thanks in advance for the replies.

$\endgroup$
1
$\begingroup$

Ok, so let me answer some of these questions for you:

  • What is the purpose of a confusion matrix?

A confusion matrix is merely a visual help for you to better interpret the performance of your model. It's a way to graphically visualize the true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN). Confusion matrices become more and more useful when you deal with a large number of different classes. It can give you some great insights about how your model is doing. Let's say you are training an image detection classifier. It might be good to know that your model is confused between dogs and wolves but isn't confused between cats and snakes.

Another purpose of the confusion matrix is to define a related cost matrix. In my example, being confused between dogs and wolves might be understandable and it shouldn't mean that your model is bad at what it does. However, if it is confused between classes it shouldn't be confused about, this should be properly represented in your performance metric.

Here is a good blog detailing these concepts: https://medium.com/@inivikrant/confusion-cost-matrix-helps-in-calculating-the-accuracy-cost-and-various-other-measurable-a725fb6b54e1

  • Shouldn't we already know what metric we need to optimize given the type of problem we are trying to solve?

Here you are confusing two things. On the one hand, yes, you should know in advance which metric you want to optimize (i.e. accuracy, precision, recall, etc.) but it doesn't mean that you know the value of that metric in advance. If you dumb down hyperparameter tuning, it's roughly this:

  1. train model $M$ with hyperparameters $H$
  2. evaluate the performance $P$ of model $M$
  3. choose new hyperparameters $H$ and repeat step 1 and 2
  4. pick model $M$ with hyperparameters $H$ such that $P$ is optimized

Computing the accuracy, precision, recall, or F1-score can be done if you know the TP, FP, TN, FP (see this link for more info). So technically, you don't have to create the confusion matrix per se, but you definitely need to compute the TP, FP, TN, FP to evaluate the performance of your model.

  • if we have enough computational power, we could basically group steps 2 and 5

You can only optimize your hyperparameters if you compute the performance of your model for each set of hyperparameters. You could skip step 3 as it doesn't technically affect your training process. It only helps you understand better what's going on. But you definitely cannot skip step 4.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for the answer. I get what you say and it makes sense to me only if i have just one single model. If I had to multiple models to test in order to pick the best one, is it correct to assume that I the only steps i need to take are: choose the metric, run grid-search, get the model with best metric? $\endgroup$ – Federico Gentile May 15 at 8:47
  • $\begingroup$ Yes, the best model is the model that optimizes the value of the metric you chose. Grid-search is just a way to explore the hyperparameter space. And the confusion matrix is only there for you to better interpret the results. However, I don't understand what you mean when you talk about the difference between having 1 or multiple models? Can you elaborate, it's very unclear to me $\endgroup$ – Valentin Calomme May 15 at 8:51
  • $\begingroup$ you are right, 1 or multiple models does't matter at all. After thinking about it the problem for me boils down to the use of grid-search (and cross-validation). If i do not use it, I can calculate for each trained model the confusion matrix to have a better understanding of the problem. On the other hand, if I use grid-search i cannot compute the confusion matrix as the scoring parameter needs to me selected before hand. Since grid-search is very good because it reduces the variance and finds the optimal parameters, I don't understand why i would need the confusion matrix? $\endgroup$ – Federico Gentile May 15 at 9:01
  • $\begingroup$ To be clear, Gridsearch and cross-validation does not train your model. What it does is that it finds which hyperparameters should lead to the best model. The use of cross-validation is to get an estimate of the performance without relying on your test data. Your confusion matrix is not built with the performance of the models created during cross-validation. It is built by evaluating the model that is trained on all of your training data on your test data $\endgroup$ – Valentin Calomme May 15 at 9:55
1
$\begingroup$

Adding 2 points to the above answer -

  • Confusion matrix gives a better comparative picture for a Business guy. e.g. If you inform your business that F1 Score is 0.9 is of lesser usage to him.
    But he would love when you say - Model will miss 9 out of 100 Cancer cases and report 50 out of 10000 as Cancer when it is not.

  • When you have more than 2 classes, CM gives an idea about the learning mistake the model is making. e.g. In Fashion MNIST data, we can observers that model is confused between Shirt and Coat. You can adapt accordingly. See image below

Image credit - https://www.kaggle.com/fuzzywizard/fashion-mnist-cnn-keras-accuracy-93 Confusion matrix

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.