Ok, so let me answer some of these questions for you:
- What is the purpose of a confusion matrix?
A confusion matrix is merely a visual help for you to better interpret the performance of your model. It's a way to graphically visualize the true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN). Confusion matrices become more and more useful when you deal with a large number of different classes. It can give you some great insights about how your model is doing. Let's say you are training an image detection classifier. It might be good to know that your model is confused between dogs and wolves but isn't confused between cats and snakes.
Another purpose of the confusion matrix is to define a related cost matrix. In my example, being confused between dogs and wolves might be understandable and it shouldn't mean that your model is bad at what it does. However, if it is confused between classes it shouldn't be confused about, this should be properly represented in your performance metric.
Here is a good blog detailing these concepts: https://medium.com/@inivikrant/confusion-cost-matrix-helps-in-calculating-the-accuracy-cost-and-various-other-measurable-a725fb6b54e1
- Shouldn't we already know what metric we need to optimize given the type of problem we are trying to solve?
Here you are confusing two things. On the one hand, yes, you should know in advance which metric you want to optimize (i.e. accuracy, precision, recall, etc.) but it doesn't mean that you know the value of that metric in advance. If you dumb down hyperparameter tuning, it's roughly this:
- train model $M$ with hyperparameters $H$
- evaluate the performance $P$ of model $M$
- choose new hyperparameters $H$ and repeat step 1 and 2
- pick model $M$ with hyperparameters $H$ such that $P$ is optimized
Computing the accuracy, precision, recall, or F1-score can be done if you know the TP, FP, TN, FP (see this link for more info). So technically, you don't have to create the confusion matrix per se, but you definitely need to compute the TP, FP, TN, FP to evaluate the performance of your model.
- if we have enough computational power, we could basically group steps 2 and 5
You can only optimize your hyperparameters if you compute the performance of your model for each set of hyperparameters. You could skip step 3 as it doesn't technically affect your training process. It only helps you understand better what's going on. But you definitely cannot skip step 4.