How to quantify the performance of the classifier (multi-class SVM) using the test data?

Question

I am working on a traffic sign recognition code in MATLAB using Belgian Traffic Sign Dataset. The dataset consists of training data and test data.

I resized the given images and extracted HOG features using the VL_HOG function from VL_feat library and then trained a multi class SVM using all of the signs inside the training dataset.

I am particularly interested detecting 8 signs. Question 1: Am I suppose to train the model using only the training set of these 8 signs or train the model with the entire training dataset and ignore the signs that I am not interested in at the detection stage?

Upon training the multi-class SVM, I want to test the classifier performance using the test data. Question 2: Can someone guide me through how to do this? I found something similar to what I am looking for available here. It would be great if I can obtain quantitative data such as a confusion matrix in percentage form.

Thanks!

Answer 1

Machine Learning process usually follows three steps:

1. Training (on training data): this data is used in the learning stage. The model optimally adjusts the weights in order to minimize the loss function over this dataset.
2. Evaluation (on evaluation data): it serves to fine-tune the model, for instance you can decide if your model overfits
3. Testing (on test data): tells you about the inference power of your model

The intersection of these three datasets has to be zero, and they should come from the same disttibution.

Recalling that, the first thing is the naming convention for your two datasets:

1. If training and test datasets come from the same distribution (I guess yes, since they are all Belgian sign traffics) you can name it Training and Validation datasets.
2. If the same condition holds, you can perform cross-validation over your Training dataset, and carry on naming it Training and Tets datasets.

I am particularly interested detecting 8 signs. Question 1: Am I suppose to train the model using only the training set of these 8 signs or train the model with the entire training dataset and ignore the signs that I am not interested in at the detection stage?

It depends on whether the practical application you are designing the algorithm for will encounter those remaining signs. If so, I will give these signs a label, for examle 'Don't care'. The algorithm should have the capability of recognizing your 8 signs and to know that the others are of non importance, but do not fall on your desired targets. On the contrary, if the algorithm will be always fed with these 8 signs, you can use only those images. There would be other approaches: you can train a previous SVM in which you set two groups of signs (binary classifier or Support Vector Data Description), one containing the target 8, and other with the remaining ones. Then, another SVM can recognize the sign only if it has been previously classified among the 8 desired one (this SVM is only trained with the 8 signs).

Upon training the multi-class SVM, I want to test the classifier performance using the test data. Question 2: Can someone guide me through how to do this? I found something similar to what I am looking for available here. It would be great if I can obtain quantitative data such as a confusion matrix in percentage form.

Taking it into account, I guess you have to name your datasets as Training and Evaluation. You first train the model with the training data, and then you evaluate how it generalizes with the evaluation data. First you have to decide which metrics you will use, I recommend precision and recall, per class. Once you have your trained model, you perform predictions over the evaluation test and you'll get the metrics for this dataset. For instance, if your results over the evaluation data are worse than on over the training data, your model is overfitting the training dataset.

However, I strongly recommend you to use cross-validation on the Training-dataset, visit this link

Answer 2

There are many different parameters which can evaluate the performance of your method by comparing the real and predicted labels. I suggest PyCM module which can give a vast variety of this parameters.

You can use these simple following code in python:

>>> from pycm import *
>>> y_actu = [2, 0, 2, 2, 0, 1, 1, 2, 2, 0, 1, 2] # or y_actu = numpy.array([2, 0, 2, 2, 0, 1, 1, 2, 2, 0, 1, 2])
>>> y_pred = [0, 0, 2, 1, 0, 2, 1, 0, 2, 0, 2, 2] # or y_pred = numpy.array([0, 0, 2, 1, 0, 2, 1, 0, 2, 0, 2, 2])
>>> cm = ConfusionMatrix(actual_vector=y_actu, predict_vector=y_pred) # Create CM From Data
>>> print(cm)