Finding Feature Importance in CNN's?

Question

Let's say I have images of cars. For each image in the dataset, I have let's say 3 pictures of the same car but in different angles.

1) The first image is the picture of the car from the front. 2) The second image is the picture of that car from either of the sides. 3) The third image is the picture of that car from the back.

If I have 1000 cars, I would have 3000 images as my dataset. Is there a way to find which angle of the car represents its label the most? In other terms, Which angle of the car image is most helpful in determining whether a particular image is a car or not?

My thoughts were to build a random guessing model to determine the lower bound of our loss and then iteratively use each of angle of images as train and predict on the same angle and come up with a loss score. Ex: First I train and predict with all images of angle 1 and come up with a loss score. So If I have 3 angles of images, I would end up with 3 loss values. So the angle with the lowest loss score is determined as the winner. But the problem is, I'm not sure if this is approach is right because if I have multiple classes in the dataset then the dominant class would have lower loss compared to any other non-dominant classes and moreover we would not be using the entire data at any point of time and so would possibly end up with wrong conclusions.

Thanks for replying in advance. This question has been lingering in my head for a long time now.

Answer 1

If you are training a classifier on only one angle, you will overfit to that particular position - it won't be able to recognise the same car from a different angle. It may learn something, I guess, and you could test generalisation by testing on the unseen images from the other two angles. So train on each set individually and test on all 3 sets individually with the usual caveats on training vs testing.

A better approach, however, would be to train on all your data. The network will automatically learn what is the most efficient and you can then look at the results on each of the 3 test sets to see which gives the best results. I am guessing that this is an example problem, so more generally it makes sense to do a statistical analysis on why the classifier fails. If you can identify which features make classification hard, you can work harder to create a training strategy that will improve the model. The main goal would be to make a model that generalises. I suspect that training using extra information from other angles will only improve the effectiveness for all cases.

Answer 2

From your question, I'm not sure what your motivation is, but if you just want to find out which visual features are the most representative for a given class label on a dataset of objects captured from different viewing angles, I would suggest the following approach:

• For the training process, use pictures from all different angles, shuffle them, and train your CNN.
• Then, take the test dataset and measure the accuracy as an overall benchmark.
• In a second step, divide your data according to the viewing angle and measure the accuracy of the previously trained model on each viewing angle.
• Now, take the angle on which you got the highest accuracy and choose a few images that have been classified correctly. Analyse those images with saliency maps.
• Subsequently, take a few misclassified images from the angle with the lowest accuracy as well and analyse them in the same way.

From the saliency maps, you will get a deeper understanding which visual features cause your trained CNN to predict a certain label.