Why use convolutional NNs for a visual inspection task over classic CV template matching?

Question

I had an interesting discussion come up based on a project we were working on: why use a CNN visual inspection system over a template matching algorithm?

Background: I had shown a demo of a simple CNN vision system (webcam + laptop) that detected if a particular type of object was "broken"/defective or not - in this case, a PCB circuit board. My CNN model was shown examples of the proper and broken circuit boards (about 100 images of each) on a static background. Our model used the first few conv/maxpool layers of pre-trained VGG16 (on imagenet), and then we added a few more trainable convs/pools, with a few denses, leading to a dim-3 one hot encoded vectored output for classification: (is_empty, has_good_product, has_defective_product).

The model trained pretty easily and reached 99% validation acc no problems; we also trained with various data augmentation since we know our dataset was small. In practice, it worked about 9 times out of 10, but a few random translations/rotations of the same circuit board would occasionally put it in the opposite class. Perhaps more aggressive data augmentation would have helped. Anyways, for a prototype concept project we were happy.

Now we were presenting to another engineer and his colleague, and he brought up the argument that NNs are overkill for this, should just use template matching, why would one want to do CNNs?

We didn't have a great answer for why our approach could be better in certain applications (e.g. other parts to inspect). Some points we brought up:

1) More robust to invariances (through e.g. data augmentation)

2) Can do online learning to improve the system (e.g. human can tell the software which examples it got wrong)

3) No need to set thresholds like in classical computer vision algorithms What do you guys think, are there more advantages for a CNN system for this type of inspection task? In what cases would it be better than template matching?

A few more random ideas for when deep NNs could be the tech for the job: for systems that require 3D depth sensing as part of the input, or any type of object that can be deformed/stretched/squished but still be "good" and not defective (e.g. a stuffed animal, wires, etc). Curious to hear your thoughts :)

Answer 1

The engineer in question that proposed traditional CV methods for your application simply did so out of habit. Using template matching is extremely outdated and has been shown to perform very poorly. However, I do think a CNN is overkill depending on your dataset's size.

How does template matching work?

Template matching slides a window across your image that will provide a percent match with the template. If the percent match is above a certain predefined threshold then it is assumed to be a match. For example if you have an image of a dog and you want to determine if there is a dog in the image, you would slide a dog template around the entire image area and see if there is a sufficiently large percent match. This will likely result in very poor performance because it requires the template to overlap the image identically. What is the likelihood of that in practice? Not very high.

The only time template matching is a sufficient technique is if you know exactly what you are looking for and you are confident that it will appear almost identically in every example of a given class.

Why use machine learning instead?

Machine learning techniques are not rigid. Unlike what stmax said, CNNs are able to generalize a dataset very well. That is why they are so powerful. Using the dog example, the CNN does not need to see a picture of every dog in existence to understand what constitutes as a dog. You can show it maybe 1000 images from a Google search, and then the algorithm will be able to detect that your dog, is in fact a dog. The fact that machine learning algorithms generalize very well is the reason that they replaced all the ancient CV techniques. Now the problem is the amount of data that you need to train a CNN. They are extremely data intensive.

I do not think that 100 data points is sufficient to train a robust CNN. Due to the deep complexity of the model in order to limit the bias you need to increase your number of examples. I usually suggest 100 examples for every feature for deep models and 10 examples for every feature for shallow models. It really all depends on your feature-space.

What I suggest.

What you are truly doing is anomaly detection. You have a lot of examples that will be presented of PCBs that are otherwise in good shape. You want to detect those which are broken. Thus I would attempt some anomaly detection methods instead. They are much simpler to implement and you can get good results using shallow models especially in skewed datasets (1 class is over represented).

Answer 2

The answer depends on the task. Template matching can work for some tasks but not for all. CNNs potentially have the ability to generalize to unseen inputs that don't match any of your templates, so can potentially generalize better.

But whether CNNs will beat template matching will depend on the specific task and what specifically you're trying to achieve. This is an empirical science; ultimately, the way you find out which works better is to try them both -- or learn from others who have tried them (e.g., by reading the literature). I don't think you're going to find some theory or taxonomy that is going to substitute for empirical evaluation on real-world data.

Answer 3

One problem you might run into with a NN (and other classification methods) is that since you've only shown it certain defects, it might not know how to react to completely new / yet unseen defects that might pop up in the future.

You want the NN to learn "anything that doesn't look like a non-defective PCB is a defective PCB". But what if it has learned "anything that doesn't look like a defective PCB is a non-defective PCB"?

You could try to modify some images of non-defective PCBs by adding a small white spot (or another small perturbation) to them at random locations and have the neural network classify these modified images. It should definitely classify them as defective, right? But it'll probably miss some (or quite many) because it has never seen such defects before.

To detect completely new defects, anomaly detection methods / one class classifiers might be more.. trustworty, because they should pick up anything that's never been seen before.

As D.W. said, you're just going to have to try both methods and find out which one works better. Just make sure to have a really good test set that also contains completely new defects!