# Algorithms to do a CTRL+F (find object) on an image

We all know the CTRL+F "Find text..." feature in text editors / browsers.
I'd like to study the available algorithms to do something similar on an image.

Example of UI/UX: let's say you have an electronic schematic. You draw a rectangle around a diode, and then the algorithm will automatically find all similar diodes on this image.

In order to find a pattern on an image, I know (and have already used some of them) the classical tools:

• openCV matchTemplate (pros: works with a single training example, cons: doesn't support rotation or scaling)
• YOLO (pros: I thing it accepts rotation and scaling, cons: it requires 100s or 1000s of training examples)

Which available algorithms are there which would do their best with 1 or 2 training examples only and accept scaling / rotation?

• You could try using a siamese network for this (cs.cmu.edu/~rsalakhu/papers/oneshot1.pdf). The idea is not to classify images but to train a neural network to encode similar images in a similar way. You can train it once on a dataset (generating some similar and sissimilar pairs) and use it on new patterns afterwards. Nov 23, 2021 at 10:29
• @mprouveur can it work on segmentation of an image too? example: input=big electronic schmetatic, output=all the diodes are selected with a rectangle, i.e. they are localized on the image. From what I read in the PDF, it seems to (only?) classify pairs of images as similar or dissimilar. Can it also do localization of patterns in a bigger image? Do you have examples about this? Thanks
– Basj
Nov 26, 2021 at 9:46
• You can check SIFT, SURF. These are some of the scale and rotation invariant feature extraction algorithms. These algorithms can be used for finding the matching between source and target image Nov 27, 2021 at 5:07
• you can use this as a patch classifier and use it to detect object instances inside an image. here's a blog post on pig detection on images : erdem.pl/2021/02/… In this example it is only used to detect one class (pig) but it can be used for any number of classes. (I agree with @UdayaUnnikrishnan SIFT should be a faster first approach) Nov 29, 2021 at 15:17

Invariant object recognition(IOR), refers to rapid and accurate recognition of objects in the presence of variations such as size, rotation and position.

SIFT and SURF are the most popular among them, but unfortunately both SIFT & SURF are patented.

If you are looking for opensource algorithm, I would suggest to go for openCV's Oriented FAST and rotated BRIEF (ORB), OpenCV ORB reference link
quotes

ORB is a good alternative to SIFT and SURF in computation cost, matching performance and mainly the patents.

RESEARCH papers

First two research papers, I found that dealt with one training example.

• Rotation Invariant Object Recognition from One Training Example

This paper presented a rotation invariant local descriptor based on the Gaussian derivatives. We use “steerable filter” to implement the derivative responses. Rotation invariance is achieved by “steering” the descriptor to the main orientation at a center pixel location. An advantage of this strategy is that the main orientation can be computed directly from the first order derivative responses. They also consider feature selection in the case where only a single example is available. Virtual images are generated by rotating and rescaling the image. Rotationally and scale unstable features are computed by estimating the main orientation at the center pixel location and count the number of correctly estimated. Unstable features are removed during the learning step. The resulting object recognition system performs robustly under various illumination changes, viewpoint changes, scale changes, and rotation in the image plane, and under partial occlusion

Go for this approach if you want to start with one training example and wish to add more examples when they become available like an on-line learning scheme

• Object Recognition from Local Scale-Invariant Features introduces SIFT

This paper presents a new method for image feature generation called the Scale Invariant Feature Transform (SIFT). This approach transforms an image into a large collection of local feature vectors, called SIFT keys. Each image generates order of 1000 SIFT keys. These keys are used as input to a nearest-neighbor indexing method that identifies candidate object matches This system can learn an object model from a single image.

• This research paper ORB: an efficient alternative to SIFT or SURF compares performance of ORB with SIFT & SURF with two images set, an indoor and an outdoor scene. ORB outperformed SIFT & SURF images taken in outdoor and almost same performance for indoor one

Below papers supports object recognition with few training examples only(not with rotation/scaling)

• Few-Example Object Detection with Model Communication

This paper considers the problem of generic object detection with very few training examples (bounding boxes) per class, named “few-example object detection (FEOD)”. Existing works on supervised/semi-supervised/weaklysupervised object detection. Multi-modal Self-Paced Learning for Detection (MSPLD) algorithm was proposed which combines self-paced learning and the multi-modal learning
Unfortunately there is no existing implementation in python package as mentioned here. This paper though focus on few training samples for object detection, however doesn't support rotation and scaling of the objects

• Object Recognition from very few Training Examples for Enhancing Bicycle Maps

This paper introduces a system for object recognition that is trained with only 15 examples per class on average. To achieve this, we combine the advantages of convolutional neural networks and random forests to learn a patch-wise classifier. In the next step, we map the random forest to a neural network and transform the classifier to a fully convolutional network. Thereby, the processing of full images is significantly accelerated and bounding boxes can be predicted. However, this paper doesn't support rotation and scaling of the objects

POPULAR ALGORITHMs

• Scale Invariant Feature Transform (SIFT)

• Speeded Up Robust Feature (SURF): as SIFT was slow in performance a speeded up version SURF was introduced

• Features from Accelerated Segment Test (FAST) corner detector: Feature detection methods weren't fast enough in SIFT & SURF

• SIFT uses a feature descriptor with 128 floating point numbers. Consider thousands of such features. It takes lots of memory and more time for matching. We can compress it to make it faster. But still we have to calculate it first. There comes BRIEF which gives the shortcut to find binary descriptors with less memory, faster matching, still higher recognition rate.

• Oriented FAST and rotated BRIEF (ORB)

OpenCV devs came up with a new “FREE” alternative to SIFT & SURF, and that is ORB. good alternative to SIFT and SURF in computation cost, matching performance and mainly the patents

• RIFT: is a rotation-invariant generalization of SIFT

• RootSIFT

• G-RIF

• PCA-SIFT

• Gauss-SIFT

• KAZE and A-KAZE (KAZE Features and Accelerated-Kaze Features)

• GLOH (Gradient Location and Orientation Histogram)

Below algorithm supports object recognition with few training examples only(not with rotation/scaling)

• Fast R-CNN
• R-FCN

Object Detection via Region-based Fully Convolutional Networks. paper and package. Unfortunately does not support CPU-only mode

• Detectron2

Detectron2 is Facebook AI Research's next generation library that provides state-of-the-art detection and segmentation algorithms. It is the successor of maskrcnn-benchmark and Detectron. Detectron implements these object detection algorithms: Mask R-CNN ,RetinaNet, Faster R-CNN, RPN, Fast R-CNN, R-FCN

The following algorithms also outperforms SIFT and SURF

• KAZE and A-KAZE
• FAST corner detector
• ORB
• Part of this answer appear to have been copied from external sources, without indication that they are not your words. Specifically, "ORB is a good alternative ..." seems copied from opencv24-python-tutorials.readthedocs.io/en/latest/py_tutorials/…. Please revise this answer to properly credit the original source of all copied material: datascience.stackexchange.com/help/referencing
– D.W.
Dec 27, 2021 at 6:14
• Since this appear to have affected a number of your answers across the network, may I ask you to go through each one of your previous answers to check for similar instances of failure to credit external sources and edit all of them so they are brought into compliance with our rules? Thank you.
– D.W.
Dec 27, 2021 at 6:15