# Computer vision: Identifying different items in screenshot

Lets say I have a screenshot like this:

I want to be able to detect/localize each item on the floor, however, there 1) can be any number of items in the image and 2) each item is different

I have a candidate list of all possible items. In reality each one is labelled and separated into individual image files:

I've thought about training something like a convnet, but that feels like it might be slow because I'd need to segment each screenshot with multiple sliding windows and feed each one forward through the net. Creating those sliding window segments for each screenshot will likely take a long time. I'd like the entire detection process to be completed quickly (<1sec)

Whats the most efficient way of doing this detection task? I will be implementing this using Javascript

The main problems are:

1. The item count is unknown. There could be 0 items in the screenshot, or there could be many
2. There are a lot of possible targets, each one of which are different in shape, colour, and structure
3. Javascript isn't the fastest of languages for this type of work. This is a semi strict requirement unless its absolutely impossible to do this using JS. The fallback language would be Python
• This problem is called object detection and you can do it in Javascript: js-objectdetect, jsfeat. If node is an option: node-yolo. – Emre Jun 29 '17 at 3:10

This is a good case for computer vision. You know exactly what the icon looks like (basically pixel-perfect), and you want to find whether it exists anywhere in the screenshot. This is an image matching problem.

Normalized cross-correlation (NCC) is a standard way to check whether small image $I$ (your icon) appears anywhere in the large image $I'$: you compute the NCC and then look for a location where the value of the NCC is large; that is your candidate match. You can now apply this to the set of candidate icons: for each candidate icon, search whether it appears anywhere in the screenshot, using NCC. OpenCV and other computer vision libraries should include normalized implementations of the NCC computation.

For speed, you can also try using image pyramids, but try using NCC directly on the full screenshot first.

You really have two problems:

1. Object detection / bounding boxes to locate an icon you want to recognize.
2. Image recognition of what each icon is. Because you have labeled training data already and they are all roughly the same size, this part should be relatively easy as compared to #1.

You need to get #1 working and passing a subset of the pixels to #2.

• Yeah but its #1 that I don't know how to figure out, because finding the objects somewhat depends on #2, i.e. how do you know how to do the detection if you don't already know what the icons are? This is simple in the case of something like face detection, because all faces look somewhat similar and you can just use a template to do the matching for #1, but when there are so many possible target icons I'm not sure how to proceed... – Simon Jun 29 '17 at 13:32
• You have a huge advantage in that all those icons are roughly the same size and the background is only a couple different shades of brown. It mostly needs to figure out what the right window size is to iterate across the image with. It won't be recognizing what the target icons look like, it will be trying to recognize what the BACKGROUND looks like. It would basically need to spot when all four edges of the image are brown, call that "icon!" and then pass what is inside to #2. – CalZ Jun 29 '17 at 15:35