How would one separate digits for number recognition?

Question

Assuming that one has a neural network capable of returning the numerical digit from a given image of size 28x28px.

How would one split an image of unknown size and an unknown amount of digits into a series of 28x28px images to feed to this network? (The order of the digits must be obtainable.)

For example:

How can this be split up to:

Assuming that there are not always 5 digits and the initial image is not always the same size.

Firstly, my thoughts would be to create a secondary neural network. This neural network would output an (x, y) coordinate. This could be used to crop the image to 28x28px about this coordinate. However, this neural network would only be able to locate 1 digit at a time.

Secondly, another idea is that a series of random crops could be performed and then all given to the number recognising neural network. However, this would create a high error rate and the number recognising neural network has no way of telling that there was no valid number given (Unless that output was added). But more importantly the order of the digits would/could be lost.

I am struggling to find any resource explaining a possible solution. Google's house number recognition takes the entire image of digits and returns a value. This can be found in How Google Cracked House Number Identification in Street View and Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks.

Answer 1

Actually this is called text line extraction. What I'm going to tell you is inspired from the lectures of this scientist. For finding digits you don't need to design a network. You should extract them and then feed them to a network in turn.

First of all you have to read your image.

import cv2
import numpy as np
from matplotlib import pyplot as plt
%matplotlib inline
img = cv2.imread('./doc1.png')
plt.imshow(img)

Then you have to make your image as a binary array.

img.shape

(2360, 1649, 3)

img = cv2.imread('./doc1.png',0)
img.shape

(2360, 1649)

plt.imshow(img)

plt.imshow(img,cmap = 'gray')

The following code shows some of the lines of the document:

plt.imshow(img[900:1020,500:900],cmap = 'gray')

bimg=cv2.cvtColor(img[900:1020,500:900],cv2.COLOR_GRAY2RGB)
bimg.shape

(120, 400, 3)

Next, you have to find the lines in the image, then you have to find the characters.

Basically the code was completely clear to me. If you don't understand it, let me know.

Answer 2

In Image Processing, this task is known as localization. You basically want to localize each digit in the image and then use your digit recognizer over the digits. A cursory google search for digit localization in images gives me following papers which seem to be very helpful.

• Reading Digits in Natural Images with Unsupervised Feature Learning

• Character Recognition In Natural Images

Answer 3

I have worked on a similar condition where I needed to separate each digit. I have done this using Image segmentation, segmenting only the white pixel column wise.