4
$\begingroup$

I am trying to make a Sudoku solver and for the image recognition I trained a CNN but the problem that I am facing is that I don't know how to make it see a clear distinction between numbers and blank images. (My neural network is trained for MNIST data set only)

For example in a Sudoku like this :

enter image description here

I want the classifier to classify the blank spaces as "0"

Here is what I have already tried:

import numpy as np
import cv2
from PIL import Image
import pytesseract
import matplotlib.pyplot as plt
from tensorflow import keras



#open the image
img = Image.open(r'D:\\D_Apps\\Sudoku Solver\\image\\1_9Tgak3f8JPcn1u4-cSGYVw.png').convert('LA')

#take only the brightness value from each pixel of the image
array = np.array(img)[:,:,0]

#invert the image (this is how MNIST digits is formatted)
array = 255-array

#this will be the width and length of each sub-image
divisor = array.shape[0]//9

puzzle = []
for i in range(9):
    row = []
    for j in range(9):
        #slice image, reshape it to 28x28 (mnist reader size)
        row.append(cv2.resize(array[i*divisor:(i+1)*divisor,
                                    j*divisor:(j+1)*divisor][3:-3, 3:-3], #the 3:-3 slice removes the borders from each image
                              dsize=(28,28), 
                              interpolation=cv2.INTER_CUBIC))
    puzzle.append(row)

model = keras.models.load_model(r'C:\Users\Ankit\MnistModel.h5')

template = [
    [0 for _ in range(9)] for _ in range(9)
]

for i, row in enumerate(puzzle):
    for j, image in enumerate(row):
        #if the brightness is above 6, then use the model
        if np.mean(image) > 6:
            #this line of code sets the puzzle's value to the model's prediction
            #the preprocessing happens inside the predict call
            template[i][j] = model.predict_classes(image.reshape(1,28,28,1) \
                                                   .astype('float32')/255)[0]
            
print(template)

(read about this in a blog)

This algorithm took the average brightness and checked if the other cells had brightness less than 2 and classified them as blanks.

But this algorithm dose not work if the image did not have a white background for example the output for this image:

enter image description here

the output was :

[[7, 7, 0, 7, 7, 7, 1, 7, 1], [2, 1, 8, 1, 8, 1, 1, 0, 8], [7, 7, 1, 8, 8, 1, 7, 7, 7], [7, 1, 1, 1, 1, 1, 1, 0, 1], [7, 7, 1, 1, 7, 8, 1, 7, 7], [8, 7, 8, 1, 7, 7, 7, 4, 9], [7, 1, 1, 1, 1, 0, 7, 8, 7], [7, 4, 7, 8, 8, 7, 7, 7, 4], [2, 7, 7, 7, 8, 0, 4, 7, 7]]

What can I do to improve this? Should I retrain my model to work with other image colors? Or Should i retrain the model with blank spaces? If so how can I find the dataset? I have done a lot of research but can't find a clear answer to my questions

$\endgroup$

1 Answer 1

2
$\begingroup$

Try thresholding on the image. I believe, you will get ~95% of what is required.
Then try other classical image processing techniques depending on the issue.

import cv2 as cv
import numpy as np
from matplotlib import pyplot as plt
img = cv.imread('/content/sample_data/issue_image.png',0)
ret,thresh1 = cv.threshold(img,127,255,cv.THRESH_BINARY)

_, ax = plt.subplots(1,2,figsize=(12,6))
ax[1].imshow(thresh1,'gray')
ax[0].imshow(img)

enter image description here

Ref - OpenCV py_thresholding

$\endgroup$
4
  • $\begingroup$ I'll try it out. Thanks for answering. $\endgroup$ Jul 20, 2020 at 9:01
  • $\begingroup$ So i tried using a bunch of different thresholding techniques, Adaptive gaussian seemed to perform better but the results were not accurate. Do u think the answer is coming wrong maybe because I am considering the size of each cell too big? Because the border in each cell can also cause problems in classification right? $\endgroup$ Jul 21, 2020 at 10:33
  • $\begingroup$ I saw it coming. Data cleaning is a tough stuff for any type of data i.e. text/tabular/image. Is your model expecting images the same as your first sample image? $\endgroup$
    – 10xAI
    Jul 21, 2020 at 11:06
  • $\begingroup$ Yes I think so. I trained my model on MNIST but I guess the images that are being input are different since they have a border around them. @10xAI $\endgroup$ Jul 23, 2020 at 2:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.