# Document Image Layout Classification

I am trying to create a OCR model that can extract text from images of documents. There are two types of resumes.

1. Resumes that are kind of plain in terms of 'layout' (unlike the resumes that I have attached)
2. Well designed resumes like the one I have attached along with this post

Before I send the document image into the "OCR", I need to classify if the resume belongs to Type 1 or 2

So what is the best way to approach the 'classification' idea?

• As a first check, you can use Python to check if there is a significant amount of color that deviates from black and white. Next, you could filter by a lenient "range" for the black:white pixel ratio. This will help you filter out white text on black images. I'd think that this would capture majority of your cases. I don't think there's a need to go into a "deep-learning" solution given your simple problem – Jason K Lai Jan 8 at 18:18
• @JasonKLai I have updated my post. Pease give it a read, again – Deepak Sharma Jan 8 at 18:36
• I have many resumes which have complex layouts, that are visually challenging for OCR. I want to separate it into multiple classes in the future.. So a deep learning solution is required – Deepak Sharma Jan 8 at 18:37
• Based on the examples in the question (as of this comment), the two checks I described will immediately tell you that the image examples are of type 2. Then, you can feed it to the appropriate OCR. The checks can be in an automated data pipeline. – Jason K Lai Jan 8 at 18:41
• In the future, I might want to change this binary classifier into multiple classes... So if I follow the approach that you proposed, I cannot scale it up when I need to do so. – Deepak Sharma Jan 8 at 18:45

Well I kind did something like what you are trying to do. I have a lot of images that are very similar so I tried to "search" for then using the following code (I simplify ):

def rmsdiff(im1, im2):
"Calculate the root-mean-square difference between two images"
diff = ImageChops.difference(im1, im2)
h = diff.histogram()
sq = (value*((idx%256)**2) for idx, value in enumerate(h))
sum_of_squares = sum(sq)
rms = math.sqrt(sum_of_squares/float(im1.size[0] * im1.size[1]))
return rms

im1 = Image.open(pathOfKnowImage)
folder_img = os.listdir(pathOfUnknowImages)
for arq in folder_img:
if arq.endswith('.jpg'):
im2 = Image.open('pathOfUnknowImages/%s' % arq)
v = rmsdiff(im1,im2)
if v < 0.2:
print('very similar')
else:
print('diferent')


For my understanding the layout of the resumes would be the same (colors, boxes, font sizes) so using this range 0-0.2 you might find and classify the resumes.

You could take 3 types os resumes and 10 or mores of this resumes in one folder and try the code above, you could move the images to folders and later see if it is working