I am currently facing an issue with identifying sections within a check images, something like object identification. Initially it seemed I could use YOLOv5, because it is good with object detection. I was able to test the model performance on a problem where I had to identify certain objects like standing human, chicken, other objects etc., and it had impressive results.
Then I decided to see how well it would do with identifying portions with the check images, and I had already felt it may not work. Below is a concept sample I used for annotation, (I am using real check images for model training).
which I annotated something like below with several hundred samples.
And as I expected this did not go any where, the mAP@.5 value came about 0.355, which is very low.
What I understand is that the problem at the core is very different than how YOLO looks at a problem and how the DL is prepared for YOLO. For object identification YOLO divides the annotated sections into grids and then looks at the silhouette lines of the annotated objects and other features like color and it's label.
However, when we have the features like addresses and names these annotated objects are not truly represented as silhouette, and we human ourselves also do not say a particular text portion is address by just looking at the outline, but by reading it.
So, what's next?
- I was thinking of applying TesseractOCR (which I am already able to do successfully) and then write regex rules to find the addresses, but this does not entirely address the problem. Because I am more interested in predicting the text portion that are address vs date vs check-numeral on the check itself.
- Another think I was thinking was to apply text recognition, something like described here https://github.com/awslabs/handwritten-text-recognition-for-apache-mxnet , but this also is different. This method identifies printed text vs handwritten text, and also OCRs the text out of the handwritten text, which does not exactly match my use case.
I am not able to find any discussion or tutorials that touches on labeling of portions within check or postcard etc. of any sort. And I know we are dealing with two separate problems at once here, which needs to be put together. To say something is an address vs date, the model should be able to segment the area that contains text and then also read the text to say its an address vs date.
I have been thinking around this for few days already and scouring the internet, but if someone can share something useful that would be great.