I'm experimenting with OCR on book spines, as a way of cataloguing books which are on shelves without the tedium of taking out each one and scanning a barcode.
I need a way to tell if I have successfully done OCR or not on each book spine and extract the text from within that section of the image. That way I would know if I'd missed a book, and it would then help when searching to find out which book it is - because I won't have multiple book titles/authors in the same chunk of text.
Ideally I would use something which was open source and could run on the browser side in real time on a video. There are vendors who offer that for native mobile apps, but it may be too ambitious for HTML5. So I'm letting the user click a button, then grabbing a video frame, uploading it to a server, and using Google Vision to do OCR. Vision seems to have better OCR than Tesseract.js in my tests, though I have not experimented with the very many Tesseract input parameters.
What I think I need is something (ideally in JS which can run in a browser) which can identify visually similar rectangles in an image, to identify each spine (and ideally superimpose it on the video feed). With Vision I could correlate this with the boundingPoly of the returned annotations, which would then allow me to determine what if anything I had managed to OCR for each spine. With Tesseract the corresponding thing might be to use Regions of Interest.
It seems like it ought to be possible to do this because most book spines are fairly uniform in colour, and will have visual boundaries at the edge of the book. The adjacent book will often have a different spine colour, which also ought to help.
So, does anyone have any experience/suggestions/algorithms for identifying rough rectangles of interest from an image and then using OCR on them?