# Handwritting Recognition moving from character level to word level

Given the experience on MIST, I try this problem as a character level.

I have a handwritten text and I want to "OCR" it. Even though I made progresses with openCV (on the image pre-processing, before a DNN classifier).

I think that the best approach would be to move to word level (into RNN). I am writing this in Python. But I don't know how to change the last layer of the DNN into an input layer for RNN.

Is there any code for dummies to see how is this done (for images as input of the first DNN layer)? Besides I see that you could put a language model on top (this would be a dictionary right?). Advice?

• You should give more detail: What is the exact process of your code. What is your input ? (series of pictures of characters or the entire document). Then what is your current output ? RNN seems good only if you have an ordered input. – Robin Jan 10 '17 at 15:55
• Hi. The inputs are words like "hello" but the model is training is done with "h" "e" "l" "o". That is why i think i need a RNN. Right now i am croping the letters but i think is not effcient at all. – donpresente Jan 11 '17 at 8:38
• Then your input is a sequence of images of handwritten letters. Your first output is then classes of letters like "a" = [1,0,0,0,0,...,0] ? You can first transform this output to "a" = 1, "b" = 2... and then have a second input for your RNN or whatever like "hello" = [8,5,12,12,15]. Then your second classifier will only try to guess what word was written. Also you can keep the first output as it is and use it on the RNN. But the architecture of the RNN will be a bit more complex. I might give way better results since you don't erase scores of every classes in your first NN. – Robin Jan 11 '17 at 13:01
• @debzsud thanks!1 but this force to have a language model on top right? What would the RNN need to do that classification instead? – donpresente Jan 11 '17 at 17:48
• You train your RNN with words as classes e.g. 'hello' = [1,0, ...] with your entire words dictionary. In the first case, it will only learn to correct wrong assignation like if you have 'jello', it corrects to 'hello'. The second case can distinguish mistakes: a letter 'j' can be corrected as a 'l' in some case and as a 'i' in some others depending on the scores of each letters. Plus, if you setup your RNN right, it could help with missed or mixed letters. Finally, I think a better approach is to directly train a RNN on words, without the previous CNN model for letters. – Robin Jan 12 '17 at 13:12

I do not think that you would need a recurrent neural net here. This will be much slower to train than a ConvNet. Additionally, your data is images, correct? I think you could use a deep convolutional net to a fully-connected net. This should be fast and perform well. Please look at the following archived blogs from keras.

https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html

• Thanks! The images are words. And the DNN is trained on characters. That is why i see the RNN role. Can you elaborate more, why you dont see it. – donpresente Jan 10 '17 at 15:22
• I don't see it because they are images. What you refer to as a DNN is a Convolutional Neural Net. Words/characters doesn't matter. What matters is that they are images. Yes the CNN is trained on characters, but images of characters. Therefore, this would be the same process, because you have images of words. Get it? – Samuel Sherman Jan 10 '17 at 19:58
• i see your point. and thanks for the explanation. maybe i am newbie but if i train with characters, then i cant test it on a word. Correct? – donpresente Jan 11 '17 at 8:36
• I assume you still have a dataset of images that are words. The standard approach for any data science problem would be to split your data into training and test. This way you have labels for both your testing and training data. You can predict on your test data and compare the predictions to the correct label to get a measure of performance. I would look at metrics like precision, recall, area under ROC curve, log loss. – Samuel Sherman Jan 11 '17 at 16:38
• Additionally, you could potentially segment your images by character and predict on that, if you want. That is technically the same thing, except it is predicting on a more granular level. If it guesses each character correct, then it guessed the whole word correct. – Samuel Sherman Jan 11 '17 at 16:40

I think that this "Tutorial" is what you need.

https://hackernoon.com/latest-deep-learning-ocr-with-keras-and-supervisely-in-15-minutes-34aecd630ed8

The architecture is described in this video : https://www.youtube.com/watch?v=uVbOckyUemo. It is based on a CNN and some LSTM neural networks.

I dont know if its the best architecture for this kind of stuf, but I think that it gives a good understanding on how solving the problem.

What I'm going to refer to is introducing some papers which are about this context. The papers have available data-set and there code can be accessed easily. Actually there are a lot of works in this context, but I suggeest you reading the following papers which are relevant to your question. In English, the nature of the language is in a way that you make a word using combination of letters and usually it is not needed to connect the letters to construct words. There are other languages that letters have to be connected to make words. In OCR problems of those languages it is a commen practice to consider the connected components. The papers that I'm going to refer to have a good data-set that can be used for both supervised and unsupervised methods.

The last paper is a great paper and is somehow the result of the previous ones.