4
$\begingroup$

Given the experience on MIST, I try this problem as a character level.

I have a handwritten text and I want to "OCR" it. Even though I made progresses with openCV (on the image pre-processing, before a DNN classifier).

I think that the best approach would be to move to word level (into RNN). I am writing this in Python. But I don't know how to change the last layer of the DNN into an input layer for RNN.

Is there any code for dummies to see how is this done (for images as input of the first DNN layer)? Besides I see that you could put a language model on top (this would be a dictionary right?). Advice?

Improve this question
$\endgroup$
6
  • $\begingroup$ You should give more detail: What is the exact process of your code. What is your input ? (series of pictures of characters or the entire document). Then what is your current output ? RNN seems good only if you have an ordered input. $\endgroup$
    – Robin
    Jan 10, 2017 at 15:55
  • $\begingroup$ Hi. The inputs are words like "hello" but the model is training is done with "h" "e" "l" "o". That is why i think i need a RNN. Right now i am croping the letters but i think is not effcient at all. $\endgroup$
    – donpresente
    Jan 11, 2017 at 8:38
  • $\begingroup$ Then your input is a sequence of images of handwritten letters. Your first output is then classes of letters like "a" = [1,0,0,0,0,...,0] ? You can first transform this output to "a" = 1, "b" = 2... and then have a second input for your RNN or whatever like "hello" = [8,5,12,12,15]. Then your second classifier will only try to guess what word was written. Also you can keep the first output as it is and use it on the RNN. But the architecture of the RNN will be a bit more complex. I might give way better results since you don't erase scores of every classes in your first NN. $\endgroup$
    – Robin
    Jan 11, 2017 at 13:01
  • $\begingroup$ @debzsud thanks!1 but this force to have a language model on top right? What would the RNN need to do that classification instead? $\endgroup$
    – donpresente
    Jan 11, 2017 at 17:48
  • $\begingroup$ You train your RNN with words as classes e.g. 'hello' = [1,0, ...] with your entire words dictionary. In the first case, it will only learn to correct wrong assignation like if you have 'jello', it corrects to 'hello'. The second case can distinguish mistakes: a letter 'j' can be corrected as a 'l' in some case and as a 'i' in some others depending on the scores of each letters. Plus, if you setup your RNN right, it could help with missed or mixed letters. Finally, I think a better approach is to directly train a RNN on words, without the previous CNN model for letters. $\endgroup$
    – Robin
    Jan 12, 2017 at 13:12

3 Answers 3

Reset to default
1
$\begingroup$

What I'm going to refer to is introducing some papers which are about this context. The papers have available data-set and there code can be accessed easily. Actually there are a lot of works in this context, but I suggeest you reading the following papers which are relevant to your question. In English, the nature of the language is in a way that you make a word using combination of letters and usually it is not needed to connect the letters to construct words. There are other languages that letters have to be connected to make words. In OCR problems of those languages it is a commen practice to consider the connected components. The papers that I'm going to refer to have a good data-set that can be used for both supervised and unsupervised methods.

The last paper is a great paper and is somehow the result of the previous ones.

Improve this answer
$\endgroup$
1
$\begingroup$

I think that this Tutorial is what you need.

The architecture is described in this video. It is based on a CNN and some LSTM neural networks.

I dont know if its the best architecture for this kind of stuf, but I think that it gives a good understanding on how solving the problem.

Improve this answer
$\endgroup$
1
$\begingroup$

I do not think that you would need a recurrent neural net here. This will be much slower to train than a ConvNet. Additionally, your data is images, correct? I think you could use a deep convolutional net to a fully-connected net. This should be fast and perform well. Please look at the following archived blogs from keras.

Building powerful image classification models using very little data

How convolutional neural networks see the world

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thanks! The images are words. And the DNN is trained on characters. That is why i see the RNN role. Can you elaborate more, why you dont see it. $\endgroup$
    – donpresente
    Jan 10, 2017 at 15:22
  • $\begingroup$ I don't see it because they are images. What you refer to as a DNN is a Convolutional Neural Net. Words/characters doesn't matter. What matters is that they are images. Yes the CNN is trained on characters, but images of characters. Therefore, this would be the same process, because you have images of words. Get it? $\endgroup$
    – Samuel Sherman
    Jan 10, 2017 at 19:58
  • $\begingroup$ i see your point. and thanks for the explanation. maybe i am newbie but if i train with characters, then i cant test it on a word. Correct? $\endgroup$
    – donpresente
    Jan 11, 2017 at 8:36
  • $\begingroup$ I assume you still have a dataset of images that are words. The standard approach for any data science problem would be to split your data into training and test. This way you have labels for both your testing and training data. You can predict on your test data and compare the predictions to the correct label to get a measure of performance. I would look at metrics like precision, recall, area under ROC curve, log loss. $\endgroup$
    – Samuel Sherman
    Jan 11, 2017 at 16:38
  • $\begingroup$ Additionally, you could potentially segment your images by character and predict on that, if you want. That is technically the same thing, except it is predicting on a more granular level. If it guesses each character correct, then it guessed the whole word correct. $\endgroup$
    – Samuel Sherman
    Jan 11, 2017 at 16:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.