How to pass features extracted using CNN into RNN?

I have word images as below:

Let's say it's a 256x64 image. My aim is to extract the text from the image as 73791096754314441539 which is basically what an OCR does.
I am trying to build model which can recognise word from images.
When I am saying word it can be any of the follwing:

1. Any dictionary word, non-dictionary word
2. a-z,A-Z, special characters including spaces

I have built a model (snippet because of company policies) in tensorflow as below:

inputs = tf.placeholder(tf.float32, [common.BATCH_SIZE, common.OUTPUT_SHAPE[1], common.OUTPUT_SHAPE[0], 1])
# Here we use sparse_placeholder that will generate a
# SparseTensor required by ctc_loss op.
targets = tf.sparse_placeholder(tf.int32)

# 1d array of size [batch_size]
seq_len = tf.placeholder(tf.int32, [common.BATCH_SIZE])

model = tf.layers.conv2d(inputs, 64, (3,3),strides=(1, 1), padding='same', name='c1')
model = tf.layers.max_pooling2d(model, (3,3), strides=(2,2), padding='same', name='m1')
model = tf.layers.conv2d(model, 128,(3,3), strides=(1, 1), padding='same', name='c2')
model = tf.layers.max_pooling2d(model, (3,3),strides=(2,2), padding='same', name='m2')
model = tf.transpose(model, [3,0,1,2])
shape = model.get_shape().as_list()
model = tf.reshape(model, [shape[0],-1,shape[2]*shape[3]])

cell = tf.nn.rnn_cell.LSTMCell(common.num_hidden, state_is_tuple=True)
cell = tf.nn.rnn_cell.DropoutWrapper(cell, input_keep_prob=0.5, output_keep_prob=0.5)
stack = tf.nn.rnn_cell.MultiRNNCell([cell]*common.num_layers, state_is_tuple=True)

outputs, _ = tf.nn.dynamic_rnn(cell, model, seq_len, dtype=tf.float32,time_major=True)


My current approach is to use take input a word image pass it through a CNN extract high level image features, convert the image features to sequential data as below
[[a1,b1,c1],[a2,b2,c2],[a3,b3,c3]] -> [[a1,a2,a3],[b1,b2,b3],[c1,c2,c3]]
then pass it through a RNN(LSTM or BiLSTM), then use CTC (Connectionist Temporal Loss) to find the loss and train network.
I am not getting results as expected, I wanted to know if:

1. There is some other better way to do this task
2. If I am converting features to sequence correctly
3. Any research paper where something like this is done.

1 and 2. You are in the right direction, you need to extract the features using a CNN, then instead of predicting the class you want to reshape the last layer of features and feed it directly into the RNN.

A couple of things to pay attention to:

• With a rather shallow CNN you aren't taking advantage of the higher-level feature extraction these architectures can offer. If all your images are as simple as the example you've shown it is adequate.
• If you are considering a larger CNN, along with the RNN there is a substantial number of parameters to be trained. For this you need lots of data and lots of computational resources (either very strong GPUs or time).
• In order for you to get the best of the two, I would suggest incorporating a pre-trained CNN into your model (and just fine tune the latter layers). This pre-trained model can even be trained in generic images (i.e. ImageNet) and will substantially increase the performance of the CNN without computational cost. You then can train the latter layers of this CNN jointly with the RNN.

3.This is a good example of what you are trying to do. They basically try to recognize text from street photographs among other things with the same methodology you describe.

Similar methodologies can be found in other research domains such as multi-label image classification, sequence labelling, facial expression recognition, etc

• One problem/ bottleneck with the approach that I am using is that since the CNN models have fixed input image size, in case I have a longer word I have to decrease either font_size or image dimension which is going to effect my accuracy. What do you think ? But if I only use a BiLSTM as in ocropy I can feed in image of various dimension ( probably, still experimenting ), What's your opinion on this ? Jul 11 '17 at 14:43
• Yes the input shape must stay the same, meaning that you have to lower your resolution, which could result in a reduction of accuracy. There is another approach you could consider, I'm not sure if it would work though: Object recognition. You could try having a system that makes region proposals (of 1 number each) and have a second simple CNN trained on MNIST try to classify those proposals. Then concatenate all the outputs into a single number. Jul 12 '17 at 11:53