I am trying to build a CNN+RNN model for a computer vision problem. below is my code

def cnn_with_rnn(shape):
    model = Sequential()
    model.add(Conv2D(32, (3, 3), strides=(2, 2), activation="relu",kernel_initializer='truncated_normal',bias_initializer='truncated_normal', input_shape=shape))
    model.add(MaxPool2D(pool_size=(2, 2)))
    model.add(Conv2D(64, (3, 3), strides=(2, 2),kernel_initializer='truncated_normal',bias_initializer='truncated_normal',  activation="relu"))
    model.add(MaxPool2D(pool_size=(2, 2)))
    model.add(Conv2D(128, (3, 3), strides=(2, 2),kernel_initializer='truncated_normal',bias_initializer='truncated_normal',  activation="relu"))
    model.compile(optimizer=Adam(lr=1e-5), loss="mse", metrics=[custom_metric])

My image is a RGB image with the following shape - [66,200,3], where 3 is the number of color channels.

When i am trying to run the above code , i am getting the following error

ValueError: Input 0 is incompatible with layer lstm_3: expected ndim=3, found ndim=4

How can i combine CNN+RNN for colored images and how to solve my above problem?


2 Answers 2


It might not be directly possible to shoehorn the output of your CNN directly into an LSTM (at least without being a lot more thorough with your dimensions).

Another approach is to have a look at the Keras wrapper layer: TimeDistributed. You essentially extract features using your Conv layers as usual, but do that over time-steps, not just a random mini-batch. The features the come out then are e.g. over 5 consecutive timeframes -> this would be one single sample for the LSTM. If you perform a minibatch of say 10 samples, this means 10 * 5 = 50 input frames.

Straight from the linked documentation:

This wrapper applies a layer to every temporal slice of an input.

The input should be at least 3D, and the dimension of index one will be considered to be the temporal dimension.

Consider a batch of 32 samples, where each sample is a sequence of 10 vectors of 16 dimensions. The batch input shape of the layer is then (32, 10, 16), and the input_shape, not including the samples dimension, is (10, 16).

You can then use TimeDistributed to apply a Dense layer to each of the 10 timesteps, independently:

# as the first layer in a model
model = Sequential()
model.add(TimeDistributed(Dense(8), input_shape=(10, 16)))
# now model.output_shape == (None, 10, 8)

As a hint for dimensions, make use of model.summary() on a compiled model in order to inspect the dimensions of each layer through the model. You will likely have to get a pen and paper out to work through the dimensions, as things start to become a little complicated when using such a wrapper layer.

Unfortunately I don't have any links/tutorials to show how this is exactly done in Keras, but hopefully the description above gives you enough info for some useful Bing! searches. Just kiddin... Google searches ;-)

  • $\begingroup$ i was able to find the problem and solve it. Now, i am able to build a CNN+RNN using keras. Thanks for the support. your answer helped me a bit to get started, but..i think, i will write an answer soon $\endgroup$
    – rawwar
    Aug 29, 2018 at 19:12
  • $\begingroup$ @InAFlash - I look forward to reading your answer! :-) $\endgroup$
    – n1k31t4
    Aug 29, 2018 at 21:11
  • $\begingroup$ i have answered it. take a look $\endgroup$
    – rawwar
    Sep 1, 2018 at 12:35

the best way to do CNN+LSTM is using Time distributed layer. following code show's how we can add Time Distributed layer

model = Sequential()
model.add(TimeDistributed(Conv2D(24, 5, 5,activation='relu',subsample=(5, 4),
      border_mode='valid'), input_shape=input_shape))
model.add(LSTM(64, return_sequences=True))

So, here input_shape should be 4 dimensional as shown below


You have to prepare data for this such that, each data point have n points where n is the number of time steps. So, if you are using 5 time steps, your data point should have a list of 5 elements which are past 4 time steps data along with current step.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.