ValueError in CNN+RNN model in keras

Question

I am trying to build a CNN+RNN model for a computer vision problem. below is my code

def cnn_with_rnn(shape):
    model = Sequential()
    model.add(Conv2D(32, (3, 3), strides=(2, 2), activation="relu",kernel_initializer='truncated_normal',bias_initializer='truncated_normal', input_shape=shape))
    model.add(MaxPool2D(pool_size=(2, 2)))
    model.add(Conv2D(64, (3, 3), strides=(2, 2),kernel_initializer='truncated_normal',bias_initializer='truncated_normal',  activation="relu"))
    model.add(MaxPool2D(pool_size=(2, 2)))
    model.add(Conv2D(128, (3, 3), strides=(2, 2),kernel_initializer='truncated_normal',bias_initializer='truncated_normal',  activation="relu"))
    model.add(LSTM(50))
    model.add(Dense(1))
    model.compile(optimizer=Adam(lr=1e-5), loss="mse", metrics=[custom_metric])

My image is a RGB image with the following shape - [66,200,3], where 3 is the number of color channels.

When i am trying to run the above code , i am getting the following error

ValueError: Input 0 is incompatible with layer lstm_3: expected ndim=3, found ndim=4

How can i combine CNN+RNN for colored images and how to solve my above problem?

Answer 1

It might not be directly possible to shoehorn the output of your CNN directly into an LSTM (at least without being a lot more thorough with your dimensions).

---

Another approach is to have a look at the Keras wrapper layer: TimeDistributed. You essentially extract features using your Conv layers as usual, but do that over time-steps, not just a random mini-batch. The features the come out then are e.g. over 5 consecutive timeframes -> this would be one single sample for the LSTM. If you perform a minibatch of say 10 samples, this means 10 * 5 = 50 input frames.

Straight from the linked documentation:

This wrapper applies a layer to every temporal slice of an input.

The input should be at least 3D, and the dimension of index one will be considered to be the temporal dimension.

Consider a batch of 32 samples, where each sample is a sequence of 10 vectors of 16 dimensions. The batch input shape of the layer is then (32, 10, 16), and the input_shape, not including the samples dimension, is (10, 16).

You can then use TimeDistributed to apply a Dense layer to each of the 10 timesteps, independently:

# as the first layer in a model
model = Sequential()
model.add(TimeDistributed(Dense(8), input_shape=(10, 16)))
# now model.output_shape == (None, 10, 8)

---

As a hint for dimensions, make use of model.summary() on a compiled model in order to inspect the dimensions of each layer through the model. You will likely have to get a pen and paper out to work through the dimensions, as things start to become a little complicated when using such a wrapper layer.

Unfortunately I don't have any links/tutorials to show how this is exactly done in Keras, but hopefully the description above gives you enough info for some useful Bing! searches. Just kiddin... Google searches ;-)

Answer 2

the best way to do CNN+LSTM is using Time distributed layer. following code show's how we can add Time Distributed layer

model = Sequential()
model.add(TimeDistributed(Conv2D(24, 5, 5,activation='relu',subsample=(5, 4),
      border_mode='valid'), input_shape=input_shape))
model.add(TimeDistributed(Flatten()))
model.add(LSTM(64, return_sequences=True))
model.add(Dense(output_dim=1))

So, here input_shape should be 4 dimensional as shown below

input_shape=(TimeSteps,height,width,color_channels)

You have to prepare data for this such that, each data point have n points where n is the number of time steps. So, if you are using 5 time steps, your data point should have a list of 5 elements which are past 4 time steps data along with current step.