Keras + Tensorflow CNN with multiple image inputs

Question

I have a CNN that needs to take in 68 images that are all 59x59 pixels. The CNN should output 136 values on the output layer

My training data has shape (-1, 68, 59, 59, 1).

My current approach is to use concatenate to join multiple networks like so:

input_layer = [None] * 68
x = [None] * 68
for i in range(68):
    input_layer[i] = tf.keras.layers.Input(shape=training_data.shape[1:][1:])
    x[i] = Conv2D(64, (5,5))(input_layer[i])
    x[i] = LeakyReLU(alpha=0.3)(x[i])
    x[i] = MaxPooling2D(pool_size=(2,2))(x[i])
    x[i] = Model(inputs=input_layer[i], outputs=x[i])

combined = concatenate(x)

However, this always gives the error:

ValueError: A `Concatenate` layer should be called on a list of at least 2 inputs

Is this approach a suitable approach or am I doing this completely wrong?

Answer 1

Yes it is wrong, each (68, 59, 59) input should go through one model not an array of them.

1. You can treat each of 68 images as a channel, for this, you need to squeeze your data axes from (-1, 68, 59, 59, 1) to (-1, 68, 59, 59) to have a 59x59 image with 68 channels corresponding to Input((68, 59, 59)), and set data_format='channels_first' in conv2D, to let the layer know that channels are in the first dimension (it expects them to be in the last dimension by default). This is similar to an RGB image that has 3 channels corresponding to Input((59, 59, 3)). The rest is the same.

2. If 68 images are consecutive frames from a movie, you can use conv3D to extract motion patterns across neighbor frames too; this is done by 3D kernels instead of 2D kernels. It requires (-1, 68, 59, 59, 1) data shape corresponding to Input((68, 59, 59, 1)). Also, we should use the default data_format='channels_last' since now there is only one channel as the last dimension. Commonly, temporal axis is placed third, i.e. (-1, 59, 59, 68, 1), which can be accomplished by moving the axes.