Shaping data for ConvLSTM for many-to-one image model

Question

Ultimately, I am trying to obtain a binary segmentation mask for an image sequence.

I have n number of image sequences, each with 500 greyscale images of size 256px by 400px. Each of these sequences is matched to a single segmentation mask, or image of shape 256px by 400px. This gives:

input shape : (n, 500, 256, 400, 1)
output shape: (n,   1, 256, 400, 1)

Understandably, and as explained in this blog post it is necessary to reshape my input array into sequence fragments (time steps) because LSTMs prefer sequences smaller than sequence_length = 400 (my GPU cant handle anything larger than 100 anyway). So, with this in mind, I have chosen a sequence "fragment" length of 50, or 50 time steps. Without overlap this brings the shape of a single input sample:

from:      (500, 256, 400, 1)
to  : ( 10,  50, 256, 400, 1)

In my mind it made sense to feed a single sample through the ConvLSTM2D at a time with a data generator, to have:

input shape : (10, 50, 256, 400, 1)
output shape: ( 1,  1, 256, 400, 1)

I thought this meant that the network would associate the 500 images with a single output image ie. remembering the state across the 10 fragment sequences of 50 images each in order to result in a single image output and computing the loss function only when the network has seen all 500 images for example. It did not make complete sense to me when I did it and now I have proven it to make even less sense because I have gotten the following error:

 ValueError: Input arrays should have the same number of samples as target arrays. Found 10 input samples and 1 target samples.

So I am thus a bit stuck with a flawed understanding of Keras's LSTMS and how to get this ConvLSTM2D to manage these image sequences in the way that I want ie. how I might be able to get a many-images (of a fairly long sequence) to one-image model to work. I can simply duplicate the output image for each of the sequence fragments so that there is a dimension match but this is not the problem I am trying to solve because the segmented image is a summary across the entire sequence, and not across sequence fragments.

Answer 1

The problem was for each ConvLSTM layer I was using keep_dims = True which means that the number of dimensions in the input is reflected in the ouput. To obtain a many to one model I just needed to set keep_dims = False. I was then able to manipulate the size of the output using dense layers and convolutions. In this way I could have a video sequence as an input and a single 2D image as the output and I was able to obtain the relationship:

input shape : (10, 50, 256, 400, 1) output shape: (10, 256, 400, 1)

Where I was able to get a 2D segmentation for each fragment in the video sequence.

It is not exactlly what I needed, but was much better than getting a segmentation result for every frame in the video sequence.