Ultimately, I am trying to obtain a binary segmentation mask for an image sequence.
I have n number of image sequences, each with 500 greyscale images of size 256px by 400px. Each of these sequences is matched to a single segmentation mask, or image of shape 256px by 400px. This gives:
input shape : (n, 500, 256, 400, 1) output shape: (n, 1, 256, 400, 1)
Understandably, and as explained in this blog post it is necessary to reshape my input array into sequence fragments (time steps) because LSTMs prefer sequences smaller than sequence_length = 400 (my GPU cant handle anything larger than 100 anyway). So, with this in mind, I have chosen a sequence "fragment" length of 50, or 50 time steps. Without overlap this brings the shape of a single input sample:
from: (500, 256, 400, 1) to : ( 10, 50, 256, 400, 1)
In my mind it made sense to feed a single sample through the ConvLSTM2D at a time with a data generator, to have:
input shape : (10, 50, 256, 400, 1) output shape: ( 1, 1, 256, 400, 1)
I thought this meant that the network would associate the 500 images with a single output image ie. remembering the state across the 10 fragment sequences of 50 images each in order to result in a single image output and computing the loss function only when the network has seen all 500 images for example. It did not make complete sense to me when I did it and now I have proven it to make even less sense because I have gotten the following error:
ValueError: Input arrays should have the same number of samples as target arrays. Found 10 input samples and 1 target samples.
So I am thus a bit stuck with a flawed understanding of Keras's LSTMS and how to get this ConvLSTM2D to manage these image sequences in the way that I want ie. how I might be able to get a many-images (of a fairly long sequence) to one-image model to work. I can simply duplicate the output image for each of the sequence fragments so that there is a dimension match but this is not the problem I am trying to solve because the segmented image is a summary across the entire sequence, and not across sequence fragments.