1
$\begingroup$

Ultimately, I am trying to obtain a binary segmentation mask for an image sequence.

I have n number of image sequences, each with 500 greyscale images of size 256px by 400px. Each of these sequences is matched to a single segmentation mask, or image of shape 256px by 400px. This gives:

input shape : (n, 500, 256, 400, 1)
output shape: (n,   1, 256, 400, 1)

Understandably, and as explained in this blog post it is necessary to reshape my input array into sequence fragments (time steps) because LSTMs prefer sequences smaller than sequence_length = 400 (my GPU cant handle anything larger than 100 anyway). So, with this in mind, I have chosen a sequence "fragment" length of 50, or 50 time steps. Without overlap this brings the shape of a single input sample:

from:      (500, 256, 400, 1)
to  : ( 10,  50, 256, 400, 1)

In my mind it made sense to feed a single sample through the ConvLSTM2D at a time with a data generator, to have:

input shape : (10, 50, 256, 400, 1)
output shape: ( 1,  1, 256, 400, 1)

I thought this meant that the network would associate the 500 images with a single output image ie. remembering the state across the 10 fragment sequences of 50 images each in order to result in a single image output and computing the loss function only when the network has seen all 500 images for example. It did not make complete sense to me when I did it and now I have proven it to make even less sense because I have gotten the following error:

 ValueError: Input arrays should have the same number of samples as target arrays. Found 10 input samples and 1 target samples.

So I am thus a bit stuck with a flawed understanding of Keras's LSTMS and how to get this ConvLSTM2D to manage these image sequences in the way that I want ie. how I might be able to get a many-images (of a fairly long sequence) to one-image model to work. I can simply duplicate the output image for each of the sequence fragments so that there is a dimension match but this is not the problem I am trying to solve because the segmented image is a summary across the entire sequence, and not across sequence fragments.

$\endgroup$
1
$\begingroup$

The problem was for each ConvLSTM layer I was using keep_dims = True which means that the number of dimensions in the input is reflected in the ouput. To obtain a many to one model I just needed to set keep_dims = False. I was then able to manipulate the size of the output using dense layers and convolutions. In this way I could have a video sequence as an input and a single 2D image as the output and I was able to obtain the relationship:

input shape : (10, 50, 256, 400, 1) output shape: (10, 256, 400, 1)

Where I was able to get a 2D segmentation for each fragment in the video sequence.

It is not exactlly what I needed, but was much better than getting a segmentation result for every frame in the video sequence.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.