TimeDistributed with different input / output sequence length

Question

I'm looking into using TimeDistributed in my LSTM to see if it would improve the accuracy of my model. I'll be honest, I'm still not 100% sure what the specific use case for TimeDistributed, I just thought I'd give it a try and see if it improved my accuracy. Aside from blind trial/error, why would someone use TimeDistributed?

Now, my next question is probably tied to my lack of understanding of exactly what TimeDistributed's specific use case is, but I understand that the output from TimeDistributed is a sequence, and that sequence is the same length as the input sequence because you use return_sequences=True on the final layer of the model (all of them in my case, as it's stateful).

In my case I'm inputting a sequence which is 100 length, and I'm then forecasting a sequence of length 3 from that input sequence, but I get this error:

Error when checking target: expected time_distributed_1 to have shape (100, 3) but got array with shape (3, 1)

Am I right in assuming that you can only use TimeDistributed if your output sequence is the same length as the input sequence?

Many thanks

Answer 1

I have looked for an answer to this question for quite a while, and couldn't really find a clear yes/no answer.

From what I understand, TimeDistributed really only works for similar input/output lengths. According to what Jason Brownlee writes in the comments section of his TimeDistributed tutorial, he would treat the different input/output lengths as a seq2seq/encoder-decoder problem (which he describes here).

If you find a better answer, or find a way to make TimeDistributed work with different output sizes, please let me know.

---

*Update (copying my comment from below, with better formatting):

Actually, I have found a different solution which does use TimeDistributed: You can trim the output sequence you get from TimeDistributed using Cropping1D layer. So that my model now looks like this:

model.add(LSTM(hidden_size, return_sequences=True, input_shape=(num_input_timepoints, 1) ,dropout=0.05)
model.add(TimeDistributed(Dense(1))) 
model.add(Cropping1D(cropping=(num_input_timepoints - num_output_timepoints,0))) # cropping the end 

Which, in your case, would trim the first 97 time points and would leave you with the last 3. That would fit the shape of your target data.

Answer 2

Linking this.

My understanding (which I am still working on) is to set an initial LSTM layer with your input sequence and return_sequences set to False. Then add a RepeatVector layer which simply repeats the output from your last timestep of the above LSTM layer N times. Input this into another LSTM layer with return_sequences=True and a final TimeDistributed(Dense(1)) layer.

Like so:

model = Sequential()
model.add(LSTM(units = UNITS , input_shape = (input_shape), return_sequences = False)) 
model.add(RepeatVector(N))  
model.add(LSTM(UNITS, return_sequences=True))
model.add(TimeDistributed(Dense(1)))

I am getting this from the linked post I've provided. This makes intuitive sense to me for the following reasons:

For a "many to many" LSTM model with varying output size, the first LSTM layer learns the pattern of the first timesteps number of samples and makes a prediction for the last timestep; the RepeatVector layer then repeats this output N times. The next LSTM layer uses this same output prediction as the first hidden state fed into each of the timesteps we are trying to predict (I think). I would really like some clarification on this part.

Finally, the TimeDistributed(Dense(1))) passes each timestep you are wanting to predict into a single Dense node for the final prediction. This returns an output vector of (timesteps, feature(s?)).

Note: I think this is a particularly tricky problem and is a workaround for a type of problem that Keras doesn't explicitly support. I would appreciate any corrections from more folks more experienced in the field, as I'm still learning as well!