# How to deal with different length entities in a Keras DataGenerator?

I'm solivng a prediction problem where I need to predict the demand of multiple articles based on their performance during the last 7 days. To get the most out of the data I am trying to implement a rolling window approach where I use 7 days as the training input (x) and the 8th day's performance as the training label (y).
As I am using Keras as a deep learning framework, how do I combine this rolling window approach with a custom data generator that feeds my network during training? The different articles have different data lengths (i.e. article A has 200 days worth of data, article B only has 10 days worth of data).
I implemented a version where I only use one sample per article but that drastically decreases the amount of data I have for training.

I faced a similar problem and the quick solution I came up with was iterate through the main data set, create an instance of TimeseriesGenerator for each of the subsets and feed it to the model. So in your case that would be: go through the collection of articles, instantiate TimeseriesGenerator for each article and feed the windowed samples to your model, ie.:

from keras.preprocessing.sequence import TimeseriesGenerator

# Set the training parameters
window_length = ...
ix0, ix1 = ...
batch_size = ...

for article in articles:
# Insert your code to generate the article's features and targets
x, y = ...

model.fit_generator(TimeseriesGenerator(
x,
y,
window_length,
batch_size=batch_size,
start_index=ix0,
end_index=ix1
),
steps_per_epoch=int(np.ceil((x.shape[0] - ix0 - ix1 - window_length)/float(batch_size))),
epochs=50,
verbose=1
)


Now depending on your dataset size and composition you can either:

• run a number of epochs per article
• or train one article per epoch, but for that you would need another loop.