0
$\begingroup$

I've been having performance issues with my LSTM implementation. Whenever I use a sliding window, the performance seems to get better. Moreover the size of the window seems to have a large impact on the performance of the LSTM. I'd really like to understand why this is the case.

Background of the problem I am applying LSTM to:
I'm applying LSTM to the prediction of future events. These events are recurring in the sense that they follow some pattern (e.g going to the gym every other day, visiting family every 7 days, etc.). This pattern is relatively steady over time but not always the same (e.g. an event following a 7-day pattern may also sometimes occur 8-10 days after the last one).

In the simple case of one event I can just let a 0 denote a non-occurence and a 1 denote an occurence. Without a sliding window, my data could then look like: $x\_train = [0,0,0,1,0,0,0,1]^T \text{and }y\_train = [0,0,1,0,0,0,1,0]^T$ showing a four day pattern.

What I mean by a sliding window:
With a window_size of two, the above data would transform into: $x\_train = [[0,0]^T,[0,0]^T,[0,1]^T,[1,0]^T,[0,0]^T,[0,0]^T]^T \text{and} y\_train = [0,1,0,0,0,1]^T$. So we use the previous $window\_size$ days to predict the next day.

Findings with and without sliding window:
Now without a sliding window, the LSTM performs very poorly for the problem above. In particular it predicts the same probability of an event occuring for almost every day, except the one day right after an event occuring which gets a lower probability. The predicted probabilties are also very low.

When I apply a sliding window, the probabilities start to become both larger and more distinct. When I choose 'one of the best' window sizes I even get very good predictions (<1% for non-occurences and 80+% for occurences).

I cannot really find an explanation for this anywhere, although I have found multiple people reporting that sliding windows improved their performance. I would like to at least have an intuitive idea of why sliding windows can help LSTMs perform better. It would also be nice to know how and why the window_size matters. It could also be that I'm not applying LSTM to my problem correctly, so to be sure I included code of my implementation below. If someone can help me with understanding this better, that would be greatly appreciated.

Some Python code for more context as to what I'm doing:

n_days = len(bigarray) #bigarray contains my data for every day
train_days = n_days - 200
test_days = n_days - train_days
window_length = 30
n_trainbatches = train_days - window_length
n_testbatches = test_days - window_length

data_days_train = bigarray[0:train_days]
data_days_test = bigarray[train_days:]

x_train = np.zeros((n_trainbatches,window_length,len(bigarray[0])))
y_train = np.zeros((n_trainbatches,len(bigarray[0]))) 
x_test = np.zeros((n_testbatches,window_length,len(bigarray[0])))
y_test = np.zeros((n_testbatches,len(bigarray[0])))

for i in range(0,n_trainbatches):
    x_train[i] = data_days_train[i : (i + window_length)]
    y_train[i] = data_days_train[i + window_length]

for i in range(0, n_testbatches):
    x_test[i] = data_days_test[i:(i + window_length)]
    y_test[i] = data_days_test[i + window_length]
model = keras.Sequential()
model.add(
    layers.LSTM(len(transactions), input_shape=(x_train.shape[1], x_train.shape[2])) 
)  
model.add(layers.Dropout(0.01)) 
model.add(layers.Dense(y_train.shape[1], activation="sigmoid")) 
opt = keras.optimizers.Adam(learning_rate=0.01)
model.compile(loss="binary_crossentropy", optimizer=opt)
model.summary()
model.fit(x_train, y_train, epochs=1000, verbose = 0)
x_train.shape
y_train.shape
--------------------------------------------------------------------------
(561, 30, 1)
(561, 1)
$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.