I'm a noob in the ML world and am currently building an LSTM to forecast the next page a user is going to visit on a website. My dataset is pretty much a mapping (with sliding window) from one page to the next page (encoded as integers) looking like this:

[1, 2, 3, 4] -> [5]
[2, 3, 4, 5] -> [6]
[3, 4, 5, 6] -> [7]

To not have to do padding and throw away very long sessions of different users, I just concatenated the sessions together and cut this huge array into many small lists of a specific sequence length. After that, my data gets batched and fed into the model.

The Model is an LSTM (stateful) with an Embedding (because I have over 30000 individual pages), sometimes Dropout and a Dense Layer. I use sparse_categorical_crossentropy as my loss function and the Adam-Optimizer. Here is a summary of one of my tries:

Layer (type)                 Output Shape              Param #
embedding (Embedding)        (128, None, 300)          9301200
cu_dnnlstm (CuDNNLSTM)       (128, 2048)               19251200
dense (Dense)                (128, 31004)              63527196
Total params: 92,079,596
Trainable params: 92,079,596
Non-trainable params: 0

As of now, I ran over a hundred tests with different sizes of my dataset, different sizes of the LSTM (32 - 4096) and with 1 or 2 LSTM layers, different batch sizes (8 - 128), etc. and I am at my wits end, because I can't get the validation loss to go down (and the validation accuracy to go up) especially with bigger datasets. The validation accuracy tends mostly to rise, while the training loss goes down very fast (at least when using only one layer of LSTMs). It looks as follows in the most cases and validation accuracy never rises above about 0.35 in whatever configuration I try:

Training loss sort of ok, but val loss not improving

If I'm not mistaken, then this is overfitting and as I've read, adding dropout or reducing the number of rnn-units will counteract this. Once I add dropout, it mainly changes how good my training loss is getting, while almost not affecting validation loss. Same with the reduction of rnn units. The fewer I use, the worse my training loss gets and the validation loss does not move very much.

Also I discovered, when I'm using a two layered LSTM, my Training Loss assumes a very weird, sort of oscillating pattern which approaches a certain value. If I reduce the number of units in both layers, the oscillating pattern vanishes, but at the cost of overall performance.

Oscillating training loss and accuracy

In essence, it seems like whatever I do, the validation loss won't change in my favor and I can't help but feel like I am missing something very obvious. If you have any idea what I'm doing wrong, please let me know.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.