I'm a noob in the ML world and am currently building an LSTM to forecast the next page a user is going to visit on a website. My dataset is pretty much a mapping (with sliding window) from one page to the next page (encoded as integers) looking like this:
[1, 2, 3, 4] ->  [2, 3, 4, 5] ->  [3, 4, 5, 6] ->  ...
To not have to do padding and throw away very long sessions of different users, I just concatenated the sessions together and cut this huge array into many small lists of a specific sequence length. After that, my data gets batched and fed into the model.
The Model is an LSTM (stateful) with an Embedding (because I have over 30000 individual pages), sometimes Dropout and a Dense Layer. I use sparse_categorical_crossentropy as my loss function and the Adam-Optimizer. Here is a summary of one of my tries:
_________________________________________________________________ Layer (type) Output Shape Param # ================================================================= embedding (Embedding) (128, None, 300) 9301200 _________________________________________________________________ cu_dnnlstm (CuDNNLSTM) (128, 2048) 19251200 _________________________________________________________________ dense (Dense) (128, 31004) 63527196 ================================================================= Total params: 92,079,596 Trainable params: 92,079,596 Non-trainable params: 0 _________________________________________________________________
As of now, I ran over a hundred tests with different sizes of my dataset, different sizes of the LSTM (32 - 4096) and with 1 or 2 LSTM layers, different batch sizes (8 - 128), etc. and I am at my wits end, because I can't get the validation loss to go down (and the validation accuracy to go up) especially with bigger datasets. The validation accuracy tends mostly to rise, while the training loss goes down very fast (at least when using only one layer of LSTMs). It looks as follows in the most cases and validation accuracy never rises above about 0.35 in whatever configuration I try:
If I'm not mistaken, then this is overfitting and as I've read, adding dropout or reducing the number of rnn-units will counteract this. Once I add dropout, it mainly changes how good my training loss is getting, while almost not affecting validation loss. Same with the reduction of rnn units. The fewer I use, the worse my training loss gets and the validation loss does not move very much.
Also I discovered, when I'm using a two layered LSTM, my Training Loss assumes a very weird, sort of oscillating pattern which approaches a certain value. If I reduce the number of units in both layers, the oscillating pattern vanishes, but at the cost of overall performance.
In essence, it seems like whatever I do, the validation loss won't change in my favor and I can't help but feel like I am missing something very obvious. If you have any idea what I'm doing wrong, please let me know.