I have a simple network with 1st level an LSTM, dropout, fully-connected and softmax layers; loss is cross-entropy (four classes, well balanced). Sequence length to LSTM is 172 samples, data is z-score normalized, LSTM outputs at the end of a sequence.
Sequences are EEG recordings from 17 sensors (channels). Training set has 2146 sequences, validation set has 271 sequences. Minibatch size is 48.
Depending on LSTM layer sizing (number of cells), the netowrk either overfits or underfits, but never generalizes. When it overfits, after initial warm-up, as the training loss starts to decrease, validation loss starts to go up.
I'm using drouput with 0.5 probability, L2 regularization with 0.0005 weight and Adam solver.
What could be the reason and how to solve this?