I have a simple network with 1st level an LSTM, dropout, fully-connected and softmax layers; loss is cross-entropy (four classes, well balanced). Sequence length to LSTM is 172 samples, data is z-score normalized, LSTM outputs at the end of a sequence.

Sequences are EEG recordings from 17 sensors (channels). Training set has 2146 sequences, validation set has 271 sequences. Minibatch size is 48.

Depending on LSTM layer sizing (number of cells), the netowrk either overfits or underfits, but never generalizes. When it overfits, after initial warm-up, as the training loss starts to decrease, validation loss starts to go up.

I'm using drouput with 0.5 probability, L2 regularization with 0.0005 weight and Adam solver.

What could be the reason and how to solve this?

  • $\begingroup$ Could you add more information regarding the size of the training set and validation set? and maybe about the type data (images? what size)? $\endgroup$ – Mark.F Jan 30 '19 at 12:22
  • $\begingroup$ @Mark.F Updated the question as requested $\endgroup$ – qwer1304 Jan 30 '19 at 12:34
  • 1
    $\begingroup$ Maybe your problem is with the choice of training and validation set. Try using different combinations of them: random division seldom is the best pick. You can also try using early stopping: stop training when validation loss goes up three, four, five ... times in a row. It will not prevent overfitting from occuring, but maybe network performance produces in this way will be good enough for you. $\endgroup$ – maksylon Jan 30 '19 at 15:24
  • $\begingroup$ @maksylon I tried different training/validation partitions with the same result. Early stop wouldn't work since this happens just after the network exits warm up phase where the error is high (random choice) and already at that point validation error begins to increase. $\endgroup$ – qwer1304 Jan 30 '19 at 15:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.