15
$\begingroup$

I have a convolutional + LSTM model in Keras, similar to this (ref 1), that I am using for a Kaggle contest. Architecture is shown below. I have trained it on my labeled set of 11000 samples (two classes, initial prevalence is ~9:1, so I upsampled the 1's to about a 1/1 ratio) for 50 epochs with 20% validation split.I was getting blatant overfitting for a while but I thought it got it under control with noise and dropout layers.

Model looked like it was training wonderfully, at the end scored 91% on the entirety of the training set, but upon testing on the test data set, absolute garbage.

Loss by epoch

Validation accuracy by epoch

Notice: the validation accuracy is higher than the training accuracy. This is the opposite of "typical" overfitting.

My intuition is, given the small-ish validation split, the model is still managing to fit too strongly to the input set and losing generalization. The other clue is that val_acc is greater than acc, that seems fishy. Is that the most likely scenario here?

If this is overfitting, would increasing the validation split mitigate this at all, or am I going to run into the same issue, since on average, each sample will see half the total epochs still?

The model:

Layer (type)                     Output Shape          Param #     Connected to                     
====================================================================================================
convolution1d_19 (Convolution1D) (None, None, 64)      8256        convolution1d_input_16[0][0]     
____________________________________________________________________________________________________
maxpooling1d_18 (MaxPooling1D)   (None, None, 64)      0           convolution1d_19[0][0]           
____________________________________________________________________________________________________
batchnormalization_8 (BatchNormal(None, None, 64)      128         maxpooling1d_18[0][0]            
____________________________________________________________________________________________________
gaussiannoise_5 (GaussianNoise)  (None, None, 64)      0           batchnormalization_8[0][0]       
____________________________________________________________________________________________________
lstm_16 (LSTM)                   (None, 64)            33024       gaussiannoise_5[0][0]            
____________________________________________________________________________________________________
dropout_9 (Dropout)              (None, 64)            0           lstm_16[0][0]                    
____________________________________________________________________________________________________
batchnormalization_9 (BatchNormal(None, 64)            128         dropout_9[0][0]                  
____________________________________________________________________________________________________
dense_23 (Dense)                 (None, 64)            4160        batchnormalization_9[0][0]       
____________________________________________________________________________________________________
dropout_10 (Dropout)             (None, 64)            0           dense_23[0][0]                   
____________________________________________________________________________________________________
dense_24 (Dense)                 (None, 2)             130         dropout_10[0][0]                 
====================================================================================================
Total params: 45826

Here is the call to fit the model (class weight is typically around 1:1 since I upsampled the input): 

class_weight= {0:1./(1-ones_rate), 1:1./ones_rate} # automatically balance based on class occurence
m2.fit(X_train, y_train, nb_epoch=50, batch_size=64, shuffle=True, class_weight=class_weight, validation_split=0.2 )

SE has some silly rule that I can post no more than 2 links until my score is higher, so here is the example in case you are interested: Ref 1: machinelearningmastery DOT com SLASH sequence-classification-lstm-recurrent-neural-networks-python-keras

Improve this question
$\endgroup$

2 Answers 2

Reset to default
10
$\begingroup$

I am not sure if the validation set is balanced or not. You have a severe data imbalance problem. If you sample equally and randomly from each class to train your network, and then a percentage of what you sampled is used to validate your network , this means that you train and validate using balanced data set. In the testing you used imbalanced database. This means that your validation and testing sets are not equivalent. In such case you may have high validation accuracy and low testing accuracy. Please find this reference that talks mainly about data imbalance problem for DNN, you can check how they sample to do the training, validation and testing

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ The validation set is pulled from the same set as the training set. I'm not 100% sure but I believe Keras takes the validation cut before shuffling and training (i.e. if you feed it unshuffled data, the validation split chops off part of your distribution). So I manually balance and shuffle before passing to Keras. Only issue might be that I am duplicating the 1's and adding gaussian noise, which may be enhancing overfit. $\endgroup$
    – DeusXMachina
    Nov 21, 2016 at 13:37
  • 2
    $\begingroup$ I think you do not need to depend on Keras to do the validation , you can divide your data into three parts. Training , validation and testing. Sample from the training data and train your network. Do not play with the distribution in the validation and testing sets. Optimize your network on the validation set and then check on the testing set. I edited my answer to add a useful reference $\endgroup$
    – Bashar Haddad
    Nov 22, 2016 at 14:10
5
$\begingroup$

If your training loss goes under you validation loss, you are overfitting, even if validation is still dropping.

It is the sign that your network is learning patterns in the train set that are not applicable in the validation one

Improve this answer
$\endgroup$
2
  • $\begingroup$ I understand the training loss going less than validation loss - this is just overfitting to the training set. But the validation accuracy is higher than the training accuracy. That is the weird part! $\endgroup$
    – DeusXMachina
    Nov 22, 2016 at 22:18
  • $\begingroup$ Not if your sets are imbalanced. For example, in a binary classifier, if you have less 1 in proportion in the validation set, and your model only outputs 0s, you would have a higher validation accuracy $\endgroup$
    – mxdbld
    Nov 23, 2016 at 9:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.