The validation loss < training loss and validation accuracy < training accuracy

Question

I have a binary classification problem. I get the following results: a val_loss (far) lower than the train_loss, but the accuracy is also lower for the validation compared to the training set. How is that possible ?

Epoch 1/10
10708/10708 [=======] - loss: 0.6356 - acc: 0.8289 - val_loss: 0.4981 - val_acc: 0.7760
Epoch 2/10
10708/10708 [=======] - loss: 0.6243 - acc: 0.8248 - val_loss: 0.5075 - val_acc: 0.7609
Epoch 3/10
10708/10708 [=======] - loss: 0.6204 - acc: 0.8302 - val_loss: 0.5152 - val_acc: 0.7694
Epoch 4/10
10708/10708 [=======] - loss: 0.6215 - acc: 0.8307 - val_loss: 0.4981 - val_acc: 0.7824
Epoch 5/10
10708/10708 [=======] - loss: 0.6180 - acc: 0.8318 - val_loss: 0.4942 - val_acc: 0.7848

This is the Keras model I'm using:

model = Sequential()
model.add(LSTM(16, input_shape=(1, 1)))
model.add(Dense(2, activation='softmax'))

adam = Adam(lr=0.001)
model.compile(loss='categorical_crossentropy',
          optimizer=adam,
          metrics=['accuracy'])

Answer 1

How did you split your training and test sets? My guess is that the minority class is under-represented in your test set. Consider the case where the minority class has frequencies of 25% and 5% in the training and test sets, respectively. If my model predicts everything is the majority class, the recall for the minority class is 0% on both sets but we get 75% training accuracy and 95% validation accuracy. Now let's say we train the model a bit more and achieve 50% recall while still correctly labeling all of the majority class samples: now we're up to 87% training and 97% accuracy. Because the training and test sets exhibit significantly different class frequencies, the test set will basically always have a higher accuracy than the training set (up to the point where you overfit, at which point the validation accuracy could be be lower than training accuracy but not necessarily).

The solution here is to either use stratified sampling to make sure your classes have equal representation between your different datasets, or to monitor other evaluation metrics like precision and recall. Plotting the taining and validation accuracies could also be helpful, since it should be visually pretty obvious if your model starts to overfit, regardless of class imbalance between your training and test sets.

Answer 2

This is a common thing with neural networks and different batch sizes. The training loss is the average of losses for the minibatch. Naturally for the first few batches you'll have a higher loss and as it goes through the data the loss gets smaller. Mean while the loss for the validation set is calculated against the entire dataset.