When using padding in sequence models, is Keras validation accuracy valid/ reliable?

Question

I have a group of non zero sequences with different lengths and I am using Keras LSTM to model these sequences. I use Keras Tokenizer to tokenize (tokens start from 1). In order to make sequences have the same lengths, I use padding.

An example of padding:

# [0,0,0,0,0,10,3]
# [0,0,0,0,10,3,4]
# [0,0,0,10,3,4,5]
# [10,3,4,5,6,9,8]

In order to evaluate if the model is able to generalize, I use a validation set with 70/30 ratio. In the end of each epoch Keras shows the training and validation accuracy.

My big doubt is whether Keras validation accuracy is reliable when using padding (When you run Keras over several epochs, in the end of each epochs it prints training accuracy and validation accuracy). Because the validation set can simply be sequences of 0's --> [0,0,0]. Since there are a lot of sequences of 0's (because of padding), the model can easily learn and predict the sequences of 0's correctly, and as a result, create a fake high validation accuracy. In other words the model may learn sequences of zeros and not learn the real sequences.

So, does padding influences the validation accuracy in Keras?

Answer 1

I know this answer is too late but I think it can be useful for other readers.

The short answer is YES! The padding influences the accuracy.

For handling the bad effect of padding, you can define new metrics. This new metric must ignore the class related to padding.

This article presents a BiLSTM model for POS tagging as a sequence tagging task. A special accuracy metric that ignores the padding class is provided below:

from keras import backend as K
 
def ignore_class_accuracy(to_ignore=0):
    def ignore_accuracy(y_true, y_pred):
        y_true_class = K.argmax(y_true, axis=-1)
        y_pred_class = K.argmax(y_pred, axis=-1)
 
        ignore_mask = K.cast(K.not_equal(y_pred_class, to_ignore), 'int32')
        matches = K.cast(K.equal(y_true_class, y_pred_class), 'int32') * ignore_mask
        accuracy = K.sum(matches) / K.maximum(K.sum(ignore_mask), 1)
        return accuracy
    return ignore_accuracy

Note that in this case one-hot label is used. Finally you can pass your new accuracy like this:

model.compile(loss='categorical_crossentropy',
              optimizer=Adam(0.001),
              metrics=['accuracy', ignore_class_accuracy(0)])

In training model, an output like this will reported (91% for normal accuracy and 81% for new special accuracy):

Epoch 1/10 1679/2054 [=======================>......] - ETA: 2:33 -
loss: 0.2901 - accuracy: 0.9147 - ignore_accuracy: 0.8118