Face matching using VGGFace library keras

Question

I am working on face matching model (matching between id-card faces and selfies), where I am using the resnet50 pre-trained model from VGGFace library, and then retraining all the layers. The training set accuracy is around 97% the validation set accuracy is around 90%. However when I test the model on completely (same image type i.e. a selfie and id-card face) unseen data it fails and the accuracy there is around 65-70%. I am unable to understand why this huge difference between validation set and test set accuracy. My model code is as follows:

epoch=50
learning_rate=0.0001 #earlier 0.0001,0.001
decay=learning_rate/epoch
batch_size=16 #32,16 earlier
keep_prob=0.8

def get_base_model(input_shape=(224,224,3):
   base_model=VGGFace(model='resnet50',include_top=False,input_shape=input_shape,pooling=None)
   output=base_model.get_layer('add_12').output
   activation=LeakyReLU(alpha=0.3)(output)
   flat=GlobalAvgPool2D()(activation)
   adam = optimizers.Adam(lr=learning_rate,decay=decay)
   final_model=Model(inputs=base_model.input,outputs=flat)

   for layers in final_model.layers:
      layers.trainable=True
   
   final_model.compile(loss=binary_crossentropy,optimizer=adam,metrics=['accuracy',keras_metrics.precision(), keras_metrics.recall()])  


   return final_model


base_model=get_base_model()
selfie_input=Input(shape=input_shape)
pan_input=Input(shape=input_shape)

encoded_selfie=base_model(selfie_input)
encoded_pan=base_model(pan_input)

adam = optimizers.Adam(lr=learning_rate,decay=decay)



L1_layer=Lambda(lambda tensors : K.square(tensors[0]-tensors[1]))
L1_distance=L1_layer([encoded_selfie,encoded_pan])
    
drop=Dropout(1-keep_prob)(L1_distance)
output=Dense(output_shape,activation='sigmoid',kernel_initializer='glorot_uniform')(drop)
siamese_net=Model(input=[selfie_input,pan_input],outputs=output)
siamese_net.compile(loss=binary_crossentropy, optimizer=adam,metrics=['accuracy',keras_metrics.precision(), keras_metrics.recall()])


siamese_net.fit_generator(generator=train_generator,validation_data=valid_generator,epochs=epoch,callbacks=callbacks,class_weight={0:1.2,1:1.5})

My training size is around 110K unique faces My validation size is around 7K unique faces My test set size is around 7K unique faces

The issue that I am facing is when I save the model using save_model() and then I predict (by loading the model and then sequentially predicting it...in a different script) the values on completely different faces different from those in train, test, validation set it predicts poorly and the accuracy drops to around 65%.

I do not know why such a huge gap. I think it has something to do with the BatchNormalization (different performance during training and inference) layer as mentioned here but I am not sure. Please help me out.

PS : I train my models on google colab GPU and run the evaluation script on an AWS ec-2. Keras version 2.3.0 and tensorflow version 1.15 (gpu)