# Face matching using VGGFace library keras

I am working on face matching model (matching between id-card faces and selfies), where I am using the resnet50 pre-trained model from VGGFace library, and then retraining all the layers. The training set accuracy is around 97% the validation set accuracy is around 90%. However when I test the model on completely (same image type i.e. a selfie and id-card face) unseen data it fails and the accuracy there is around 65-70%. I am unable to understand why this huge difference between validation set and test set accuracy. My model code is as follows:

epoch=50
learning_rate=0.0001 #earlier 0.0001,0.001
decay=learning_rate/epoch
batch_size=16 #32,16 earlier
keep_prob=0.8

def get_base_model(input_shape=(224,224,3):
base_model=VGGFace(model='resnet50',include_top=False,input_shape=input_shape,pooling=None)
activation=LeakyReLU(alpha=0.3)(output)
flat=GlobalAvgPool2D()(activation)
final_model=Model(inputs=base_model.input,outputs=flat)

for layers in final_model.layers:
layers.trainable=True

return final_model

base_model=get_base_model()
selfie_input=Input(shape=input_shape)
pan_input=Input(shape=input_shape)

encoded_selfie=base_model(selfie_input)
encoded_pan=base_model(pan_input)

L1_layer=Lambda(lambda tensors : K.square(tensors[0]-tensors[1]))
L1_distance=L1_layer([encoded_selfie,encoded_pan])

drop=Dropout(1-keep_prob)(L1_distance)
output=Dense(output_shape,activation='sigmoid',kernel_initializer='glorot_uniform')(drop)
siamese_net=Model(input=[selfie_input,pan_input],outputs=output)

The issue that I am facing is when I save the model using save_model() and then I predict (by loading the model and then sequentially predicting it...in a different script) the values on completely different faces different from those in train, test, validation set it predicts poorly and the accuracy drops to around 65%.
I do not know why such a huge gap. I think it has something to do with the BatchNormalization (different performance during training and inference) layer as mentioned here but I am not sure. Please help me out.