I am trying to train a Siamese network for an application very similar to this and this. From what I have read about training Siamese networks dissimilar pairs of images outnumber the similar pairs and obviously so. In the papers I have linked to, the authors talk about a 1:20 ratio for similar to dissimilar pairs, i.e., for every similar pair of images, the training set consists of 20 dissimilar pairs. If I have understood it correctly, for a batch size of 64, my batch will consist of 3 similar pairs and the corresponding 20 dissimilar pairs for each of the similar pair.
I implemented the same in Keras, and the network overfits from the very first epoch. I have approximately 6000 similar pairs of images and I am generating the dissimilar pairs randomly. How do I reduce the overfitting? Any tips would help.
NOTE: I am using the last two Inception modules of Inception V3 to train my data. EDIT: We managed to fix this issue with getting more training data, and using a network pretrained on a very similar task.