I am working on an object detection problem where the final loss that is being optimized is the sum of an L2 loss (for the error in the predicted w, h values), and three binary cross entropy losses (for the predicted class error, the object/no-object confidence error and the x,y coordinate error).
I see that during the training phase, the x,y and w,h losses are overfitting (validation losses go up while the training losses go down). The other losses all do seem to be optimized normally.
How do I solve this, and why am I seeing this? The overall loss seems to be going down, as the trend is influenced by the confidence loss which is substantially higher than the rest of the losses.
For a bit more context, I'm using a YOLOv3-based object detection setup.
Edit: Attaching a picture showing the xy training and validation losses overfitting. The wh losses behave in a similar fashion.
Edit 2: The setup is to detect objects on a custom dataset of ~9000 train images and ~1000 validation images. I believe the dataset and its distribution is proper because it was validated using other YOLO-based object detection models with a couple of different backend networks, all of which gave me high accuracies. This particular problem began when I switched to a different backend network with a similar number of parameters and layers as one of the previous ones that were tested and deemed proper.