So the question asks why you are seeing a decrease in the loss function (for both training and validation?), but you are also observing decreasing generalisation performance over iterations.
One first thought could be due to the loss function that you have chosen might not be appropriate for your task.
I have the same issue as you. I did not come up with a proper answer.
when I added more classes to the model, some of the previously detected class are gone.
And the total number of the identified objects does not increases, Even after I've put the model into training for a while.
this is my question: TF object detection - The total number of detected ...
The general consensus in machine learning problems is that it becomes tougher to get higher accuracy results when there is more data with more class splits. The simplest of examples would be cifar 10 and cifar 100. While they are practically the same models tend to vary very differently with respect to the efficiency.
The moment more classes are added, there ...
The process you have used it spot on. Your problem: Give more importance to one class than the rest will have to be hard-coded in. There will be no pre-trained networks for that class especially.
One thing to note could be that assuming you have to segment roads in a road scene problem. You can create a function which can bias the weights towards the road ...