I have a dataset of images of damaged cars. Each image has an associated mask of overall damage and severity index of each type of damage. Unet successfully predicts the overall mask of damaged area of a car. However, the model yields poor results when trying to detect small features like scratches. This is understandable, because most of the "positive mask" of a scratch contains other types of damage. Hence, if the model correctly predicts masks of the scratched areas it will be severely punished for not labeling most other pixels of the overall damage area.
What is the right tool to classify images, if each image has an overall label, but doesnot have a reliable mask for the scratch?