I use EfficientNetB0 for performing image classification with one of the class as "stone countertop table", the other class is "not stone countertop table". I use the reddest part in gradCAM to put a point saying that this is the "stone countertop table". But in some cases, it is pointing to the basin (as most of the stone countertop accompany with a basin, and lead efficientNETB0 focus on the wrong thing). Is there a way to tell EfficientNetB0 not to use basin as a hint to look for "stone countertop table", and therefore the reddest part in gradCAM will always point to the surface of "stone countertop table"?
This is what is desired:
This is not:
This is not as well: