I have a convolution neural network for regression, where medical scans of many people are trained to predict some continuous variable (body related phenotype). I get reasonable performance (R2 ~ 0.9).
To see which parts of the images are important in the prediction, I obtain the heatmap for the final convolution layer (as obtained using gradient of the predicted value w.r.t. output of this layer), and superimpose it with the original scan when displaying. The issue is substantial differences are seen in terms of which parts get highlighted for different people, to the point that for some pairs of scans, there is almost no overlap in highlighted parts even when both scans have similar error in predicted value.
One would think roughly same portions of the scan should be important in predicting the output (esp. if predictions are close to the truth). So why might this be happening, and is something like this expected in general?