I'm working with a CNN on a regression task, using MSE as the loss function. Here's a plot of predictions vs targets for the training set.
Note: Legend is wrong. Blue = prediction vs target | Red = fit | Green = ground truth (x=x)
I know that the approximate mean for the target is 0.8 so I initialise the bias of the last fully connected layer to this. (In my case it's reasonable to assume this for unseen samples as well). That's why the predicted fit crosses the ground truth at (0.8, 0.8).
What's interesting is that the R^2 is pretty good, but the slope is off. If I wanted to calibrate my model without further training I could easily apply a linear transformation to change the slope (note that the validation data doesn't look too dissimilar).
I feel like the model should have learned this extra factor. What could cause this lack of "sensitivity" or "dynamism"?
For the signal processing people here, this reminds me a lot of when a PID regulation loop doesn't have a strong enough actuator to keep up with fast transients. Although it's a completely different concept, I feel like somehow the weights aren't large enough to produce the necessary gradient???