I'm very familiar with neural networks for classification, but I'm trying a regression task for the first time. I'm finding that the network tends to go towards guessing a mean for the whole dataset rather than making case specific predictions.
What are the potential reasons for this behaviour and how can I stop it?
In my specific case this behaviour is present in both the training and validation set. The training set is very small, and it's questionable as to whether the inputs have a strong correlation with the target.
Also might be worth noting that I'm training on images. So I'm using a CNN.
I did some inspection of my model and found that it drives the output of my CNN backbone to 0 and simply uses the single bias on my fully connected layer to produce the output. I also found that if I freeze the bias during training, I can increase the learning rate indefinitely without the loss blowing up...