3
$\begingroup$

I'm very familiar with neural networks for classification, but I'm trying a regression task for the first time. I'm finding that the network tends to go towards guessing a mean for the whole dataset rather than making case specific predictions.

What are the potential reasons for this behaviour and how can I stop it?

Hint

In my specific case this behaviour is present in both the training and validation set. The training set is very small, and it's questionable as to whether the inputs have a strong correlation with the target.

Also might be worth noting that I'm training on images. So I'm using a CNN.

Update

I did some inspection of my model and found that it drives the output of my CNN backbone to 0 and simply uses the single bias on my fully connected layer to produce the output. I also found that if I freeze the bias during training, I can increase the learning rate indefinitely without the loss blowing up...

Improve this question
$\endgroup$
3
  • $\begingroup$ May be there is no causility as expected. So if its not possible to conclude the output, given the input, the best a model can do is to output the mean. $\endgroup$
    – Graph4Me Consultant
    Commented Sep 9, 2020 at 20:03
  • 1
    $\begingroup$ @Graph4Me I’d still expect thousands, maybe millions, or parameters in a CNN to play connect-the-dots and overfit like crazy, even if the predictors and outputs are unrelated. $\endgroup$
    – Dave
    Commented Sep 10, 2020 at 3:05
  • $\begingroup$ @Dave good point. I only have 45k params, but only 170 training samples (stupidly small I know - they are 3D CAT scans of lungs). I would have expected that for the training set at least I'd get the NN to look at a small handful of pixels and perfectly overfit $\endgroup$
    – Alexander Soare
    Commented Sep 10, 2020 at 9:50

1 Answer 1

Reset to default
1
$\begingroup$

The regression tasks are not very different from classification and the behavior you faced with is probably due to a bug in the code. If your training set is small and the network is comparably large, it should overfit on it (correlation with inputs doesn't matter, there was a paper https://arxiv.org/pdf/1611.03530.pdf that shows you can randomly shuffle imagenet labels and train one of the version of resnet on it and get perfect accuracy on that training set). Sample agnostic predictions mean underfitting. You can also try to overfit a small batch and get a zero loss. If you won't be unable to do this, there's a bug, definitely. Possibly your gradients affect the latter layers much more than the first layers, which means after some epochs of training you still have random outputs from the first layer and it's no wonder the net is trying to predict the mean of the dataset. This can happen if your net is very deep and you're not using residual connections. But it's not the case for a net of 10 (for instance) or less layers. Maybe your learning rate is too high and your first layers can't be tuned correctly. To check whether your first layers extract some meaningful information, try to look at the outputs of some of your first layers. If those outputs look like a complete noise, double check the code for optimization. You can find more interesting cases of bugs and the behavior of a buggy code in Andrej Karpathy's blog http://karpathy.github.io/2019/04/25/recipe/

Improve this answer
$\endgroup$
7
  • 1
    $\begingroup$ Great idea to lower the learning rate for the FC layer. I'd suspect it's doing away completely with the convolutional weights and then putting all the "learning" into the FC layer. Yet to test it out though. Indeed though, my net only has 4 conv layers before the FC. $\endgroup$
    – Alexander Soare
    Commented Sep 10, 2020 at 9:54
  • 1
    $\begingroup$ I went ahead and tried some of the ideas you shared. If you are still on this please see the update in OP. It feels like the network is not able to find any relationship at all between my input data and the target. $\endgroup$
    – Alexander Soare
    Commented Sep 10, 2020 at 16:30
  • 1
    $\begingroup$ Your observations confirm the hypothesis that your net doesn't extract any information from input images. Actually if outputs of your backbone are just random noises, it's easier for the net to just cancel them all out. I assume you have different outputs from your backbone for a single object each time you feed it in the net. This may happen due to a high learning rate (we as discussed earlier) or some bugs in image preprocessing. Give me some additional details please: nn architecture, learning rate value, size of the training set. $\endgroup$
    – Michał Słodki
    Commented Sep 10, 2020 at 17:41
  • $\begingroup$ I suggest you to try to overfit a single image (let it be your training set). If it works, try to gradually increase the number of images. Remember, neural nets are extremely powerful. They can learn an arbitrary continuous function (given enough capacity) and even some discontinuous functions. It means during training phase (only) they should be able to find relationships even between different samples from gaussian distribution and their corresponding arbitrary (maybe probabilistically independent) target values (that's exactly what Generative Adversarial Networks do btw) $\endgroup$
    – Michał Słodki
    Commented Sep 10, 2020 at 17:51
  • $\begingroup$ Some details: LR around 1e-3 with Adam optimizer. About 170 training samples 40x512x512 3D image. Architecture is currently much like VGG (kernel size 3, double channels every time you downsample), except I'm using strided convolutions for downsampling instead of pooling, and I do a global avg pool to feed to FC. I actually tried overfitting on a single batch of 4 and managed to get an improvement beyond global avg (with 3M model params). But using the whole training set goes back to regressing to the mean. May need to step up the number of params. $\endgroup$
    – Alexander Soare
    Commented Sep 10, 2020 at 19:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.