How does one train a neural network model that does regression over real values, using a gaussian output layer? ie estimating the mean and std parameters of the prediction. Since during training there will only be one real valued target. Would one sample from the output gaussian distribution and then perform loss with the target? Or would there somehow be two targets, one for the mean or std.

The Deep AR forecasting model by Amazon does something like this for timeseries forecasting, where the FC layer at the output of the RNN predicts the mean and std parameters of a gaussian distribution. However, the target label would only contain one real valued number. So how would the loss be computed? Sample from the gaussian distribution and then use that? This kind of sampling is done in RL to encourage exploration but i’m not sure how it’s done in training regression models


1 Answer 1


You can optimize for the likelihood of the data under the predicted Gaussian distribution. In other words

  1. You have a predicted mean (u) and standard deviation (s) for each datapoint (x)
  2. Calculate the negative log likelihood of that datapoint under that distribution. Here is the pytorch implementation, but you can solve this analytically from the PDF of a Gaussian and calculate it with whatever automatic diff library you want.
  3. Treat this as your loss and optimize.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.