I have a true value y that I'd like to predict with a regression, but I'm interested in the probabilities that y will be different values. Y is theoretically continuous but in the dataset it is rounded to integers. Let's say y could be 0-9. I want 10 probabilities, one for each possible value. I tried doing this categorically, with a neural network having 10 output nodes, this hurts the predictions since we lose the relationships between categories, 1 is closer to 2 than it is to 10.

Example toy problem:

y is the weight of an object in pounds. The dataset has Y values rounded s.t. y can be from 0 to 9 pounds. Predict the probabilities that y will be 0-9 pounds based on the features X.

Example output: [0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1,] (ten # summing to 1)

I'd like to be able to accomplish this with Keras.


Here are two approaches that use the idea of havng your prediction be a distribution over your continuous variables rather than a single value.

"Easy" but computationally expensive

Independently train K regressors. You now have K samples from a distribution, and you can

Big downside - you have to train lots of networks, and run lots of networks at inference time.

For some details "Confidence and prediction intervals for neural network ensembles"


Have one network explicitly predict a parametrized (e.g. Gaussian/ mixture of gaussians) distribution. For details see "Mixture density network", for example

So in the case of a single Gaussian, the network would output a mean and standard deviation, and so you have an estimate of the uncertainty.

A possible danger here is that you might need to be clever about how to train the variance parameter. You could imagine a case where a network does a good job on the means but is either over- or under- confident in its predictions.

| improve this answer | |
  • $\begingroup$ I tried gradient boosting, which the "Easy" method basically is doing but with neural networks instead of trees, at least that's the extent of my knowledge on those. $\endgroup$ – Kuwabara Sep 28 '19 at 1:06
  • $\begingroup$ How did that go? Were the results"ok" $\endgroup$ – bogovicj Sep 28 '19 at 1:34
  • $\begingroup$ The results were not that great but I was doing it as an ensemble of binary classifications because of a detail that I left out of my question, which I realize now matters more than I though when asking this question with a simplified example. In my actual problem I am predicting what/when an event will happen in the future. The crux is that there are two events, A and B. I ultimately want to predict when one of the two will happen (but the distribution, not just a value). A and B are exclusive of each other, only one can occur. $\endgroup$ – Kuwabara Sep 28 '19 at 2:40
  • $\begingroup$ I see. I was really thinking "just" regression with uncertainty when writing the answer. Maybe edit your question? $\endgroup$ – bogovicj Sep 28 '19 at 12:57
  • 2
    $\begingroup$ That question is so different I should probably just close this one and pose a new question! $\endgroup$ – Kuwabara Sep 28 '19 at 17:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.