# Neural Networks: Predicting probabilities of the possible values of y, instead of just predicting y

I have a true value y that I'd like to predict with a regression, but I'm interested in the probabilities that y will be different values. Y is theoretically continuous but in the dataset it is rounded to integers. Let's say y could be 0-9. I want 10 probabilities, one for each possible value. I tried doing this categorically, with a neural network having 10 output nodes, this hurts the predictions since we lose the relationships between categories, 1 is closer to 2 than it is to 10.

Example toy problem:

y is the weight of an object in pounds. The dataset has Y values rounded s.t. y can be from 0 to 9 pounds. Predict the probabilities that y will be 0-9 pounds based on the features X.

Example output: [0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1,] (ten # summing to 1)

I'd like to be able to accomplish this with Keras.

Here are two approaches that use the idea of havng your prediction be a distribution over your continuous variables rather than a single value.

## "Easy" but computationally expensive

Independently train K regressors. You now have K samples from a distribution, and you can

Big downside - you have to train lots of networks, and run lots of networks at inference time.

## Harder

Have one network explicitly predict a parametrized (e.g. Gaussian/ mixture of gaussians) distribution. For details see "Mixture density network", for example

So in the case of a single Gaussian, the network would output a mean and standard deviation, and so you have an estimate of the uncertainty.

A possible danger here is that you might need to be clever about how to train the variance parameter. You could imagine a case where a network does a good job on the means but is either over- or under- confident in its predictions.

• I tried gradient boosting, which the "Easy" method basically is doing but with neural networks instead of trees, at least that's the extent of my knowledge on those. – Kuwabara Sep 28 '19 at 1:06
• How did that go? Were the results"ok" – bogovicj Sep 28 '19 at 1:34
• The results were not that great but I was doing it as an ensemble of binary classifications because of a detail that I left out of my question, which I realize now matters more than I though when asking this question with a simplified example. In my actual problem I am predicting what/when an event will happen in the future. The crux is that there are two events, A and B. I ultimately want to predict when one of the two will happen (but the distribution, not just a value). A and B are exclusive of each other, only one can occur. – Kuwabara Sep 28 '19 at 2:40
• I see. I was really thinking "just" regression with uncertainty when writing the answer. Maybe edit your question? – bogovicj Sep 28 '19 at 12:57
• That question is so different I should probably just close this one and pose a new question! – Kuwabara Sep 28 '19 at 17:12