-1
$\begingroup$

If I have a very large set of data (~ 1TB). How can I use Neural Network on this data to understand which underlying distribution (eg. let's say a Gaussian or a Poissonian with a certain mean, sd) is favoured by the data. At least, can I conclude, if one of the distributions (out of the 2 distributions) is not favoured?

hence the main goal was to see, if when forecasting, after learning from this dataset, can I have the knowledge that based on which underlying distribution (or favouring which distribution) the algorithm is predicting.

$\endgroup$
1
  • $\begingroup$ can someone explain why is it downvoted ? I think I have a valid question posted here $\endgroup$ – Ayan Mitra Apr 25 at 7:36
1
$\begingroup$

To be clear, do you think the data could be distributed according to a particular probability distribution? If so you should model it directly without a neural network. Your model parameters are simply the parameters of that probability distribution ($\mu$ and $\sigma^2$ for a gaussian, $\lambda$ for a poisson etc.,).

If you think your data is distributed according to a distribution with parameters conditioned on some input features you should have a model (for example a linear model, neural network etc.,) output the parameters of that probability distribution.

In either of these cases, you could then take a fully Bayesian approach (See, for example, chapter 5.3 of Machine Learning A probabilistic Perspective K. Murphy), but a simpler approach would be to perform Maximum Likelihood Estimation. The calculations for Maximum Likelihood Estimation are straightforward and readily available for common distributions. For a Gaussian you simply calculate the sample mean and variance and those are your model parameters. If you are outputting the distribution parameters from a neural network you will need to set your loss as the negative log likelihood associated with that particular distribution. Perform Maximum Likelihood Estimation for all models you are considering, and then compare each of the model's likelihood under a held out validation set. The likelihood is calculated as the product of probabilities of each data sample under that model. The distribution that fits the data the best is the one which the has highest likelihood on the held-out validation set.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.