I am implementing a Gaussian Naive Bayes classifier (so each feature is continuous and assumed to be coming from a Gaussian distribution). When evaluating the probability of a feature value in the test set, if the value is sufficiently far away from the mean (e.g. the mean and s.d. on the training data is say 0 and 1 but the test value is 10^10) then there is underflow. This is an issue because then the probability will be calculated as 0.0 so the log probability is undefined. Is there a standard way of handling underflow in this case?


The standard answer is to work in log space, and manipulate the log of probabilities instead of probabilities, for exactly this reason. This classifier involves products of probabilities which just become sums of log probabilities.

You allude to that already, but the problem you suggest isn't a problem. Internally you don't calculate a probability and then take the log again. It stays in log space. So for very small P, log P is a large negative number while P itself may underflow to 0.0. But you may never need to evaluate P internally.

| improve this answer | |
  • $\begingroup$ You're right, of course - I was being stupid. In other words, don't ever calculate the probability using the Gaussian PDF, but instead take the log of the Gaussian PDF and then compute log probability. $\endgroup$ – chirpchirp Mar 26 '16 at 21:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.