Analysis of probability distribution of each features and Machine Learning

Question

While I know that probability distributions are for hypothesis testing, confidence level constructions, etc. They definitely have many roles in statistical analysis.

However, it is not obvious to me now how probability distributions come in handy for machine learning problems? In ML algorithms, they are expected to automatically pick up distributions from dataset. I wonder if there are any places of probability distributions in better solving ML problem?

Shortly put, how could statistical techniques related to probability distributions can benefit the solving of ML problems? If yes, in what ways explicitly?

Answer 1

That's a good question.

We can take a look at the distribution of the output, the distribution of the model parameters, and the distribution of the input.

The probability distribution of the output plays a central role in classification problems, where we assume that the classes follow a categorical distribution. We ensure this by applying some sort of normalization, e.g., a softmax in neural networks. Then we minimize some information-theoretic measure that is based on the predicted output distribution and the true output distribution, such as cross-entropy (X-ent) loss or Kullback-Leibler (KL) loss. Otherwise, we would have to resort to simple classification losses, such as simply the MSE. But X-Ent and KL losses provide a smoother loss landscape and thus allow gradient descent to convergence more quickly. It is also central in reinforcement learning, where we assume that the continuous actions the agent takes are Gaussian distributed: our model learns the mean $\mu$ and the log-variance $\log\sigma$ of the action distribution. In each step, we then proceed according to our policy $p(a|s) = \mathcal{N}(\mu,\sigma)$. It is similar for categorical actions (softmax over the Q-values). This allows us to incorporate uncertainty and model exploration, which would not be possible if we took the action as a deterministic output.

We can also place a distribution over the parameters of a model, such as is done in stochastic neural networks. The weights $\omega$ are then given by a distribution $p(\omega|x)$. This allows us to incorporate uncertainty into the model.

When we are interested in the distribution of the input, we are dealing with a generative model. Here, the goal is to model the data generating process to create new data. Prominent approaches are Generative Adversarial Networks and Variational Auto-Encoders.

Answer 2

One of the direct applications probability distributions can have in ML is for evaluating the predictive power of your model. For instance if you are modeling a binary classifier you can use it for:

• comparing several prospect models
• comparing among thresholds for your selected model

About this second option, have a look at the following example from a use case I had sometime ago:

you can use the concept of probability distributions of the model predicted probabilities to check that, in this case, the first one detects the 0's and the 1's better that the others, and has a better confusion matrix and higher sensitivity.

Another metric which uses this probability distributions of the outputs is the Kolmogorov-Smirnov metric (paper), which gives you a measure for how separated the distributions of your positives and negatives are from each other:

where:

• red being left of the threshold are true negatives, and red right of the threshold false positives
• blue right of the threshold true positives, and blue on the left are false negatives

The point here is that, the less area of overlaping colors and the more separation between such distributions, means a better separation capability of your model between classes.