7
$\begingroup$

I know that for categorical features we just calculate the prior and likelihood probability assuming conditional independence between the features.

How does it work for continuous variables? How can we calculate likelihood probability for continuous variable?

$\endgroup$
9
$\begingroup$

The difference boils down to "how we define $P(x_i|C_k)$?", where $x_i$ is a single feature, and $C_k$ is a class from a total of $K$ classes.

Discrete

In discrete case, $P(x_i|C_k)$ is represented by a table as follows:

x_i   P(x_i|C_k)
a     0.5
b     0.2
c     0.3

We have one of these tables for each feature-class pair $(i, k)$.

Lets denote $i$-th feature of data point $n$ as $x_{n,i}$. Each row of this table can be estimated using

$$\hat{P}(x_i=a|C_k) = \frac{\sum_{n:n \in C_k}{\mathbb{1}_{x_{n,i}=a}}}{N_k}$$

which divides the number of samples that have i-th feature equal to $a$ by total number of samples in class $C_k$ (of course from the training set).

Also, check out pseudocount that avoids zero estimations.

Continuous

In continuous case, we either discretize the continuous interval of $x_i$ into bins $\{b_1,..,b_m\}$ and proceed the same as discrete case, or we assume a function like Gaussian (or any other one), as follows:

$$P(x_i|C_k)=\frac{1}{\sqrt{2\pi}\sigma_{i,k}}e^{-(x_i-\mu_{i,k})^2/2\sigma_{i,k}^2}$$

This way, for each feature-class pair $(i, k)$, $P(x_i|C_k)$ is represented with two parameters $\{\mu_{i,k}, \sigma_{i, k}\}$ instead of a table in discrete case. The estimation of the parameters is the same as fitting a Gaussian distribution to one dimensional data, that is:

$$\hat{\mu}_{i,k} = \frac{\sum_{n:n \in C_k}{x_{n,i}}}{N_k}, \hat{\sigma}^2_{i, k} = \frac{\sum_{n:n \in C_k}{(x_{n,i} - \hat{\mu}_{i,k})^2}}{N_k-1}$$

Instead of Gaussian, we can opt for a more complex function, even a neural network. In that case, we should look for a technique to fit the function to data just like what we did with Gaussian.

The rest is independent of feature type

Representing $P(C_k)$ is the same for both discrete and continuous cases and is estimated as $\hat{P}(C_k)=N_k/N$.

Finally, the classifier is $$C(x_n) = \underset{k \in \{1,..,K\}}{\mbox{argmax }}\hat{P}(C_k)\prod_{i}\hat{P}(x_i=x_{n,i}|C_k)$$

Or equivalently using log-probabilities,

$$C(x_n) = \underset{k \in \{1,..,K\}}{\mbox{argmax }}\mbox{log}\hat{P}(C_k)+\sum_{i}\mbox{log}\hat{P}(x_i=x_{n,i}|C_k)$$

$\endgroup$
2
  • 1
    $\begingroup$ Isn't $\hat{P}(x_i = x_{n,i}| C_k) = 0$ when we are talking about continuous probability distributions? That is, the probability that a continuous random variable takes an exact value is 0, you can only get non-zero probabilities for it being in some interval $[a,b]$ and compute that as the integral of the density function from $a$ to $b$. How does one actually compute $C(x_n)$ when the density function is continuous? Are you just plugging $x_{n,i}$ into $\hat{P}$? $\endgroup$ – kilgoretrout Aug 24 '20 at 17:39
  • 1
    $\begingroup$ @kilgoretrout You are right, I have used the notations loosely here. For continuous random variables, $\hat{P}$ denotes probability density function (p.d.f.) which could be even larger than 1. For the second question, yes just plug, $\hat{P}$ is a parameterized p.d.f. like Gaussian that receives a one-dimensional value like $x_{n, i}$. $\endgroup$ – Esmailian Aug 25 '20 at 7:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.