1
$\begingroup$

I am a student and I am studying machine learning. I am focusing on probabilistic generative models for classification and I am having some troubles understanding this topic.

In the slide of my professor it is written the following:

enter image description here

which I don't understand.

So far, I have understood that in the generative probailistic models, we ant to estimate $P(C_i|x)$, which is the probability of having class $i$ given a data $x$, using the likelihood and the Bayes theorem.

So, it starts by writing the Bayes rule, but the the slides says that we can write this as a sigmoid, but why?

If I have to try to give an answer to it, I would say because the sigmoid gives a number from $0$ to $1$, and so a probability, but it is just a guess I am doing.

Moreover, it continues by saying that we can use a gaussian distribution for $P(x|C_i)$, and so $P(x|C_i)=N(\mu ,\sigma )$, and so :

enter image description here

I don't understand what it is doing, can somebody please help me?

I don't know if my question is clear so sorry if it is not but I am really confused. If it is not lcear please tell me I will try to edit it. Thanks in advance.

Note: if it can be useful, this has been taken from the Bishop book at page 197

$\endgroup$
2
+50
$\begingroup$

My understanding is:

  1. The first line is OK - it derives from Bayes Rule

  2. Assume that this probability follows a logistic function, that is that

$$P(C_1|x) = \frac{1}{1+exp(-a)}$$

  1. Then if

$$ y = \frac{1}{1+exp(-z)}$$

then

$$ z = ln(\frac{y}{1-y})$$

(some lines below but with $a$ and $\sigma$)

  1. Therefore:

$$ a = ln(\frac{P(C_1|x)}{1-P(C_1|x)})$$

  1. If there are only two classes then $1-P(C_1|x)= P(C_2|x)$.

  2. Then

$$a = ln(\frac{P(C_1|x)}{P(C_2|x)}$$

  1. Using Bayes on $P(C_1|x)$ and $P(C_2|x)$

$$a = ln(\frac{P(x|C_1) P(C_1)}{P(x|C_2)P(C_2)}$$

  1. The whole thing now is to model $P(x|C_i)$. He assumes that it is a normal distribution. And from there you should the second set of equations. The important part (as far as I remember) is that the two covariance matrix (for $C_1$ and for $C_2$) are the same. I do not have access to the book now but I will look it up tomorrow and see if there is any difficult step in the derivations.

tldr: The whole point is he assumes that the probability as a logistic, then he gets the formula for the $a$, then he assumes a normal distribution for $P(x|C_i)$, then he assumes same covariance matrix.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.