I read chapter 5.1.3 of Joshua Bengio's deeplearning book, which says: supervised learning involve observing examples of random vectors $\textbf{x}$ and associated value or vector $\textbf{y}$ and learning to predict $\textbf{y}$ from $\textbf{x}$ by estimating $p(\textbf{y} \vert \textbf{x})$.

What does $p(\textbf{y} \vert \textbf{x})$ mean?

From basic statistics, I know that $p(\textbf{y} \vert \textbf{x})=\frac{p(\textbf{y},\textbf{x})}{p(\textbf{x})}$. How do we find $p(\textbf{y},\textbf{x})$ and $p(\textbf{x})$?


3 Answers 3


You are correct that \begin{equation*} p(y|\mathbf{x})=\dfrac{p(\mathbf{x},y)}{p(\mathbf{x})}. \end{equation*} Similarly, we can write the joint probability $p(\mathbf{x},y)$ as follows: \begin{equation*} p(\mathbf{x},y)=p(\mathbf{x}|y)\cdot p(y) \end{equation*} From the above two equations, we obtain \begin{equation*} p(y|\mathbf{x})=\dfrac{p(\mathbf{x}|y)\cdot p(y)}{p(\mathbf{x})} \end{equation*} In the context of supervised learning, the variable $y$ is used to denote the class labels, and the vector $\mathbf{x}$ for measurement vector or feature vector. For the purpose of discussion, let us assume that the class label $y$ takes values in the set $\{1,2\}$, where $1$ denotes male and $2$ denotes female. Similarly, $\mathbf{x}$ is a measurement vector on two variables say, $(x_{1},x_{2})$, where $x_{1}$ stands for height and $x_{2}$ stands for weight of individuals.

$p(y|\mathbf{x})$ denote the posterior density for $y$ given the observation $\mathbf{x}$. For example, $p(1|\mathbf{x})$ means that given the observation $\mathbf{x}$, what is the probability that sample belongs to the class males. Similarly, we can interpret $p(2|\mathbf{x})$. $p(\mathbf{x}|y)$ stands for the class conditional probability density. For example, $p(\mathbf{x}|1)$ denote the probability density for males and $p(\mathbf{x}|2)$ denote the probability density for females respectively. In supervised learning, these class conditional densities are usually known in advance. Finally $p(y)$ denote the prior probability for the class label $y$. For example, $p(1)$ denotes the probability that an individual/example selected randomly from the population is a male.

Assuming that, the prior probabilities for the classes and class conditional densities are known, Bayes rule says, assign an observation $x_{0}$ whose class label is not known, to the class, say, male, if \begin{equation*} p(\mathbf{x_{0}}|1)p(1)>p(\mathbf{x_{0}}|2)p(2). \end{equation*} Note that, $p(\mathbf{x})$ is not used in the classifier. Classification rules usually require class conditional densities.


$P(y|x)$ stands for the probability of y occurring, given x has occurred.

Using bayes theorem, this can also be written as:

$P(y|x) = \frac {{P(x|y) \times P(y)}}{ P(x)}$

It is these values that you generally know and can use.

  • $\begingroup$ Why has this been down voted? $\endgroup$
    – grldsndrs
    Nov 8, 2016 at 6:15
  • $\begingroup$ I have vote it up $\endgroup$ Jan 2, 2017 at 3:35

Say $x$ is sex and $y$ is employed. You can restate your $p(y|x)$ as:

$$ p(employed=1~|~sex=male) $$

Now, it's a simple matter of counting the relevant examples or looking up in the appropriate contingency table.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.