1
$\begingroup$

The Bayes theorem states that:

\begin{equation} P(h|D) = \frac{P(D|h)P(h)}{P(D)} \end{equation} where $D$ is the dataset and $h$ is an hypothesis from the hypothesis space $H$. Now (I'm not sure so if I'm wrong please correct me) I can consider:

  • $P(h|D)$ = the probability $h$ has generated the dataset $D$. More specifically, for each $h$ we have a probability that it has generated the dataset $D$.
  • $P(D|h)$ = the probability that $D$ has been generated by $h$. More specifically, for each possible dataset $D$, a certain hypothesis $h$ (that we have) can have generated it.

And I can represent them visually, for example:

enter image description here

enter image description here

Now, if we know the prior probability $P(h)$ then we can compute the maximum a posteriori hypothesis with the following formula:

\begin{equation} h_{MAP} = argmax_{h \in H} P(h|D) = argmax_{h \in H} \frac{P(D|h)P(h)}{P(D)} \end{equation}

Otherwise, we can consider the maximum likelihood hypothesis:

\begin{equation} h_{ML} = argmax_{h \in H} P(D|h) \end{equation}

At this step I don't understand how I compute $h_{ML}$ because if I consider $P(D|h)$ represented as in the previous example in the cartesian space we have $D$ in the x-axis, so if I consider the $argmax P(D|h)$ I will find the best $D$ and not the best hypothesis $h$.

What am I doing wrong? Are probabilities $P(h|D)$ and $P(D|h)$ not well interpreted in the cartesian space?

$\endgroup$
0
$\begingroup$

We can define a function $f(D,h) =P(D|h)$, that is it is a function of both $h$ and $D$ and we want to maximize it.

In the event that $D$ is already fixed to be $\hat{D}$, then the goal should be to maximize the function $$g(h) = f(\hat{D}, h) = P(\hat{D}|h).$$

We can plot the function $g$, here $g$ is a function of $h$ rather than $D$.

$\endgroup$
  • $\begingroup$ So you are saying "Since I know the dataset (and I define it as $\hat{D}$, I can compute $P(\hat{D}|h)$ for each hypothesis $h \in H$, after that I return the argument $h$ which maximize $P(\hat{D}|h)$", right? $\endgroup$ – alex_the_great Oct 29 '18 at 10:32
  • $\begingroup$ yup, that is the meaning of $\arg \max_{h \in H} P(D|h)$. $\endgroup$ – Siong Thye Goh Oct 29 '18 at 11:47
  • $\begingroup$ ok but in this way I need to know all possible $h \in H$ and in practical problems this is not feasible, so how actually is computed the best hypothesis $h$? $\endgroup$ – alex_the_great Oct 29 '18 at 12:06
  • $\begingroup$ It depends on your hypothesis space. It can be finite or $h$ can depends on certain parameters that we try to choose to maximize the likelihood. $\endgroup$ – Siong Thye Goh Oct 29 '18 at 12:14
  • $\begingroup$ parameters you are referring to with respect to $h$ are those defined with the vector $\theta$ or the weight vector $w$? If yes, those weights are for example used in machine learning models like neural networks? $\endgroup$ – alex_the_great Oct 29 '18 at 12:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.