Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
A posterior $p(\theta\vert x) = \frac{p(x \vert \theta)p(\theta)}{p(x)} $
Many materials say that since the $p(x)$ is a constant, the $p(x)$ can be ignored. Thus, $p(\theta\vert x) \propto p(x \vert \theta)p(\theta)$
My question is why $p(x)$ is a constant and ignored. Is this because even though we don't know the distribution x, there is a corresponding true distribution for $x$? So, $p(x) $ is a constant (we don't know but already determined) and thus, can be ignored?
You are right, $P(x)$ is the underlying distribution of data, and it can be assumed constant, simply because it is independent of our modeling practice. However $P(x)$ may change gradually over time, this phenomenon is called "distribution shift". For example, the distribution of height in a country may change over years.
Note that "prior" is a reserved word for $P(\theta)$ which is the distribution of model parameters in Bayesian modeling. In non-Bayesian modeling there is no notion of "prior". $P(x|\theta)$ is called likelihood and $P(x)=\sum_{\theta}P(x|\theta)P(\theta)$ is called marginalized likelihood (likelihood marginalized over model parameters).
The maximum a posteriori definition usually goes like this:
$$ \arg\max_{\theta} p(\theta|x) = \arg\max_{\theta} \frac{p(x|\theta)\cdot p(\theta)}{p(x)} $$
and given that $p(x)$ is independent of $\theta$, it's not needed for finding the $\arg\max_{\theta}$, then you have
$$ \arg\max_{\theta} p(\theta|x) = \arg\max_{\theta} \frac{p(x|\theta)\cdot p(\theta)}{p(x)} = \arg\max_{\theta} p(x|\theta)\cdot p(\theta)$$
Yes. Presumably whatever process 'generates' data produces data that follows some distribution. $p(x)$, however, is just a number, and whatever it is, it is a constant w.r.t. $\theta$.
I wouldn't call $p(x)$ a prior, as it implies there's some posterior probability of the data we compute. We don't; nothing about this process updates belief about the probability of the. "Prior" refers to $p(\theta)$.
$p(x)$ can't be ignored if you really want the value of $p(\theta|x)$. But usually you compute that from the posterior distribution that you will have a closed-form formula for.
