I'm familiar with the Expectation-Maximixation algorithm and, until now, I thought it was the only way to maximize the likelihood of the observed data, assuming a Gaussian mixture model.
In the last days while studying for Dirichlet Process I've discovered the Gibbs Sampling that can fit parameters of a GMM.
Now I'm not familiar at all with the Gibbs sampling, and I would really appreciate if you could give me an overview of it, comparing it to EM. What is the basic idea behind it?
Gibbs sampling is a probabilistic framework which assumes all parameters to be learned are random variables. Here, you assume prior distributions for the parameters and find the full conditionals based on the relations specific to the problem, e.g. if you have three parameters, $\alpha,\beta,\theta$, you derive $p(\alpha|,\beta,\theta)$ ,$p(\beta|\alpha,\theta)$ and $p(\theta|\alpha,\beta)$. After deriving the full conditionals you sample from them and execute many iterations. It's like asking for $\alpha$ if we know $\beta$ and $\theta$, asking for $\beta$ when we know $\alpha$ and $\theta$ etc. After sufficient number of iterations you'll converge to a solution. In GMM, these parameters are mixture proportions, which can have a Dirichlet prior for example, and means and deviations of each gaussian component (you can choose suitable prior distributions for these). An example design would include full conditionals $p(\mu_i|\boldsymbol{\sigma},\boldsymbol{\pi})$, $p(\sigma_i|\boldsymbol{\mu},\boldsymbol{\sigma})$ and $p(\boldsymbol{\pi}|\boldsymbol{\mu},\boldsymbol{\sigma})$. Here choice of your priors are important because when you get formulas for these full conditionals it's better to have common distributions for easy sampling, e.g. $p(\mu_i|..)$ turns out to be gamma distributed etc. If not, you have to design samplers for these probability distributions, which can sometimes cause some pain. It is also an iterative algorithm as EM, but EM is purely an optimization procedure while Gibbs sampling is a Bayesian approach.
