# Math of Gaussian Mixture Models & EM Algorithm

I'm having a hard time understanding the math behind GMMs.

A GMM is a weighted sum over K different Gaussian components with parameters $$\mu_k, \sigma_k, \pi_k$$

From my understanding, the general overview is:

1. Initialize random parameters for each k'th component
2. Use pdf $$p(x | \mu_k, \sigma_k) = \frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{(x-\mu)^2}{2\sigma^2}}$$ to compute a responsibility value for each component
3. Use this responsibility value to update parameters $$\mu_k, \sigma_k, \pi_k$$

My questions are:

1. When and where is Bayes Rule used in this?
2. When and how exactly does the log likelihood function play into this?

Great question!

1. When and where is Bayes Rule used in this?
1. Bayes Rule is used in the E-step of the Expectation-Maximization (EM) algorithm for GMMs. In the E-step, the algorithm computes the probability that each data point belongs to each component of the GMM. This is done using Bayes Rule to compute the posterior probability of each component given the data point. Specifically, Bayes Rule is used to compute the conditional probability of the data point given each component, which is then multiplied by the prior probability of the component to obtain the posterior probability. This step is sometimes referred to as the responsibility calculation since it computes the responsibility of each component for each data point.
1. When and how exactly does the log likelihood function play into this?
1. The log-likelihood function is used to evaluate the goodness of fit of the GMM to the data. In the M-step of the EM algorithm, the parameters of the GMM are updated to maximize the log-likelihood function. Specifically, the mean, variance, and weight parameters of each component are updated to maximize the expected complete data log-likelihood, which is the sum of the expected log-likelihood of each data point under the current GMM parameters. The EM algorithm iterates between the E-step and the M-step until convergence is reached.

The math behind Gaussian Mixture Models (GMMs) can be a bit tricky to understand at first, but once you get the hang of it, it becomes quite intuitive.

Bayes Rule is actually used in the initial step of the EM algorithm, which is used to estimate the parameters of the GMM. The EM algorithm is an iterative method that maximizes the likelihood function of the observed data given the GMM parameters.

In the E-step of the EM algorithm, we compute the posterior probability of each data point belonging to each Gaussian component using the Bayes Rule.

Bayes Rule states that:

$$P(A \mid B)=\frac{P(B \mid A) P(A)}{P(B)}$$

Specifically, we use the following formula:

$$p\left(z_k \mid x_n, \theta^{(t)}\right)=\frac{\pi_k^{(t)} \mathcal{N}\left(x_n \mid \mu_k^{(t)}, \sigma_k^{2(t)}\right)}{\sum_{j=1}^K \pi_j^{(t)} \mathcal{N}\left(x_n \mid \mu_j^{(t)}, \sigma_j^{2(t)}\right)}$$

where

• $$x_n$$ is the $$n^{th}$$ data point,
• $$z_k$$ is a binary random variable indicating whether $$x_n$$ belongs to the $$k^{th}$$ Gaussian component,
• $$\theta^{(t)}$$ represents the current estimate of the GMM parameters, and
• $$\mathcal{N}(x_n|\mu_k^{(t)},\sigma_k^{2(t)})$$ is the probability density function of a Gaussian distribution with mean $$\mu_k^{(t)}$$ and variance $$\sigma_k^{2(t)}$$ evaluated at $$x_n$$.

This formula gives us the probability that each data point belongs to each Gaussian component given the current estimate of the GMM parameters. These probabilities are used to update the GMM parameters in the M-step of the algorithm.

The log-likelihood function comes into play in the M-step of the EM algorithm. In this step, we update the GMM parameters to maximize the log-likelihood of the observed data. Specifically, we update the mean, variance, and weight parameters of each Gaussian component using the following formulas:

\begin{aligned} \mu_k^{(t+1)} & =\frac{\sum_{n=1}^N p\left(z_k \mid x_n, \theta^{(t)}\right) x_n}{\sum_{n=1}^N p\left(z_k \mid x_n, \theta^{(t)}\right)} \\ \sigma_k^{2(t+1)} & =\frac{\sum_{n=1}^N p\left(z_k \mid x_n, \theta^{(t)}\right)\left(x_n-\mu_k^{(t+1)}\right)^2}{\sum_{n=1}^N p\left(z_k \mid x_n, \theta^{(t)}\right)} \\ \pi_k^{(t+1)} & =\frac{\sum_{n=1}^N p\left(z_k \mid x_n, \theta^{(t)}\right)}{N} \end{aligned}

where $$N$$ is the number of data points.

After updating the parameters, we compute the log-likelihood of the observed data using the following formula: $$\ln p(X \mid \theta)=\sum_{n=1}^N \ln \sum_{k=1}^K \pi_k \mathcal{N}\left(x_n \mid \mu_k, \sigma_k^2\right)$$

The goal of the EM algorithm is to iteratively update the GMM parameters to maximize this log-likelihood. When the change in the log-likelihood between iterations is below a certain threshold, we consider the algorithm to have converged and return the final estimate of the GMM parameters.

In simple words, the log-likelihood function comes into play during the iterative optimization process. The goal of GMM is to maximize the log-likelihood function with respect to the parameters $$\mu_k, \sigma_k, \pi_k$$. In other words, we want to find the parameters that make the observed data most likely. During each iteration, we compute the log-likelihood of the data given the current parameter values and then update the parameters to increase the likelihood of the data. This iterative process continues until convergence (i.e. when the change in log-likelihood between successive iterations falls below a certain threshold).

I hope this helps clarify the use of Bayes Rule and log likelihood in GMMs!

• Amazing explanation. Thank you so much for your help. I appreciate it alot. Apr 25, 2023 at 16:06
• Glad that it helped :) Apr 25, 2023 at 17:36