I am having some difficulty in seeing connection between PCA on second order moment matrix in estimating parameters of Gaussian Mixture Models. Can anyone connect the above??
I believe the claim that you are referring to is that the maximum-likelihood estimate of the component means in a GMM must lie in the span of the eigenvectors of the second moment matrix. This follows from two steps:
- Each component mean in the maximum-likelihood estimate is a linear combination of the data points. (You can show this by setting the gradient of the log-likelihood function to zero.)
- Any linear combination of the data points must lie in the span of the eigenvectors of the second moment matrix. (You can show this by first showing that any individual data point must lie in the span, and therefore any linear combination must also be in the span.)