I want more info into if k-means with Mahalanobis distance is a mathematically/methodologically correct option for datasets with different variance clusters. The steps are:
- Create aggregate datasets (initially randomly or another way, doesn't matter)
- Estimate mu, sigma for each aggregate/cluster dataset
- Recompute clusters by calculating the Mahalanobis distance of each point to each cluster and updating the clusters.
- go to 2 until clusters do not change.
I have seen this implementation used, also have seen it in dissertations. Yet something doesn't feel comfortable about it. There is no way to have an 'absolute' best clustering (i.e. silhouette metric) with Mahalanobis. You can only estimate the error (e.g. Bayesian Information Criterion) of your model (different normal distributions) on your data. And you can definitely over-fit. Is this still 'correct' in terms of clustering and methodologically accepted? Is this still termed 'k-means' clustering?
I guess must be valid since I haven't seen any argumentation against it. Yet I feel obliged to ask, just to make sure, before I use it.
Thanks a bunch.