I'm currently trying to use cluster analysis as a tool for time-series aggregation for a project of mine. The dataset is high-dimensional (386-d), so no chance in assessing the cluster validity visually.
I'm using three different clustering algorithms (k-means++, k-medoids PAM, fuzzy c-means) to find representative periods. As I do not know how many periods (thus, how many centers/medoids k) are present in the dataset, I want to use an internal cluster validity index (cvi) for it. (Basic procedure: run the clustering with multiple k's and plot the cvi against the k; choose highest/lowest k dependent on cvi optimum).
Let's stick with k-means as an example. It is non-deterministic, thus I initiate it multiple times with varying starting points. It then tries to reduce the intra cluster variance. The result with the lowest intra cluster variance of the various initiations with the same k is then kept.
My question is: should the cluster validity index that I use consider the intra cluster variance as a cohesion measure? So to speak: should the cvi use the objective function of the clustering algorithm as a measure to assess "goodness" of clustering?
On the one hand I think, that this would be a good idea, as it can assess, where the clustering algorithm was "most successfull" in its objective function. On the other hand I think, that for a good clustering it shouldn't depend on the same (cohesion) measure. Thus, using the same measure would increase the likelihood of identifying a "bad" clustering, that might represent an unnatural cluster, drawn from random points.
What's "the truth" here?