0
$\begingroup$

I am performing $K$-means clustering on a dataset consisting of $n$ observations and $d$ variables, and I'm trying to determine the optimal number of clusters. Is there a test that can determine the statistical significance of adding another cluster?

I have considered the $F$-test with the following $F$-statistic

$$ F = \frac{ \Big( \frac{WCSS_k-WCSS_{k+1}}{d(k+1)-dk} \Big)}{ \Big( \frac{WCSS_{k+1}}{n-d(k+1)} \Big)} = \frac{ \Big( \frac{WCSS_k-WCSS_{k+1}}{d} \Big)}{ \Big( \frac{WCSS_{k+1}}{n-dk-1} \Big)}$$

where $WCSS_i$ is the within-cluster sum of squares, or inertia, for the model containing $i$ clusters. I obtained the general formula for the $F$-statistic here under "Regression Problems." In this case, I am treating inertia as a measure of error in the model, and $di$ is the number of parameters in the model with $i$ clusters because each of $i$ clusters has a $d$-dimensional mean vector at the center of the cluster.

Of course, adding another cluster will often reduce the inertia and never increase it... just as adding another predictor to a linear regression model will often reduce the RSS. The question is whether the reduction observed by adding another cluster is the result of clustering noise or the result of modelling a real pattern in the data. I am assuming a statistically significant p-value would indicate the latter.

Any thoughts?

$\endgroup$

1 Answer 1

1
$\begingroup$

I am not exactly sure where this $F$-statistic comes from but it looks like an adopted Chow-Test. I will not dive deeper into this, since there is a general problem, here:

In general you only try to optimize the within-cluster sum of squares. Your $F$-Test checks, if another cluster brings a significant improvement. The problem here is, that more clusters mean lower within-cluster sum of squares (there might be exceptions, but the general trend holds). The extreme case would be one cluster per sample (with zero within-cluster sum of squares). The significance test will prevent this extremum.

Still: more data samples from the same source/distribution will lead to more clusters. Hence looking only at the within-cluster distances will give you no meaningful clusters. It's more a question of the number of samples.

For this reason, one has to consider both, the distances between and within clusters. Measures from the literature are for example the Davies-Bouldin-Index, the Dunn-Index or Silhouette. These indices are no significance test, but it is common to use them to find an optimal number of clusters from a range of potential numbers.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.