# Statistical test for comparing number of clusters in data

I am performing $$K$$-means clustering on a dataset consisting of $$n$$ observations and $$d$$ variables, and I'm trying to determine the optimal number of clusters. Is there a test that can determine the statistical significance of adding another cluster?

I have considered the $$F$$-test with the following $$F$$-statistic

$$F = \frac{ \Big( \frac{WCSS_k-WCSS_{k+1}}{d(k+1)-dk} \Big)}{ \Big( \frac{WCSS_{k+1}}{n-d(k+1)} \Big)} = \frac{ \Big( \frac{WCSS_k-WCSS_{k+1}}{d} \Big)}{ \Big( \frac{WCSS_{k+1}}{n-dk-1} \Big)}$$

where $$WCSS_i$$ is the within-cluster sum of squares, or inertia, for the model containing $$i$$ clusters. I obtained the general formula for the $$F$$-statistic here under "Regression Problems." In this case, I am treating inertia as a measure of error in the model, and $$di$$ is the number of parameters in the model with $$i$$ clusters because each of $$i$$ clusters has a $$d$$-dimensional mean vector at the center of the cluster.

Of course, adding another cluster will often reduce the inertia and never increase it... just as adding another predictor to a linear regression model will often reduce the RSS. The question is whether the reduction observed by adding another cluster is the result of clustering noise or the result of modelling a real pattern in the data. I am assuming a statistically significant p-value would indicate the latter.

Any thoughts?

I am not exactly sure where this $$F$$-statistic comes from but it looks like an adopted Chow-Test. I will not dive deeper into this, since there is a general problem, here:
In general you only try to optimize the within-cluster sum of squares. Your $$F$$-Test checks, if another cluster brings a significant improvement. The problem here is, that more clusters mean lower within-cluster sum of squares (there might be exceptions, but the general trend holds). The extreme case would be one cluster per sample (with zero within-cluster sum of squares). The significance test will prevent this extremum.