I have applied K-means and Hierarchical Agglomerative clustering method on some data and clustered them into 5 groups. For validation (agreement) purpose I used this formula:
c=(# of cluster shared/ Total cluster)*100 %
I feel its a wrong validation technique.
Can I use entropy, Rand index, Dunn number etc to measure the validation?
Could always try a silhouette score, although it may not perform well on high dimensional data.
Silhouette scores compare a datapoint's placement relative to its own cluster to its placement relative to other cluster/clusters. The values returned will range between -1 (indicating a poor clustering) to +1 (indicating "good" clustering). Using Wikipedia's example, use 2 clusters and assume there is an observation i that has been placed in Cluster 1. The Silhouette score uses distance measures (Euclidean, for example, but others can also be used) to determine the average distance of i' to every other point in Cluster 1. Then it determines the average distance of i to every point in Cluster 2. The silhouette formula for point i would be:
silhouette(i) = $\frac{b(i)-a(i)}{Max\left \{ a(i), b(i) \right \}}$
Remembering for this example that i is in Cluster 1, then:
Run this comparison across all points and all clusters, then average, and you will obtain a global silhouette score.
You will see most people capture silhouette score in a for loop where they vary the number of clusters; this allows quick comparison of a range of clusters.
Define "shared". Don't assume cluster 1 in A is cluster 1 in B. The numbers are meaningless.
So yes, you should use standard techniques like adjusted Rand index.
