I have categorical data and I'm trying to implement k-modes using the GitHub package available here. I am trying to create clusters in my (large) dataset of say, 5-7 records, each of most similar records.
However, as of now I have no means to select the optimal 'k' which would result in maximum silhouette score, ideally. This would be ideal as k-modes works on dissimilarity/similarity measure as a distance. So I would assume that silhouette distance would then measure how close/far the clusters are based on the distance metric defined by this dissimilarity and thus, establish the silhouette score. I'm not able to find an implementation of this.
Can I perhaps use the elbow method here? But then, I'm not able to understand how to programmatically determine this, without looking at a graph as I have to do this process repeatedly a large number of times. Currently, an idea is - find k where cost drops substantially. See if the next few values introduce a very less drop in cost or not. If yes, choose this as k, if no.. then what? I'm a little confused at this point.
I was looking online and also found this, which I'm not able to interpret in terms of k modes. I'm looking for any code/suggestions to start me off on the right path.