I'm looking for an implementation of k-modes in pyspark. I found this and this as implementations.
First, I tried implementing k-modes using the first link and faced issues. So I went ahead and tried the second implementation on github. This one worked (as in ran without errors), but I'm still unable to use it in the proper way because of no guide/usage techniques. According to the github page:
n_modes=36
partitions=10
max_iter=10
fraction = 50000 * partitions / (data.count() * 1.0)
data = data.rdd.sample(False,fraction).toDF()
method=IncrementalPartitionedKMetaModes(n_partitions = partitions, n_clusters = n_modes,max_dist_iter = max_iter,local_kmodes_iter = max_iter,
similarity = "frequency", metamodessimilarity = "hamming")
cluster_metamodes = method.calculate_metamodes(data)
I'm not able to understand how to use this cluster_metamodes, or what it even means.
In short, k-modes will be performed for each partition in order to identify a set of modes (of clusters) for each partition. Next, k-modes will be repeated to identify modes of a set of all modes from all partitions. These modes of modes are called metamodes here.
Can someone explain the concept and tell how to use this method of clustering on a pyspark dataframe of categorical values?