Silhouette calculation in k-means

Question

I am trying to compute Silhouette with k-means. However I have the value really close to 0 and the clusters are very clearly separated. Do you know where can be the problem? This is the code:

n_samples, n_features = data.shape
n_digits=2
labels=data[:,-1]
data=data[:,:-1]
sample_size = 300

print("n_digits: %d, \t n_samples %d, \t n_features %d"
      % (n_digits, n_samples, n_features))
print(79 * '_')
print('% 9s' % 'init'
      '    time  inertia    homo   compl  v-meas     ARI AMI  silhouette')

def bench_k_means(estimator, name, data):
    t0 = time()
    estimator.fit(data)
    print('% 9s   %.2fs    %i   %.3f   %.3f   %.3f   %.3f   %.3f    %.3f'
          % (name, (time() - t0), estimator.inertia_,
             metrics.homogeneity_score(labels, estimator.labels_),
             metrics.completeness_score(labels, estimator.labels_),
             metrics.v_measure_score(labels, estimator.labels_),
             metrics.adjusted_rand_score(labels, estimator.labels_),
             metrics.adjusted_mutual_info_score(labels,  estimator.labels_),
             metrics.silhouette_score(data, estimator.labels_,
                                      metric='euclidean',
                                      sample_size=sample_size)))         

pca = PCA(n_components=n_digits).fit(data)
bench_k_means(KMeans(init=pca.components_, n_clusters=n_digits, n_init=1),
              name="PCA-based",
              data=data)

print(79 * '_')

The obtained Silhouette is 0.052 and this is the obtained k-means clustering.

Thanks,

Laia

Answer 1

Since the points in each cluster are fairly spread out along the direction of the separating line, the average distance within a cluster is probably not that much smaller (as you might hope) than the average distance to points in the other cluster. A small positive value could be plausible. (Maybe look at the distribution of the points along the second principal component. A big spread could be the problem.)