3
$\begingroup$

I have a dataset of some keywords in some text files. Using the append feature I have access each text file and I append all of the keywords to token_dict like this

token_dict="wrist. overlapping. direction. receptacles. comprising. portion. adjacent. side. hand. receive. adapted. finger. comprising. thumb. ..............................."

By using k-means clustering, I clustered this data by using k=3. Now, I want to calculate the distance between each data point in a cluster to its respective cluster centroid. I have tried to calculate euclidean distance between each data point and centroid but somehow I am failed at it. My code is as follows:

from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer, TfidfTransformer
from sklearn.cluster import KMeans
from sklearn.decomposition import PCA
import matplotlib.pyplot as plt
from matplotlib import style
import numpy as np
style.use('ggplot')
token_dict = []
import glob   
path = 'E:\\Project\\*.txt'   
files=glob.glob(path)   
for file in files:     
    f=open(file, 'r')
    text = f.read()
    token_dict.append(text)
vectorizer = TfidfVectorizer(max_df=0.8, max_features=10000, min_df=2, use_idf=True)

#print(X)

km = KMeans(n_clusters=3)
#labels = km.fit_predict(vectorizer)
#print(labels)
X = vectorizer.fit_transform(token_dict).todense()
km.fit(X)
pca = PCA(n_components=2).fit(X)
data2D = pca.transform(X)
# =============================================================================
# cluster_0=np.where(X==0)
# print(cluster_0)
# 
# X_cluster_0 = data2D[cluster_0]
# print (X_cluster_0)
# =============================================================================

# =============================================================================
# def euclidean(X1, X2):
#     return(X1-X2)
# =============================================================================

# =============================================================================
# distance = euclidean(X_cluster_0[0], km.cluster_centers_[0])
# print(distance)
# =============================================================================
# =============================================================================
# 
# km.predict()
# =============================================================================
order_centroids = km.cluster_centers_
centers2D = pca.transform(order_centroids)
labels = km.labels_
colors = ["y.", "b.","g."]
for i in range(len(X)):
    plt.plot(data2D[i][0], data2D[i][1], colors[labels[i]], markersize=10)

plt.scatter(centers2D[:, 0], centers2D[:, 1], marker='x', s=200, linewidths=3, c='r')
plt.show()

Can someone see where I went wrong?

$\endgroup$
  • $\begingroup$ Solved at my own $\endgroup$ – Roshni Amber Jan 2 '18 at 16:16
4
$\begingroup$
def k_mean_distance(data, cx, cy, i_centroid, cluster_labels):
        distances = [np.sqrt((x-cx)**2+(y-cy)**2) for (x, y) in data[cluster_labels == i_centroid]]
        return distances
clusters=km.fit_predict(data2D)
centroids = km.cluster_centers_

distances = []
for i, (cx, cy) in enumerate(centroids):
    mean_distance = k_mean_distance(data2D, cx, cy, i, clusters)
    distances.append(mean_distance)

print(distances)

Using this function I have solved my problem

|improve this answer|||||
$\endgroup$
0
$\begingroup$

I think this is a more elegant solution.

First of all, km.fit_transform() (or km.transform()) gives you back all distances to all clusters. Then you can summarize only the minimum values - which are the distances to the respective closest clusters.

km = KMeans(n_clusters=3)
alldistances = km.fit_transform(data2D)
totalDistance = np.min(corpus.clusterMatrix, axis=1).sum()
|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.