# KMeans clusterization on documents

Whether correct or not, I'm not able to judge being myself in the early days of the Data Science.

However, I have applied a Kmeans on a corpus where some random documents (very short sentences) have been added. These have been vectiorized so to be suitable.

With clusterization results at hands, I was somehow expecting the vectors (keyword) to fall only in one cluster at a time (and no more than that). This is not the case.

In some circumstances, I have a vector falling in two clusters and I wonder why this is the case.

• Is this because of the inappropriate usage of Kmeans on vectors made from documents?
• Is this normal as the way Kmeans works (moving the centroids, but de facto assigning objects to the nearest cluster by distance)?
• Is this overlap due to the fact that in analysing my results I assess the whole group of items within a cluster and not just (say) the top X near to the center?

-- Example:

corpus = [
'The car is driven on the road.',
'The truck is driven on the highway.',
'The train run on the tracks.',
'The bycicle is run on the pavement.',
'The flight is conducted in the air.',
'The baloon is conducted in the air.',
'The bird is flying in the air.',
'The man is walking in the street.',
'The pedestrian is crossing the zebra.',
'The pilot flights the plane].',
'On the route, the car is driven.',
'On the road, the truck is moved.',
'The train is running on the tracks.',
'The bike is running on the pavement.',
'The flight takes place in the sky.',
'Birds don''t fly when is dark',
'The baloon is in the water.',
'The bird flies in the sky.',
'In the road, the guy walks.',
'The pedestrian is passing through the zebra.',
'The pilot is flying the plane.',
'This is a Japanese doll.',
'I really want to go to work, but I am too sick to drive.',
'Christmas is coming.',
'With the daylight saving time turned off it''s getting dark soon.',
'The body fat may compensates for the loss of nutrients.',
'Mary plays the piano.',
'She always speaks to him in a loud voice.',
'Wow, does that work?',
'I don''t like walking when it is dark',
'Last Friday in three week’s time I saw a spotted striped blue worm shake hands with a legless lizard.',
'My Mum tries to be cool by saying that she likes all the same things that I do.',
'Mummy is saying that she loves me being a pilot when in reality she is scared all the time I take off.',
'Where do random thoughts come from?',
'A glittering gem is not enough.',
'We need to rent a room for our party.',
'A purple pig and a green donkey flew a kite in the middle of the night and ended up sunburnt.',
'If I don’t like something, I’ll stay away from it.',
'The body may perhaps compensates for the loss of a true metaphysics.',
'Don''t step on the broken glass.',
'It was getting dark, and we weren''t there yet.',
'Playing an instrument like the guitar takes out the stress from my day.']

from sklearn.feature_extraction.text import TfidfVectorizer

vectorizer = TfidfVectorizer(analyzer='word',
max_df=0.8,
max_features=50000,
lowercase=True
)

X = vectorizer.fit_transform(corpus)

from sklearn.cluster import KMeans

num_clusters = 11
kmean = KMeans(n_clusters=num_clusters, random_state=1021)
clusters = kmean.fit_predict(X)


--

If you explore the clusters variable, you will notice the overlaps I am talking about. For instance the keyword baloon appeara in both cluster 10 and 0.

There are 12 overlaps, which on a 33 unique keywords dataset represents 1/3, so I won't say something I could be happy with.

• What do you mean by "a vector falling in two clusters"? k-means eventually assigns each observation to the nearest cluster center's cluster, so that's only one possibility (except in the unlikely case when an observation is at equal distance from several cluster centers). Could you maybe give us an example/insight of data and results? Dec 7 '19 at 7:59
• Hi @RomainReboulleau you can read vector as observation in this case. The observation are based on the interpretation of the vector I passed to Kmeans, and the vector is made by keywords previously parsed. I'll try to put together a dataset and share. My only other guess is the possibility of two vector/keyword holding the same value, but this doesn't seem to be the case when looking at it. Dec 7 '19 at 8:15
• Hi @RomainReboulleau just added the example as previously discussed. Any hints, is massively appreciated. Dec 9 '19 at 7:45

Let us assume that your corpus has n distinct keywords. For a k-means algorithm, each keyword is an axis in n dimensional space. Document is a point in that n dimensional space.

K-means algorithm will allocate each point (a document) to a single cluster. When you say a keyword is appearing in two clusters, it probably implies: that particular dimension/keyword is important for both clusters.

Let us take hypothetical example: if you have a patient's blood pressure, cholesterol levels and bunch of other medical parameters. Let us say you discretize blood pressure to 2 or 3 levels. If you run k-means on this data, each patient will be assigned a unique cluster. But it is quite possible that two (or even more) clusters all have patients with > 120 systolic blood pressure.

You need to probably read the results of the k-means more carefully.

• Thanks @hssay. Basically you are saying that clusters can eventually intersect. I had the impression, from the many examples I have seen, that this was not the case. It would be quite fair if keywords will be overlapping. I attempted to create overlaps on purpose, so to see what would have happened, and the attribution of a vector to two or more cluster basically could be meaning the K-Means has somehow being able to understand the importance of the keyword in the different contexts? In which other way would you suggest to read the K-Means results? I have done a simple array parsing/comparison Dec 9 '19 at 17:03

I think you may be mixing up things. In the example you provided, there are 42 sentences, each is transformed through TfIdfVectorizer, which gives us a sparse matrix of shape (42, 174). Then, each sentence representation as vector is used to cluster with k-means, and each sentence is thus assigned to a cluster.

Single words are not processed, only whole sentences. If the "baloon" keyword appears in two sentences, it does not necessarily mean that both sentences will fall into the same cluster. However, I am surprised by what you state because the sentences containing "baloon" both fall into the same cluster (#7). This makes me think that you misinterpreted the results.

>>> import numpy as np
>>> np.argwhere(["baloon" in sentence for sentence in corpus])
array([[ 5],
[16]], dtype=int64)
>>> clusters[5]
7
>>> clusters[16]
7


Anyway, it could be that sentences containing "baloon" fall into different clusters. This depends on the other words in the sentence, the number of clusters, the rest of the dataset and the clustering method. For instance, it could be the case if sentences containing "baloon" were not so much alike.

• Rebolleau thanks for your time. I think there is a problem as we are looking at two things. Your np.where clause is looking at the presence of baloon in the corpus. When I am trying to check which keyword was significant in each document of the corpus, I was doing that again the kmean.cluster_centers_. Am I doing it correctly? That's what I am now puzzled with as I'm learning my way. Dec 11 '19 at 14:55
• OK, I think I'm starting to see what's wrong in my answer. However, when I look at kmean.cluster_centers_[:,7], which corresponds to the "baloon" coordinate of the 11 centers, it is null for all clusters except #7, which is actually the one containing sentences #5 and #16 (and it makes sense). How are you coming to the conclusion that clusters #0 and #10 somehow include the "baloon" keyword? Dec 12 '19 at 5:57
• I truly believe I have misinterpreted the results. I was, in-fact, picking the correspondence and intersecting the results with my vocabulary using the kmean.cluster_centers_ variable. This is because both X and the cluster_center has the exact number of items as per the keywords dictionary, hence I was under the impression the "significant" keywords (or whatever this could be in the feature) was represented by the latter. Evidently this is not the case. So, what does the kmean.cluster_centers_ represents? And what is its role? Dec 12 '19 at 13:43
• In k-means, each cluster is represented by a "center", which is a point in the $p$-dimension input space (174 dimensions in your example above), so kmean.cluster_centers_ has shape (11, 174), while X has shape (42, 174). Each of the 42 observations in X is assigned to the cluster of which center is the closest, this is why multiple assignment is only possible if a point is at equal distance from two centers. The principle of k-means is also that each center is the mean of all observations assigned to it (which is how the algorithm converges). Dec 13 '19 at 6:11
• thanks. Yes, somehow that was understandable. What I cannot get is the numerical relationship. If I look at any cluster, the number next to each of the dimensions contains something that apparently does not relate to the number shown when X is recalled. See here ibb.co/L0PRsh9. I need to understand the X <-> cluster reference. Dec 16 '19 at 14:21