# Does the length of the text data affects the score of clustering

I'm learning about clustering with Pythons Scikit-Learn lib. I have a list of sentences (strings) . Im wondering, does the length of the string affects the silhouette_score.

For example, I have sentences from 2 words to 35 words, and i have tried number of clusters from 2 to 60, and the biggest silhouette_score that I get is around 7. Can that affect the silhouette_score? Is it better to filter my data, so that I can select sentences that are much closer by number of words, for example, to set number of words from 20-25, or 5-10?

This is how my code looks like:

list_of_comments = data

#cv = TfidfVectorizer(analyzer = 'word', max_features = 6500, lowercase=True, preprocessor=None, tokenizer=None, stop_words = 'english')
cv = CountVectorizer(analyzer = 'word', max_features = 8000, lowercase=True, preprocessor=None, tokenizer=None, stop_words = 'english')

my_list = []
list_of_clusters = []
for i in range(2,35):

kmeans = KMeans(n_clusters = i, init = 'k-means++', random_state = 0)
kmeans.fit(x)
my_list.append(kmeans.inertia_)

cluster_labels = kmeans.fit_predict(x)

silhouette_avg = silhouette_score(x, cluster_labels)*100
print(round(silhouette_avg,2))
list_of_clusters.append(silhouette_avg)

• 1) How are you vectorizing your sentences? 2) Which clustering algorithm are you using, k-means? – Bruno Lubascher Sep 7 '19 at 12:37
• Im using k-means and i have used TfidfVectorizer and with it, i got around 3 for max silhouette_score , and then I have tried CountVectorizer and with it, i got around 7 for max silhouette_score . – taga Sep 7 '19 at 17:16