1
$\begingroup$

I'm learning about clustering with Pythons Scikit-Learn lib. I have a list of sentences (strings) . Im wondering, does the length of the string affects the silhouette_score.

For example, I have sentences from 2 words to 35 words, and i have tried number of clusters from 2 to 60, and the biggest silhouette_score that I get is around 7. Can that affect the silhouette_score? Is it better to filter my data, so that I can select sentences that are much closer by number of words, for example, to set number of words from 20-25, or 5-10?

This is how my code looks like:

list_of_comments = data

#cv = TfidfVectorizer(analyzer = 'word', max_features = 6500, lowercase=True, preprocessor=None, tokenizer=None, stop_words = 'english')
cv = CountVectorizer(analyzer = 'word', max_features = 8000, lowercase=True, preprocessor=None, tokenizer=None, stop_words = 'english')  

x = cv.fit_transform(list_of_comments)

my_list = []
list_of_clusters = []
for i in range(2,35):

    kmeans = KMeans(n_clusters = i, init = 'k-means++', random_state = 0)
    kmeans.fit(x)
    my_list.append(kmeans.inertia_)

    cluster_labels = kmeans.fit_predict(x)

    silhouette_avg = silhouette_score(x, cluster_labels)*100
    print(round(silhouette_avg,2))
    list_of_clusters.append(silhouette_avg)
$\endgroup$
  • $\begingroup$ 1) How are you vectorizing your sentences? 2) Which clustering algorithm are you using, k-means? $\endgroup$ – BrunoGL Sep 7 at 12:37
  • $\begingroup$ Im using k-means and i have used TfidfVectorizer and with it, i got around 3 for max silhouette_score , and then I have tried CountVectorizer and with it, i got around 7 for max silhouette_score . $\endgroup$ – taga Sep 7 at 17:16
1
$\begingroup$

Yes, the length is very likely to influence any kind of similarity score. In general:

  • String containing very few words will have their highest similarity scores with other short strings with which they share one or two words in common. However if the words they contain are not common at all, they will have a lot of zero similarity scores (possibly only zeros).
  • Long strings (i.e. with many words) usually have low similarity scores, simply because they are unlikely to have a high proportion of their words in common. However compared to short strings they rarely have no word in common so they are much less likely to have lots of zero similarity scores.

As a result strings of similar length may tend to cluster together, depending on the data.

$\endgroup$
  • $\begingroup$ Thanks! Is there any better way to cluster text data? $\endgroup$ – taga Sep 7 at 22:59
  • $\begingroup$ @taga afaik there's no generally better way, it depends a lot on the data and the task. assuming semantic similarity is more important, there are a few simple things that can help sometimes: removing stop words, lemmatize (i.e. replace words with their lemma), remove words which occur rarely (i.e. minimum frequency threshold). you could even consider using WordNet for semantic similarity. $\endgroup$ – Erwan Sep 7 at 23:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.