I'm learning about clustering with Pythons
I have a list of sentences (strings) . Im wondering, does the length of the string affects the
For example, I have sentences from 2 words to 35 words, and i have tried number of clusters from 2 to 60, and the biggest
silhouette_score that I get is around 7. Can that affect the
Is it better to filter my data, so that I can select sentences that are much closer by number of words, for example, to set number of words from 20-25, or 5-10?
This is how my code looks like:
list_of_comments = data #cv = TfidfVectorizer(analyzer = 'word', max_features = 6500, lowercase=True, preprocessor=None, tokenizer=None, stop_words = 'english') cv = CountVectorizer(analyzer = 'word', max_features = 8000, lowercase=True, preprocessor=None, tokenizer=None, stop_words = 'english') x = cv.fit_transform(list_of_comments) my_list =  list_of_clusters =  for i in range(2,35): kmeans = KMeans(n_clusters = i, init = 'k-means++', random_state = 0) kmeans.fit(x) my_list.append(kmeans.inertia_) cluster_labels = kmeans.fit_predict(x) silhouette_avg = silhouette_score(x, cluster_labels)*100 print(round(silhouette_avg,2)) list_of_clusters.append(silhouette_avg)