So I have these vectors called matrix_ after I applied TF-IDF (term frequency-inverse document frequency), and I also converted it to dataframe matrixDF_ .

I wanted to cluster these matrix_ vectors with MiniBatchKMeans from sklearn. But since the vectors are large (569 rows and 829 columns), then I need to find the optimum K-Cluster first.

That is why I use KElbowVisualizer from Yellowbrick. At first, it works fine and I got the optimum k-cluster. However, when rerunning it again, it showed me a different number of k-cluster. And that is where I am confused then.

Anyone can help me with this problem?

here is my code:

import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.cluster import MiniBatchKMeans
from scipy.spatial.distance import cdist
from sklearn.metrics import silhouette_score
from yellowbrick.cluster import KElbowVisualizer

""" vectorize the words. getting the tf-idf of each words """
vec = TfidfVectorizer(
     lowercase = True,
     stop_words = 'english',

matrix_ = vec.fit_transform(df)
matrixDF_ = pd.DataFrame(matrix_.toarray(), columns=vec.get_feature_names())

model = MiniBatchKMeans(MiniBatchKMeans(init_size=559, random_state=None))
visualizer = KElbowVisualizer(model,k=(2, 11), metric='distortion', 
timings=True, size=(1080, 720))
visualizer.fit(matrix_)        # Fit the data to the visualizer

I also tried fit the dataframe matrixDF_. to the visualizer, but each output after every run has different output number.

Here are some of output examples. First output: enter image description here

second output: enter image description here


You are using different random starting points each time you run k-means (random_state=None) This means you may get different clusterings, different clustering metrics each time. That's expected. What you may wish to do is average the results over several different runs to get a more reliable estimate of the loss at each k.

  • $\begingroup$ I tried it with random_state=100 but I still get random result @Sean Owen $\endgroup$
    – user89849
    Feb 11 '20 at 16:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.