So I have these vectors called
matrix_ after I applied TF-IDF (term frequency-inverse document frequency), and I also converted it to dataframe
I wanted to cluster these
matrix_ vectors with MiniBatchKMeans from sklearn. But since the vectors are large (569 rows and 829 columns), then I need to find the optimum K-Cluster first.
That is why I use KElbowVisualizer from Yellowbrick. At first, it works fine and I got the optimum k-cluster. However, when rerunning it again, it showed me a different number of k-cluster. And that is where I am confused then.
Anyone can help me with this problem?
here is my code:
import pandas as pd from sklearn.feature_extraction.text import TfidfVectorizer from sklearn.cluster import MiniBatchKMeans from scipy.spatial.distance import cdist from sklearn.metrics import silhouette_score from yellowbrick.cluster import KElbowVisualizer """ vectorize the words. getting the tf-idf of each words """ vec = TfidfVectorizer( lowercase = True, stop_words = 'english', use_idf=True ) matrix_ = vec.fit_transform(df) matrixDF_ = pd.DataFrame(matrix_.toarray(), columns=vec.get_feature_names()) model = MiniBatchKMeans(MiniBatchKMeans(init_size=559, random_state=None)) visualizer = KElbowVisualizer(model,k=(2, 11), metric='distortion', timings=True, size=(1080, 720)) visualizer.fit(matrix_) # Fit the data to the visualizer visualizer.show()
I also tried fit the dataframe
matrixDF_. to the visualizer, but each output after every run has different output number.