I am trying to compare different clustering algorithms for my text data. I first calculated the tf-idf matrix and used it for the cosine distance matrix (cosine similarity). Then I used this distance matrix for K-means and Hierarchical clustering (ward and dendrogram). I want to use the distance matrix for mean-shift, DBSCAN, and optics.

Below is the part of the code showing the distance matrix.

from sklearn.feature_extraction.text import TfidfVectorizer

#define vectorizer parameters
tfidf_vectorizer = TfidfVectorizer(max_df=0.8, max_features=200000,
                                 min_df=0.2, stop_words='english',
                                 use_idf=True, tokenizer=tokenize_and_stem, ngram_range=(1,3))

%time tfidf_matrix = tfidf_vectorizer.fit_transform(Strategies) #fit the vectorizer to synopses

terms = tfidf_vectorizer.get_feature_names()

from sklearn.metrics.pairwise import cosine_similarity
dist = 1 - cosine_similarity(tfidf_matrix)

I am new to both python and clustering. I found the code for K-means and hierarchical clustering and tried to understand it but I cannot apply it for other clusterings algorithms. It would be very helpful if I can get some simple explanation of each clustering algorithm and how this distance matrix can be used to implement (if possible) in different clustering.

Thanks in advance!


Each of those selected clustering algorithms can be fit using cosine distances in scikit-learn:

from sklearn.cluster import DBSCAN, MeanShift, OPTICS
from sklearn.metrics.pairwise import cosine_distances

# Define clustering algorithms
algorithms = [DBSCAN, MeanShift, OPTICS]

# Placeholder for results
results = dict.fromkeys((a.__name__ for a in algorithms))

# Fit each clustering algorithm and store results
for algorithm in algorithms:
    results[algorithm] = algorithm(metric=cosine_distances).fit(X)
| improve this answer | |
  • $\begingroup$ Thanks for the fast reply but I am getting an error. NameError: name 'clustering_algorithms' is not defined. Also, what would be X? Where I would be using 'dist' which I have calculated (in my code)? Please can you elaborate a little more? $\endgroup$ – Piyush Ghasiya Mar 5 at 3:23
  • $\begingroup$ I had a typo; I fixed it. X is the standard name for a data array in scikit-learn. You don't need dist, use cosine_distances instead. $\endgroup$ – Brian Spiering Mar 5 at 6:08
  • $\begingroup$ Does that mean that I should replace X with tfidf_matrix (as visible from my code above)? When I did that I again got an error: TypeError: __init__() got an unexpected keyword argument 'metric'. $\endgroup$ – Piyush Ghasiya Mar 5 at 7:21
  • $\begingroup$ Sorry for my naive questions. $\endgroup$ – Piyush Ghasiya Mar 5 at 7:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.