I am doing a tweet clustering using DBSCAN algorithm. I use all the preprocessing steps and convert sentences to vector format before applying the algorithm. However, It always puts a lot of tweets in to the '0' class. The following is the plot showing eps with number of clusters.

enter image description here

The following are the parameters that I pass.

dbscan = DBSCAN(eps=0.15, min_samples=2, metric='cosine').fit(x)

The following are the resulting clusters.

enter image description here

-1     1221
 0     1349
 1        2
 2        2
 3        4
 67       3
 68       3
 69       2
 70       2
 71       2

What is the reason that class 0 getting a high number of tweets than any other classes?

  • $\begingroup$ Can you please share some more insight on how you are processing the words before clustering. From initial observation all your clusters might be having some word that results in the cluster. Using word2vec embeddings and Euclidean might help $\endgroup$ Nov 7 '20 at 7:39
  • $\begingroup$ you need to tell us about how you converted tweets into vectors. That is the key part $\endgroup$ Nov 7 '20 at 12:55
  • $\begingroup$ @mahesh ghanta: Thanks, I have used Bag of Words, TFIDF, Spacy Vectors and also, Word2Vec. All produce the cluster No '0' with a large number of results. $\endgroup$ Nov 8 '20 at 9:20
  • $\begingroup$ @Kasra Manshaei: Thank you. Please see the previous comment. $\endgroup$ Nov 8 '20 at 9:20
  • $\begingroup$ Did you check the words that are key are important in this cluster ? Do they make sense? Also could you increase the neighbouring samples to atleast 5? $\endgroup$ Nov 8 '20 at 11:52

Two things: eps and quantitative representation of text.

You see that there is only for eps=0.15 a lot of clusters. But for others a lot less. This is hyper parameter that needs to be optimised (and min_samples)

And the other thing thats more important is what you use quantitative representation of text. You said Bag of Words, TFIDF, Spacy Vectors and also, Word2Vec, but did you tune them? DId you tree embeddings etc etc. There is a lot of improvement here, and when its good dbscan will function a lot better.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.