1
$\begingroup$

I am new in topic modeling and text clustering domain and I am trying to learn more. I would like to use the DBSCAN to cluster the text data. There are many posts and sources on how to implement the DBSCAN on python such as 1, 2, 3 but either they are too difficult for me to understand or not in python.
I have a CSV data that has userID and message that they wrote as follows:

user.csv (number of csv rows:400 (#message))

userID         messages
112   The car was broken and Kevin fixed it
.
.
.

I know some steps to apply DBSCAN such as:

  1. Remove stop words
  2. Find similarity distance ( I have a code that does the cosine similarity)

I am also aware that sci-kit learn has the demo at 4 but I prefer the manual implementation that I can see what's going on in the code.

It would be great if you can provide your help with code that I can run in my side to learn it.

$\endgroup$
  • $\begingroup$ Add the code that you already have to your question to make it answerable. Right now, it is too unspecific. Also beware that results of cosine on short text tend to be very poor, and choosing DBSCAN parameters will lose a big problem. So I doubt you will succeed. $\endgroup$ – Anony-Mousse Jun 30 at 6:28
  • $\begingroup$ @Anony-Mousse thank you for your comment. I just updated the post with my entire code that performs the similarity. The code above calculates both Levenstein and cosine similarity. I chose Levenstein as practice. That's correct, my CSV file has short text that the user provided. Please let me know a good alternative for similarity if you have something in mind. I will appreciate. $\endgroup$ – Bilgin Jun 30 at 19:02
  • $\begingroup$ There is no use in turning a distance matrix into a pandas data frame. That just causes more overhead, and your code already will be very slow. Stick to vectorized numpy where possible. But there is no DBSCAN in your code yet. $\endgroup$ – Anony-Mousse Jun 30 at 19:44
  • $\begingroup$ @Anony-Mousse Sure, I will keep only with numpy and avoid pandas. I just added the DBSCAN code that I found from here but it is very simple and using sci-kit learn. I am new in python and I could not able to write the algorithm from scratch. Please let me know if you have any idea about how I can do it. Thanks $\endgroup$ – Bilgin Jun 30 at 20:25
  • $\begingroup$ Do what? And don't forget to choose a meaningful epsilon and minpts or you'll get useless clusters. $\endgroup$ – Anony-Mousse Jun 30 at 20:45
1
$\begingroup$

Bilgin!

Anony-Mousse puts right questions and gives good suggestions. Before you use the self-implemented DBSCAN code - write it on paper. Perhaps it is not the best algorithm at all for your database so try sci-kit learn implementation first to see the results.

Here are the Python implementation https://github.com/chrisjmccormick/dbscan/blob/master/dbscan.py and here is the theory https://github.com/chrisjmccormick/dbscan/blob/master/dbscan.py

Good luck!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.