2
$\begingroup$

I have close to 50000 documents in plain text format.

Is there a way in which I can group similar documents together? Similarity mostly here is the content similarity.

Will transforming the text into a vector (using TFIDF) and running a K-Means (unsupervised learning) algorithm on top of that help? Are there any better approaches that could be used?

Improve this question
$\endgroup$
1
  • $\begingroup$ If the answers suits you don't forget to upvote and check. $\endgroup$
    – Carlos Mougan
    Jan 24, 2020 at 11:06

2 Answers 2

Reset to default
3
$\begingroup$

I did something similar a while ago. We wanted to classify several types of pdf.

  • We first extracted the text of the documents.
  • We created NLP features with the text
  • Then added pdf metadata: size of the file, number of pages, name of the document...
  • We then built a classification model with a few samples and did Active Learning

I guess that you could also do unsupervised learning but I like it more when you can do supervised learning.

Improve this answer
$\endgroup$
0
$\begingroup$

A common approach for this is LDA (Latent Dirichlet Allocation), which not only gives you the groups, but also a way to identify the topics of the groups by giving you the most common or distinctive words for each topic.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.