I have a data set containing 250k documents and 400k labeled topics. Each document may have 200+ topics. These topics may significantly overlap in subject and create noise. I want to reduce the documents to a significantly distinct topics.

As an example. I have a document which contains these topics:

Viral respiratory tract infections
COVID-19 pandemic
Respiratory diseases
Severe acute respiratory syndrome coronavirus 2
COVID-19 pandemic in Scotland
COVID-19 pandemic in the United Kingdom
COVID-19 pandemic in England

This level a granularity and overlap of topics isn't helpful to me. But manually sifting through hundreds of k's of topics myself and making a subjective decision how they should be grouped isn't an option and isn't scalable as I'm adding new data every day.

Is there a method for identifying topics that appear together so often that they are essentially duplicates?

  • $\begingroup$ At which level would you like to detect duplicates? At document level? At paragraph level? All data? A good solution could be a hierarchical duplicate detection to avoid too many calculations: You detect duplicates in every paragraph, then you compare them in their document, and finally you compare most frequent duplicates between documents. $\endgroup$ Aug 23 at 15:30
  • $\begingroup$ I'm trying to detect duplicates across all documents. So if in 95% of cases, anytime COVID-19 pandemic is identified, Viral respiratory tract infections is also identified, i want to consider these the same thing and combine them into a meta topic. $\endgroup$
    – micah
    Aug 23 at 16:15
  • $\begingroup$ If you detect duplicates among thousands of documents, the processing time would be too long. A good approach could be to start with 5% of random phrases from all documents and check duplicates among them. Perhaps it could be enough for your business. If not, see how to improve processing time with 5% (multi threading, efficient algorithms, etc.) and then progressively increase the quantity to 100%. $\endgroup$ Aug 24 at 7:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.