# Identifying Mixed Label Duplicates In Large Data Set

I have a data set containing 250k documents and 400k labeled topics. Each document may have 200+ topics. These topics may significantly overlap in subject and create noise. I want to reduce the documents to a significantly distinct topics.

As an example. I have a document which contains these topics:

Viral respiratory tract infections
COVID-19 pandemic
Respiratory diseases
Severe acute respiratory syndrome coronavirus 2
COVID-19 pandemic in Scotland
COVID-19 pandemic in the United Kingdom
COVID-19 pandemic in England
Coronavirus


This level a granularity and overlap of topics isn't helpful to me. But manually sifting through hundreds of k's of topics myself and making a subjective decision how they should be grouped isn't an option and isn't scalable as I'm adding new data every day.

Is there a method for identifying topics that appear together so often that they are essentially duplicates?

• At which level would you like to detect duplicates? At document level? At paragraph level? All data? A good solution could be a hierarchical duplicate detection to avoid too many calculations: You detect duplicates in every paragraph, then you compare them in their document, and finally you compare most frequent duplicates between documents. Aug 23 at 15:30
• I'm trying to detect duplicates across all documents. So if in 95% of cases, anytime COVID-19 pandemic is identified, Viral respiratory tract infections is also identified, i want to consider these the same thing and combine them into a meta topic. Aug 23 at 16:15
• If you detect duplicates among thousands of documents, the processing time would be too long. A good approach could be to start with 5% of random phrases from all documents and check duplicates among them. Perhaps it could be enough for your business. If not, see how to improve processing time with 5% (multi threading, efficient algorithms, etc.) and then progressively increase the quantity to 100%. Aug 24 at 7:37