I have a data set containing 250k documents and 400k labeled topics. Each document may have 200+ topics. These topics may significantly overlap in subject and create noise. I want to reduce the documents to a significantly distinct topics.
As an example. I have a document which contains these topics:
Viral respiratory tract infections COVID-19 pandemic Respiratory diseases Severe acute respiratory syndrome coronavirus 2 COVID-19 pandemic in Scotland COVID-19 pandemic in the United Kingdom COVID-19 pandemic in England Coronavirus
This level a granularity and overlap of topics isn't helpful to me. But manually sifting through hundreds of k's of topics myself and making a subjective decision how they should be grouped isn't an option and isn't scalable as I'm adding new data every day.
Is there a method for identifying topics that appear together so often that they are essentially duplicates?