I have a collection (around 1000) of very noisy, similar documents, that are each very long (>10 pages - 600 paragraphs) with multiple subsections - I want to perform topic modelling across the documents to discover the key themes.
I feel like I need to think carefully about how to treat the documents, but am struggling to find resources/papers. Does the following approach seem sensible, and are there any papers/sources that might be of help?:
- Iterate through each document
- Identify paragraphs
- Keep only the longest paragraphs (say only the top quartile)
- Usual NLP pre-processing (stop words, tokenize etc)
- Embed each para as tf-idf vector
- KMeans cluster across all paragraphs
- use the clustered paragraphs as new documents within the overall corpus
So this way, the final corpus will be made of up of documents that are clustered paragraphs from each original document. I can then perform topic modelling.
Some initial clustering has yielded poor results (low silhouette score). Are there any approaches that might help? Or anything that I may have missed?