I have a collection (around 1000) of very noisy, similar documents, that are each very long (>10 pages - 600 paragraphs) with multiple subsections - I want to perform topic modelling across the documents to discover the key themes.

I feel like I need to think carefully about how to treat the documents, but am struggling to find resources/papers. Does the following approach seem sensible, and are there any papers/sources that might be of help?:

  1. Iterate through each document
  2. Identify paragraphs
  3. Keep only the longest paragraphs (say only the top quartile)
  4. Usual NLP pre-processing (stop words, tokenize etc)
  5. Embed each para as tf-idf vector
  6. KMeans cluster across all paragraphs
  7. use the clustered paragraphs as new documents within the overall corpus

So this way, the final corpus will be made of up of documents that are clustered paragraphs from each original document. I can then perform topic modelling.

Some initial clustering has yielded poor results (low silhouette score). Are there any approaches that might help? Or anything that I may have missed?


1 Answer 1


K means has certain assumption which leads to high bias due to hard assignments. In case of topic modelling documents may have overlapping classes i.e. a document may have two topics in it.

I would suggest you to try Non Negative Matrix factorisation over K means to check the results.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.