Topic modelling on long documents: intra document clustering first

Question

I have a collection (around 1000) of very noisy, similar documents, that are each very long (>10 pages - 600 paragraphs) with multiple subsections - I want to perform topic modelling across the documents to discover the key themes.

I feel like I need to think carefully about how to treat the documents, but am struggling to find resources/papers. Does the following approach seem sensible, and are there any papers/sources that might be of help?:

1. Iterate through each document
2. Identify paragraphs
3. Keep only the longest paragraphs (say only the top quartile)
4. Usual NLP pre-processing (stop words, tokenize etc)
5. Embed each para as tf-idf vector
6. KMeans cluster across all paragraphs
7. use the clustered paragraphs as new documents within the overall corpus

So this way, the final corpus will be made of up of documents that are clustered paragraphs from each original document. I can then perform topic modelling.

Some initial clustering has yielded poor results (low silhouette score). Are there any approaches that might help? Or anything that I may have missed?

Answer 1

K means has certain assumption which leads to high bias due to hard assignments. In case of topic modelling documents may have overlapping classes i.e. a document may have two topics in it.

I would suggest you to try Non Negative Matrix factorisation over K means to check the results.

Answer 2

I suggest to have a look at bertopic (with default umap + hdbscan ) and dense embeddings from a Bert-like model. You best take a word embedding model with a big number of tokens like a Longformer with ~4000 input tokens which you then wrap in a sentence transformer. Potentially you can make use of an Embedder with 100000 input tokens so that all your documents fit into the input of your Embedder completely.