I have a conceptual problem that is related to a project I'm working on. I'm relatively new to the domain of NLP so this might be a poor question but I would really appreciate any help.
My dataset is web scraped news articles in a specific domain for the last 3 years and is updated with new articles every few days. The problem at hand is identifying trends. Here's my general approach so far:
- Preprocess the text
- Run an LDA with genism to cluster and 'identify' topics
- Find what topic each document relates to the most
- Group these documents by topics
- Use a moving average algorithm using the published date to see spikes in trends
What I'm trying to achieve: After every new update to the dataset, the new documents are classified into their respective topics. This way, I have a method to see what topics are trending.
The problem(s):
- LDA works on a whole set of documents. For a single new article, I don't know how to identify the topic. However, I might be able to figure out a workaround for this.
- This is the bigger problem: I'm using a pre-determined number of topics. This approach fails if there's a new news story that belongs to a completely different topic. I don't have a way of accommodating this.
I know that's a lot of information with very little context, but I've tried to explain to the best of my abilities. Is there a way I could update my LDA model to accommodate new topics? Or is there a different method I should be following altogether? Any and all suggestions are very very helpful.
I could have provided my code samples, but I don't know if they will be useful. I'll be happy to provide them as needed.