I have a conceptual problem that is related to a project I'm working on. I'm relatively new to the domain of NLP so this might be a poor question but I would really appreciate any help.

My dataset is web scraped news articles in a specific domain for the last 3 years and is updated with new articles every few days. The problem at hand is identifying trends. Here's my general approach so far:

  1. Preprocess the text
  2. Run an LDA with genism to cluster and 'identify' topics
  3. Find what topic each document relates to the most
  4. Group these documents by topics
  5. Use a moving average algorithm using the published date to see spikes in trends

What I'm trying to achieve: After every new update to the dataset, the new documents are classified into their respective topics. This way, I have a method to see what topics are trending.

The problem(s):

  1. LDA works on a whole set of documents. For a single new article, I don't know how to identify the topic. However, I might be able to figure out a workaround for this.
  2. This is the bigger problem: I'm using a pre-determined number of topics. This approach fails if there's a new news story that belongs to a completely different topic. I don't have a way of accommodating this.

I know that's a lot of information with very little context, but I've tried to explain to the best of my abilities. Is there a way I could update my LDA model to accommodate new topics? Or is there a different method I should be following altogether? Any and all suggestions are very very helpful.

I could have provided my code samples, but I don't know if they will be useful. I'll be happy to provide them as needed.

  • $\begingroup$ You could probably try something like cosine similarity between vectors of new documents and existing topic wise clustered vectors. If similarity is high with any of the topic you could assign the document to that topic. If it is not similar to any of the topics you could probably consider it as a new topic. $\endgroup$ Oct 27 at 5:39
  • $\begingroup$ @user16584277 I thought about what you said, and it sounds like a reasonable approach. But then for the case of a new topic, adding one new topic to the document set would not result in very effective training for the LDA model. I guess a workaround could be grouping a few documents together before adding them to the dataset. $\endgroup$ Oct 27 at 21:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.