1
$\begingroup$

I have a conceptual problem that is related to a project I'm working on. I'm relatively new to the domain of NLP so this might be a poor question but I would really appreciate any help.

My dataset is web scraped news articles in a specific domain for the last 3 years and is updated with new articles every few days. The problem at hand is identifying trends. Here's my general approach so far:

  1. Preprocess the text
  2. Run an LDA with genism to cluster and 'identify' topics
  3. Find what topic each document relates to the most
  4. Group these documents by topics
  5. Use a moving average algorithm using the published date to see spikes in trends

What I'm trying to achieve: After every new update to the dataset, the new documents are classified into their respective topics. This way, I have a method to see what topics are trending.

The problem(s):

  1. LDA works on a whole set of documents. For a single new article, I don't know how to identify the topic. However, I might be able to figure out a workaround for this.
  2. This is the bigger problem: I'm using a pre-determined number of topics. This approach fails if there's a new news story that belongs to a completely different topic. I don't have a way of accommodating this.

I know that's a lot of information with very little context, but I've tried to explain to the best of my abilities. Is there a way I could update my LDA model to accommodate new topics? Or is there a different method I should be following altogether? Any and all suggestions are very very helpful.

I could have provided my code samples, but I don't know if they will be useful. I'll be happy to provide them as needed.

Improve this question
$\endgroup$
2
  • $\begingroup$ You could probably try something like cosine similarity between vectors of new documents and existing topic wise clustered vectors. If similarity is high with any of the topic you could assign the document to that topic. If it is not similar to any of the topics you could probably consider it as a new topic. $\endgroup$
    – user16584277
    Oct 27, 2021 at 5:39
  • $\begingroup$ @user16584277 I thought about what you said, and it sounds like a reasonable approach. But then for the case of a new topic, adding one new topic to the document set would not result in very effective training for the LDA model. I guess a workaround could be grouping a few documents together before adding them to the dataset. $\endgroup$
    – Tanay Roman
    Oct 27, 2021 at 21:04

1 Answer 1

Reset to default
0
$\begingroup$

It is common for data to change over time. One common name is for this phenomena is dataset shift.

There are two steps when dataset shift is possible: detect and update.

Detection can be done manually or automatically. A person can inspect the new data and decide if a new topic is present. The process of detecting a new target can be also be automated. Latent Dirichlet allocation (LDA) is a probabilistic model. Thus if the probability of topic membership is below a threshold, that is evidence there is possibly a new topic.

After the detection of a new topic, the model needs to be updated. The update can be incremental retraining with the new data, frequently called online learning. Or the update can be a complete retraining of the model.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.