Real time topic identification of news article

Question

Let's say I'm constantly harvesting all the news article that are being published online (only having basic info about each one, eg. title, content, language, source (which news site)).

Let's say that I'd like to group together all the articles that are speaking about the same thing. However :

1. I want my algorithm to define the topics by itself (in opposition to "the user tells the existing topics and the algorithm assigns each article to the topics"),
2. I can't know the precise number topics there are (because, obviously, a new one has to be created each time something new happens),
3. and, as we are talking about news article, the list of topics should be expanding in real time if something new happens and new articles talk about it.

For simplicity's sake, let's assume all the articles are in the same language.

I've been using TF-IDF for this as of now, because that's what I came up with on the top of my head when I knew nothing about topic modeling, but it's not really good at its job. I'm getting into topic modeling and discovering things such as Latent Dirichlet Allocation, Correlated Topic Model, graph of words etc. I'm currently reading through all I can on the subject, but I'm probably unaware of interesting algorithms that could fit my need.

I'm interested : what approach would you take to tackle my problem ?

Thanks

Answer 1

You are looking for either "online" or "streaming" topic modeling. A hierarchical Dirichlet processes can automatically choose the number of topics. Here is a paper on streaming/online Latent Dirichlet Allocation model (LDA). A Python implementation of streaming/online can be found here. If your scale warrants it, Apache Spark has an implementation of online LDA also.