Let's say I'm constantly harvesting all the news article that are being published online (only having basic info about each one, eg. title, content, language, source (which news site)).
Let's say that I'd like to group together all the articles that are speaking about the same thing. However :
- I want my algorithm to define the topics by itself (in opposition to "the user tells the existing topics and the algorithm assigns each article to the topics"),
- I can't know the precise number topics there are (because, obviously, a new one has to be created each time something new happens),
- and, as we are talking about news article, the list of topics should be expanding in real time if something new happens and new articles talk about it.
For simplicity's sake, let's assume all the articles are in the same language.
I've been using TF-IDF for this as of now, because that's what I came up with on the top of my head when I knew nothing about topic modeling, but it's not really good at its job. I'm getting into topic modeling and discovering things such as Latent Dirichlet Allocation, Correlated Topic Model, graph of words etc. I'm currently reading through all I can on the subject, but I'm probably unaware of interesting algorithms that could fit my need.
I'm interested : what approach would you take to tackle my problem ?