2
$\begingroup$

We have data of several news sites, having quite literally millions of entries. As each news site publishes their own version of the news (also each news site may publish several different version of the same news), we have several entries that are variants of a single news. I am currently working on separating out "Unique" news from our repository. That means if a single news has several variants, only a single variant (most likely the one reported earliest) will be considered.

I believe, clustering of the news articles can be used to group together similar news. I am currently exploring DBSCAN, and Hierarchical clustering (Ward's Method). I am wondering whether am I moving in the right direction, is clustering the best solution for solving our problem? If yes, which other algorithms and techniques should I explore?

$\endgroup$
2
$\begingroup$

You don't want clustering.

What you are looking for is near duplicate detection.

Use minhash. Apparently that is what Google News uses for exactly this purpose.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks a lot, exactly the thing I was looking for. Will surely explore it. $\endgroup$ – user3422929 Apr 10 '18 at 7:35
0
$\begingroup$

I am working in the same topic right now. I am using the following algorithm:

1) extract plain content from the news, for example using dragnet.

2) tokenize each text and represent them with vectors with the bag of words technique. A simple way to perform this is using TfidfVectorizer from sklearn.

3) Clusterize them using some classification technique like k-NN(k nearest neighbors). You will find the k-NN sklearn implementation very helpfull.

The key to perform the task is using the TfidfVectorizer which weigths more the tokens that only appear in a few notices, and so I can recognize similar news that talk about same topics.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.