2
$\begingroup$

We have data of several news sites, having quite literally millions of entries. As each news site publishes their own version of the news (also each news site may publish several different version of the same news), we have several entries that are variants of a single news. I am currently working on separating out "Unique" news from our repository. That means if a single news has several variants, only a single variant (most likely the one reported earliest) will be considered.

I believe, clustering of the news articles can be used to group together similar news. I am currently exploring DBSCAN, and Hierarchical clustering (Ward's Method). I am wondering whether am I moving in the right direction, is clustering the best solution for solving our problem? If yes, which other algorithms and techniques should I explore?

Improve this question
$\endgroup$
0

2 Answers 2

Reset to default
2
$\begingroup$

You don't want clustering.

What you are looking for is near duplicate detection.

Use minhash. Apparently that is what Google News uses for exactly this purpose.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks a lot, exactly the thing I was looking for. Will surely explore it. $\endgroup$
    – user3422929
    Commented Apr 10, 2018 at 7:35
0
$\begingroup$

I am working in the same topic right now. I am using the following algorithm:

1) extract plain content from the news, for example using dragnet.

2) tokenize each text and represent them with vectors with the bag of words technique. A simple way to perform this is using TfidfVectorizer from sklearn.

3) Clusterize them using some classification technique like k-NN(k nearest neighbors). You will find the k-NN sklearn implementation very helpfull.

The key to perform the task is using the TfidfVectorizer which weigths more the tokens that only appear in a few notices, and so I can recognize similar news that talk about same topics.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.