I'm working on detecting duplicate text documents using a classifier. I am looking for training data - a corpus of text documents and corresponding metadata which lists out pairs of duplicate documents from the corpus. Where can I find such data?

  • 2
    $\begingroup$ i am also looking for the same dataset , share if you found any $\endgroup$ – shipika Jun 7 '18 at 6:10
  • $\begingroup$ I haven't found a dataset that I really like for this problem. I'm using the 20 Newsgroups data with some modifications. Not ideal. $\endgroup$ – Minu Jun 9 '18 at 13:30
  • $\begingroup$ @Minu Can you share how you are using 20 Newsgroups data? I do not find it helpful for this purpose. Any insight is very very appreciated. :) $\endgroup$ – Nabin Feb 17 '20 at 11:08

Instead of training a classifier, detecting duplicates may advantageously be done in a direct fashion: using a similarity metric like shingle signatures or a hashing/checksum function.

(A text to representation function F should preserve formal/meta elements that you already know to be identifiers or partial-identifiers, and it should also incorporate a compact representation of the content -- like sparse excerpts, checksum, or any other hashing key or canonized form. The comparison function G should then compare these text representations provided by F and define a similarity score.)

You'd still need an evaluation corpus, so looking for a corpus is still important. I don't know of any open one, but job-boards and academic databases contain lots of duplicates so you could start there.

  • $\begingroup$ I'm using a classifier on top of a similarity metric. I like to use the documents' metadata for duplicate detection and I'm using a classifier to make this distinction. $\endgroup$ – Minu Sep 10 '17 at 22:07
  • $\begingroup$ got it ! good luck for finding your corpora :) $\endgroup$ – rloth Sep 11 '17 at 13:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.