I'm working on detecting duplicate text documents using a classifier. I am looking for training data - a corpus of text documents and corresponding metadata which lists out pairs of duplicate documents from the corpus. Where can I find such data?
Instead of training a classifier, detecting duplicates may advantageously be done in a direct fashion: using a similarity metric like shingle signatures or a hashing/checksum function.
(A text to representation function F should preserve formal/meta elements that you already know to be identifiers or partial-identifiers, and it should also incorporate a compact representation of the content -- like sparse excerpts, checksum, or any other hashing key or canonized form. The comparison function G should then compare these text representations provided by F and define a similarity score.)
You'd still need an evaluation corpus, so looking for a corpus is still important. I don't know of any open one, but job-boards and academic databases contain lots of duplicates so you could start there.