I have a task to create a tool, which will be able to find articles-duplicates of a given reference article. I know word vectorization (tf-idf,word2vec), RNN methods, but i can not choose something suitable for my situation.
My requirments:
- data are being collected on the fly (program parses articles from web sites, so i don't have regular DB with collection of texts)
- there is a reference text, whose copies need to be found
- copies could be copypasted, partially copypasted (by paragraphs) or paraphrased
- reference-vs-copy comparison algorithm is preferable, but not required (instead of reference-vs-corps)
- algorithm shouldn't do deep semantic analyzis, only kind of word counting, word vectorization, substring search
- instead one algorithm, i can use a set of herurisitcs
- algorithms can do false positive dicisions
I come up with such ideas:
- download pretrained word2vec and compare means of word-vectors
- Build a dictionary word->count from every text and compare it to reference dictionary
- collect about 100 texts, vectorize them according to tf-idf and find closest to the reference
I will apreciate, if you will point specific algorithms, libs, examples based on key-word extractions, dummy substring search, line difference comparison for python or CLI.