Help to choose algorithm for computing difference between 2 texts?

Question

I have a task to create a tool, which will be able to find articles-duplicates of a given reference article. I know word vectorization (tf-idf,word2vec), RNN methods, but i can not choose something suitable for my situation.

My requirments:

• data are being collected on the fly (program parses articles from web sites, so i don't have regular DB with collection of texts)
• there is a reference text, whose copies need to be found
• copies could be copypasted, partially copypasted (by paragraphs) or paraphrased
• reference-vs-copy comparison algorithm is preferable, but not required (instead of reference-vs-corps)
• algorithm shouldn't do deep semantic analyzis, only kind of word counting, word vectorization, substring search
• instead one algorithm, i can use a set of herurisitcs
• algorithms can do false positive dicisions

I come up with such ideas:

1. download pretrained word2vec and compare means of word-vectors
2. Build a dictionary word->count from every text and compare it to reference dictionary
3. collect about 100 texts, vectorize them according to tf-idf and find closest to the reference

I will apreciate, if you will point specific algorithms, libs, examples based on key-word extractions, dummy substring search, line difference comparison for python or CLI.

Answer 1

algorithm shouldn't do deep semantic analyzis, only kind of word counting, word vectorization, substring search

For such tasks, test if built-in Elastic Search / SOLR models work well. These products have multiple models that can be tuned with configuration. With these models, you provide a document and find other documents that are "similar".

Some of the available algorithms are :

1. Cosine Similarity
2. Okapi_BM25
3. DFR
4. LM Jelinek Mercer

Docs :

https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-similarity.html

https://lucene.apache.org/solr/guide/6_6/morelikethis.html