I had used TF-IDF for text similarity but the results were not so good. I tried to implement google universal encoding (tensorflow hub). The results were satisfactory but not upto the mark.

Is there any other alternative approach?

Size of each text is around 50-70 words.

P.S - TF-IDF results were much better than Doc2Vec.

Edit 1: When I say "not upto the mark" it means the semantic similarity of the two sentence. Two sentences which are similar in terms of there meaning, although it has very few similar exacts words.

  • $\begingroup$ Please provide more information. What does "up to the mark" mean exactly? Do you have, in total, 50-70 words with which to train? $\endgroup$ – n1k31t4 Sep 10 '18 at 16:34
  • $\begingroup$ @n1k31t4 updated. $\endgroup$ – Aman Dalmia Oct 10 '18 at 4:37

Universal encoding and TF-IDF are two different beasts. I assume you mean the Vector Space Model transformed by TF-IDF. Either way: Neither tell you directly what the similarity of two texts are. Usually You'll use something like cosine distance to do that.

For the VSM there are scores of techniques to transform it. To name a few: Rocchio Transformation, LDA/LSI, stemming, stopping (or filtering on count and document count).

It sounds like you have a sparsity problem: Rocchio and LDA could help.And don't forget on some less elegant techniques in which you transform words using list of synonyms.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.