2
$\begingroup$

I had used TF-IDF for text similarity but the results were not so good. I tried to implement google universal encoding (tensorflow hub). The results were satisfactory but not upto the mark.

Is there any other alternative approach?

Size of each text is around 50-70 words.

P.S - TF-IDF results were much better than Doc2Vec.

Edit 1: When I say "not upto the mark" it means the semantic similarity of the two sentence. Two sentences which are similar in terms of there meaning, although it has very few similar exacts words.

Improve this question
$\endgroup$
2
  • $\begingroup$ Please provide more information. What does "up to the mark" mean exactly? Do you have, in total, 50-70 words with which to train? $\endgroup$
    – n1k31t4
    Sep 10, 2018 at 16:34
  • $\begingroup$ @n1k31t4 updated. $\endgroup$
    – Aman Dalmia
    Oct 10, 2018 at 4:37

1 Answer 1

Reset to default
1
$\begingroup$

Universal encoding and TF-IDF are two different beasts. I assume you mean the Vector Space Model transformed by TF-IDF. Either way: Neither tell you directly what the similarity of two texts are. Usually You'll use something like cosine distance to do that.

For the VSM there are scores of techniques to transform it. To name a few: Rocchio Transformation, LDA/LSI, stemming, stopping (or filtering on count and document count).

It sounds like you have a sparsity problem: Rocchio and LDA could help.And don't forget on some less elegant techniques in which you transform words using list of synonyms.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.