1
$\begingroup$

I am working on evaluating when a pair of string objects can be considered equal (e.g. given that we are talking about journals, is "international journal of air and water pollution" the same of "air and water pollution"?) and I was wondering what is the proper corpus to use to build a TF IDF vectorizer.

I am currently using as corpus all the distinct values of these strings belonging to the same type (in this example all the strings referring to journals). Is this a valid approach? Why? What could be other valid approaches?

$\endgroup$
3
  • $\begingroup$ Think about changing the title of the question to be about the end goal (i.e., comparing the similarity of two strings). Currently, it is just about one step in the process. $\endgroup$ – Brian Spiering Sep 8 '17 at 15:00
  • $\begingroup$ @BrianSpiering thank you for your feedback, followed your suggestion. $\endgroup$ – datapug Sep 8 '17 at 15:04
  • $\begingroup$ tf-idf is not the best approach. It will only capture the relative importance of specific words. Then the string comparison will only be counting the number of shared keywords, without regard for phrase meaning. If you switch to a word embedding approach, you can capture more shared meaning in the both strings. $\endgroup$ – Brian Spiering Sep 8 '17 at 16:38
1
$\begingroup$

Word Mover’s Distance (WMD) is an algorithm for finding the distance between pairs of strings. It is based on word embeddings (e.g., word2vec) which encoded the semantic meaning of words into dense vectors.

The WMD distance measures the dissimilarity between two text documents as the minimum amount of distance that the embedded words of one document need to "travel" to reach the embedded words of another document.

For example:

enter image description here Source: "From Word Embeddings To Document Distances" Paper

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.