0
$\begingroup$

The Jaccard similarity of two documents A and B can be defined as the size of their intersection (how many tokens are in both docs) divided by the size of their union (total number of tokens found in either document):

$jaccard(A, B) = \frac{|A \bigcap B|}{ |A\bigcup B|}$

I have implemented a simple function that, given two texts A and B as input, uses a word2vec model to generate a set of the N most similar words to the words in document A(minus stopwords), and another set for those in B. I then calculate the Jaccard similarity to get an idea of how much "overlap" there is between these two word sets:

def jaccard_similarity(self, a, b, n=100):  
    a_tokens = word_tokenize(a)
    b_tokens = word_tokenize(b)

    a_tokens = filter_stopwords(a_tokens)
    b_tokens = filter_stopwords(b_tokens)

    simset_A = set()
    simset_B = set()

    for token in a_tokens:
        sim_words = self.w2v.most_similar(token, topn=n)
        for w in sim_words:
            simset_A.add(w)

    for token in b_tokens:
        sim_words = self.w2v.most_similar(token, topn=n)
        for w in sim_words:
            simset_B.add(w)

    ab_intersect = simset_A.intersection(simset_B)
    ab_union = simset_A.union(simset_B)

    return len(ab_intersect) / len(ab_union)

The problem is that this function does not normalize for document length.

For instance, suppose that $A \subset B$, that is, every token from document A is found in document B.

If $|A| = 10$ and $|B| = 10$ then the $jaccard = 1$. But if $|A| = 10$ and $|B| = 1000000$, then $\frac{|A \bigcap B|}{ |A\bigcup B|} = \frac{10 }{100010} = 0.0001$. If A is much smaller than B, then $|A \bigcap B|$ will always be much smaller than $|A\bigcup B|$, even if all elements of A are in B. This is a problem when I want to do something like comparing a search query to a document that is much longer than the query.

How can I normalize in relation to document length and dampen the effect of differences in document length on the score (focusing more on coverage of smaller set by larger)?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.