Compute Jaccard distance between two lists of strings

Question

I have input like below:

input_list = [['Search' 'engines','using','machine','learning','pattern','detections'], 

        ['machine','learning','helped','Google','automatically','sift','pages']]

input_list1 = ['Machine','learning','ever','evolving','technology']

Expecting Jaccard similarity distance between input_list and input_list1.

Answer 1

input_list = [['Search', 'engines','using','machine','learning','pattern','detections'], ['machine','learning','helped','Google','automatically','sift','pages']]
flat_list = [item for sublist in input_list for item in sublist]
input_list1 = ['Machine','learning','ever','evolving','technology']

def jaccard_similarity(list1, list2):
    intersection = len(list(set(list1).intersection(list2)))
    union = (len(list1) + len(list2)) - intersection
    return float(intersection) / union

jaccard_similarity(flat_list, input_list1)

Output:

0.05555555555555555

Answer 2

Python lib textdistance is a "python library for comparing distance between two or more sequences by many algorithms." It includes the Jaccard index. Having the similarity, you can get the distance by $Jacc_{distance}(x,y) = 1 -Jacc_{similarity}(x,y)$.

Although installing and importing the whole module wouldn't make much sense if this is all you need, reading the module's README may give you new ideas on the various distances you could use in your project, and take advantage of having all of them already implemented, so you can test how each of them performs really quick.

Answer 3

Since u have tokenized them, the jaccard distance is simply:

size(input_list intersect input_list1)/size(input_list union input_list1)

Reference