Cosine similarity between two folders (1 and 2) with documents, and find the most relevant set of documents (in folder 2) for each doc (in folder 2)

Question

I have one folder named iir, it has 500 txt files. I have another json file named video (with dictionary structure).

I wish to compute: for each of the 500 txt files, find the cosine similarity with all the videos. After this rank the videos for each txt files and save that ranking list in another file.

So far, I am able to read all the txt files. This is the code:

import gensim
import glob
# go to directory
all_files = glob.glob("ExtratingConceptFromVideoScripts/iir/*.txt")
# go to files
indata = tuple()
for filename in all_files:
    with open(filename,'r')as file:
        for line in file.readlines():
            d = line.strip().split( ',' )
            indata = indata +  (d[0], )
print(indata[0])  # print the contents of first file

This is the output:

Online edition (c)2009 Cambridge UP An Introduction to Information Retrieval Draft of April 1       

Now, I read the contents of the json file:

import gensim
import glob
# go to directory
all_files = glob.glob('ExtratingConceptFromVideoScripts/data/corpus.json')
# go to files
vid = tuple()
for filename in all_files:
    with open(filename,'r')as file:
        for line in file.readlines():
            d = line.strip().split( ',' )
            vid = vid +  (d[:8], )
print(vid)  

Output:

(['{"0": {"metadata": {"id": "fQ3JoXLXxc4"', ' "title": "| Board Questions | 12 Maths | Equivalence Class | Equivalence Class Board Questions |"', ' "tags": ["Board Questions"', ' "12 maths"', ' "12 maths Board Questions"', ' "Previous Year Board Questions"', ' "Maths Board Questions"', ' "Board questions based on Equivalence Classes"'],)

Now, my problem is: the cosine similarity matrix finds the cosine similarity between the set of documents in one tuple (ie. it finds the cos sim between all files in doc iir). How to compute the cos sim of each iir doc with each doc in the vid.

I tried to convert the vid json file to tuple, and then compute the cos sim between iir and vid, but it gives an error.

Can anyone help me with explaining how to find cos sim between files with different formats, how to loop through files and compute cos sim, how to rank docs on the basis of highest cos sim.

Answer 1

Similarity is computed in a vector space model with a primary assumption that the pair of objects for which you want to compute similarity must be represented in the same vector space using some pre-processing operation.

Second dataset is a JSON. You'll need to extract text from it. One way to do it: you can use title and tags strings for each video, concatenate it to a string to find string representation for each video. Use any text vectorization method post that.

So get the video metadata into a text and then you can compute similarity.