# Cosine similarity between two folders (1 and 2) with documents, and find the most relevant set of documents (in folder 2) for each doc (in folder 2)

I have one folder named iir, it has 500 txt files. I have another json file named video (with dictionary structure).

I wish to compute: for each of the 500 txt files, find the cosine similarity with all the videos. After this rank the videos for each txt files and save that ranking list in another file.

So far, I am able to read all the txt files. This is the code:

import gensim
import glob
# go to directory
all_files = glob.glob("ExtratingConceptFromVideoScripts/iir/*.txt")
# go to files
indata = tuple()
for filename in all_files:
with open(filename,'r')as file:
d = line.strip().split( ',' )
indata = indata +  (d[0], )
print(indata[0])  # print the contents of first file


This is the output:

Online edition (c)2009 Cambridge UP An Introduction to Information Retrieval Draft of April 1


Now, I read the contents of the json file:

import gensim
import glob
# go to directory
all_files = glob.glob('ExtratingConceptFromVideoScripts/data/corpus.json')
# go to files
vid = tuple()
for filename in all_files:
with open(filename,'r')as file:
d = line.strip().split( ',' )
vid = vid +  (d[:8], )
print(vid)


Output:

(['{"0": {"metadata": {"id": "fQ3JoXLXxc4"', ' "title": "| Board Questions | 12 Maths | Equivalence Class | Equivalence Class Board Questions |"', ' "tags": ["Board Questions"', ' "12 maths"', ' "12 maths Board Questions"', ' "Previous Year Board Questions"', ' "Maths Board Questions"', ' "Board questions based on Equivalence Classes"'],)


Now, my problem is: the cosine similarity matrix finds the cosine similarity between the set of documents in one tuple (ie. it finds the cos sim between all files in doc iir). How to compute the cos sim of each iir doc with each doc in the vid.

I tried to convert the vid json file to tuple, and then compute the cos sim between iir and vid, but it gives an error.

Can anyone help me with explaining how to find cos sim between files with different formats, how to loop through files and compute cos sim, how to rank docs on the basis of highest cos sim.