# Comparing one small dataset with a big dataset for similar records

I create a varying small dataset (dataset: X) with 500 records in each query. Everytime I need to compare the dataset with a bigger one (dataset: A) (15 milion records) to find similar (or semi-silmilar) values from three different columns. The values are either one word or a sentences. My algorithms is like this:

1. make a vector of words in each record in both datasets
2. with a for loop, search for similarities over the big dataset (e.g. with tfidf). That means each record from the small dataset should look for possible similarities over the big dataset.

However, the problem is searching a big data is very slow. Is there any efficeint way to solve this problem? Thanks

• Store each document from A in an inverted index $$m$$, so that for any word $$w$$ $$m[w]$$ is the list of all documents in A which contain word $$w$$ (note that a document can appear several times in this data structure).
• When comparing a new query against $$A$$, instead of iterating through all documents in $$A$$ just compare against the subsets which have at least one word in common, i.e. $$m[w]$$ for every word $$w$$ in the query.