1
$\begingroup$

I am trying to match strings of words with a website that has bulletpoints whose text is most similar to it. The way I thought of doing it is to get all of the documents from each bulletpoint into one corpus per website, that I would like to match a string of words with, discard stop words, and then lemmatize everything. Then, for each string of text, I create a TF-IDF sparse matrix, with each row the text from a single bulletpoint from a single website, so that the matrix contains all the text from the bulletpoints from all the websites, as well as a row for the string of words I want to match.

How should I then decide which row my string of words is most similar to? Should I get the cosine similarity of every row with my string of words row and just take whatever one has the highest cosine similarity (I will have a way of identifying the row with the website it was scrapped from)? Or is there an actual formalized way to go about this once I have my sparse matrix?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

The problem you describe looks very close to standard information retrieval: given a predefined set of documents $D$ and an input string $s$, find the most similar document $d\in D$ to $s$ (alternatively find the top $n$ documents $d$ most similar to $s$).

The approach you describe is good, except that in general the input string $s$ is not part of the TFIDF matrix: indeed the full set of predefined documents is encoded as a TFIDF matrix, but then any input string $s$ is simply encoded using the same vocabulary and weights. The advantage is that you don't need to recompute the matrix for every different string $s$ (the matrix can be pre-computed and stored for efficiency reasons). There is no disadvantage because any word in $s$ which is not in the vocabulary cannot be used in the calculation of the similarity anyway.

Indeed the standard method for matching or ranking the documents with respect to $s$ is to calculate a similarity score (e.g. cosine) for every $d$ against $s$, and then pick the highest similarity score.

Improve this answer
$\endgroup$
12
  • $\begingroup$ Thank you. How do I encode the input string $s$? What are weights in a TF-IDF matrix? $\endgroup$
    – sangstar
    Jul 14, 2021 at 14:43
  • $\begingroup$ @sangstar if you're using python you could encode the set of documents with TfidfVectorizer using fit_transform. Then to encode a new string $s$ you would use transform, which applies the vocabulary and weights obtained before. $\endgroup$
    – Erwan
    Jul 14, 2021 at 15:11
  • $\begingroup$ The weights are the regular TFIDF values: for every word it's the product of its Term Frequency (TF) and Inverse Document Frequency (IDF) values. If you want you could implement your own functions to calculate the TFIDF matrix, it's not too complex and it's a good way to understand how TFIDF actually works. $\endgroup$
    – Erwan
    Jul 14, 2021 at 15:15
  • $\begingroup$ Ah, so I would obtain my weights with fit_transform and then can encode a new string with transform(s)? This will tell me which row had the highest similarity score? $\endgroup$
    – sangstar
    Jul 14, 2021 at 15:20
  • $\begingroup$ @sangstar No, this step only encodes $s$ using the same encoding as the matrix of documents in order to make them comparable. Once this is done you still have to calculate the cosine similarity (for instance) between the encoded $s$ and every document $d$ in the matrix. $\endgroup$
    – Erwan
    Jul 14, 2021 at 15:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.