1
$\begingroup$

I want to compare one sentence to some other sentences using the Bag of Words model. Suppose that my comparing sentence is:

I am playing football

and there are three more sentences that I want to compare my comparing sentence with. They are:

1. and I am playing Cricket

2. Why do you play Cricket

3. I love playing Cricket when I am at school

Now, if I compare my comparing sentence to the above three sentences by counting words, the number 1 and number 2 sentences have the same number of words that the comparing sentence has. and that is 3 (I, am , playing).

Now the question is, Which sentence is more related to my comparing sentence in this case? there are no semantic meanings involved at all.

In some places I saw, they say, it is less convoluted to return the shortest sentence in this case. What are your thoughts?

$\endgroup$

1 Answer 1

1
$\begingroup$

This is usually done by carefully choosing two things:

  • The sentence representation. Word count is the most simple option but there can be many others: TFIDF weights, with/without removing stop words, with/without lemmatization, etc. In a DL approach the sentence would be represented as a sentence embedding.
  • The similarity measure between two sentences. Again there are many options, in BoW approaches the standard ones would include counting words in common (e.g. Jaccard) and cosine TFIDF.

So the answer is: it depends on the similarity score. A complex similarity score like cosine TFIDF rarely produces ties so the highest score can be selected. More simple methods give ties, and then the logical answer is to return all the tied sentences.

$\endgroup$
4
  • $\begingroup$ I wrote it in the question, that there are no semantic meaning involved at all. All I want to know is that in case of multiple matches in two sentences, for example sentence1 has two words and sentence2 has also two words. both these words are the same. Now which one should I return as the best match? The shortest one? They say, there are benefits returning the shortest one, but what benefits? any resource for this reason? $\endgroup$
    – Hefaz
    Dec 24, 2020 at 15:31
  • $\begingroup$ @Hefaz there's no general reason to return the shortest one, it's just an arbitrary way to choose and it doesn't even work if the two candidates have the same length. You say that there's no semantic meaning involved, but you didn't say on what the similarity should be based. You should probably start by defining precisely what kind of similarity you want, then based on this definition define a similarity measure. You also assume that only one sentence should be returned but you didn't explain why: in most use cases in information retrieval the system returns a list of top similar sentences. $\endgroup$
    – Erwan
    Dec 24, 2020 at 15:58
  • $\begingroup$ Well, I compare them words by words, and words matched in any sentence could be a measure to this. for example the words "I", "am" and "playing" exist in both sentence 1 and sentence 3. but since the length of sentence 1 is smaller, that should be returned as the result. $\endgroup$
    – Hefaz
    Dec 24, 2020 at 17:27
  • $\begingroup$ @Hefaz I meant: what is the goal? Usually the goal of finding the most similar sentence based on words is to find the one semantically most related, but you said that semantics is not your goal. And you didn't explain why it's important for you to return a single sentence: usually when there are several good answers the system should return all of them. $\endgroup$
    – Erwan
    Dec 24, 2020 at 17:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.