Confusion with cosine similarity

Question

In information retrieval when we calculate the cosine similarity between the query features vector and the document features vector we penalize the unseen words in the query.

Example if we have two documents with features vectors

d1 = [1,1,1,0,0]  
d2 = [0,1,1,1,0]  

We can see that the two documents have the second feature so if we want to search for the second feature with query vector: q = [0,1,0,0,0] then the cosine similarity between q and d1,d2 will be $1/√3$, and not 1 because that we penalize the other features that we have not mention in the query.

From this discussion I don't understand why penalize it is a good Idea.

1. Is penalizing unseen features good?
2. Is there another similarity measure that does not penalize them?

Answer 1

Of course cosine similarity is not proper for searching a specific features in documents! To do this, you can exactly using dot product, as it will ignore zero features in query vector from documents.

Cosine similarity, in the current context, can be used to finding similarity between two documents. So, all features can be important to finding similarity. It means, if there is not a feature in query vector, but there is in a document, or vice versa, these two are 100 percent similar. Hence, it makes sense.

Answer 2

The point of your example seems to be more one of length normalization. Long story short:

1. rare terms are more informative than frequent words and and we do not want to weight all terms equally,
2. the more frequent a term, the more likely a document carries information about a term.

This video provides examples demonstrating why considering all the words in the text are important, including examples particular to the point you make in your text

To get an intuitive feel for why your search might be a problem, consider the fact that under your approach, all queries will be extremely similar to an English dictionary