When using cosine similarity in information retrieval, document vector length and query vector length are used for normalization. So if TF-IDF is used as a weighting function, then using raw frequency in TF is equal to normalizing it by document's length (mathematically speaking ).

My question is:

I am using log(TF*SomeValue + 1) as a local weight component instead of TF, where "SomeValue" is a calculated statistical property of the term. Using normalized TF in the formula gave me worse results than using raw frequency! So it's not the same in this case. How can I normalize term local weight to avoid bias toward lengthy documents?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.