0
$\begingroup$

When using cosine similarity in information retrieval, document vector length and query vector length are used for normalization. So if TF-IDF is used as a weighting function, then using raw frequency in TF is equal to normalizing it by document's length (mathematically speaking ).

My question is:

I am using log(TF*SomeValue + 1) as a local weight component instead of TF, where "SomeValue" is a calculated statistical property of the term. Using normalized TF in the formula gave me worse results than using raw frequency! So it's not the same in this case. How can I normalize term local weight to avoid bias toward lengthy documents?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.