0
$\begingroup$

I need to know how to avoid spam document(file with repeated keywords) weighting while ranking the top k documents.

$\endgroup$
1
$\begingroup$

This type of spam is called keyword stuffing and it is a widely used SEO technique. There might be different approaches to deal with this.
One is to use pre-trained classifier that will provide a "spam" score to each document. One such example is Waterloo spam classifier, covered in paper:

"Efficient and Effective Spam Filtering and Re-ranking for Large Web Datasets"

, by: Gordon V. Cormack, Mark D. Smucker and Charles L. A. Clarke.
In addition, several predictors of document (specifically, Web document) quality were proposed. One of the most effective ones is the entropy of the unigram language model of a document. Specifically, when there is a large amount of repeated keywords in a document, the entropy of the language model is unusually low, since these keywords have unusually high probabilities. Another signal of document quality is the percent of stopwords in it, since natural language has more stopwords than a spam document.
More on Web documents quality measures can be found in: "Quality-Biased Ranking of Web Documents", by Michael Bendersky, W. Bruce Croft,Yanlei Diao.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.