Let's say I have streaming textual data coming in. I run named entity recognition software to capture the entities, then I score them using tf-idf. Tf-idf scores are unbounded positively. How can I come up with an intelligible threshold (that stays current over time) to filter out the "low scoring" tf-idf values?

Currently, I'm keeping track of the last 1000 values, then taking the 0.99 quantile and only keeping those, but it has been too conservative. I can drop the quantile to 0.9, but I would like something more principled.




Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.