1
$\begingroup$

Let's say I have streaming textual data coming in. I run named entity recognition software to capture the entities, then I score them using tf-idf. Tf-idf scores are unbounded positively. How can I come up with an intelligible threshold (that stays current over time) to filter out the "low scoring" tf-idf values?

Currently, I'm keeping track of the last 1000 values, then taking the 0.99 quantile and only keeping those, but it has been too conservative. I can drop the quantile to 0.9, but I would like something more principled.

Thanks!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.