Let's say I have streaming textual data coming in. I run named entity recognition software to capture the entities, then I score them using tf-idf. Tf-idf scores are unbounded positively. How can I come up with an intelligible threshold (that stays current over time) to filter out the "low scoring" tf-idf values?
Currently, I'm keeping track of the last 1000 values, then taking the 0.99 quantile and only keeping those, but it has been too conservative. I can drop the quantile to 0.9, but I would like something more principled.
Thanks!
