I want to use a pre-trained natural-language-inference model as a zero-shot text classifier, using the transformers package.

I have a set of topics, 'technology', 'science', 'politics', 'religion'. Each document in my corpus could be about multiple of these topics (multi-label). I now have the output, scores for each topic for each document.

I would now like to set a threshold on these scores, so that I can say whether each document is or is not about each of the topics. This will help to make the output easier to understand and allow me to calculate statistics about the number of documents on each topic. How should I set this threshold? Should the threshold be the same for each topic?

  • $\begingroup$ this threshold can be set empirically (for example from a subset of your documents) and it can be different for each topic, although with care. For example 0.5 is a reasonable threshold, but some topics may be so frequent or all-pervasive that a larger threshold is needed in order to distinguish occasional references to this topic from actual full-blown intent of this topic $\endgroup$
    – Nikos M.
    Oct 23 at 17:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.