I'm studying a paper about Named Entity Recognition. The following is a part of the abstract:

To assess the robustness of NER systems, we propose an evaluation method that focuses on subsets of tokens that represent specific sources of errors: unknown words and label shift or ambiguity.

I don't know what "label shift" definition is. The paper doesn't explain it and I can not find anything that I could understand by Googling it.


1 Answer 1


Label shift is the opposite of a covariate shift.
In this case, the assumption is that even though the feature distribution remains the same, the Label distribution might changes.
e.g. Symptoms --> Diseases

It can be different for different country (based on medical education of the Country/Doctor)
It can change with time also based on advancement in Medical knowledge

Similar logic can be built for "Words --> Slang". It can change with time due to the acceptance of these words.

Read the references for a formal explanation.

Dive into Deep Learning
Detecting and Correcting for Label Shift with Black Box Predictors

  • $\begingroup$ Do these shifts affect so called class-distribution independent metrics such as TPR or FPR? $\endgroup$
    – ado sar
    Feb 23, 2023 at 18:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.