If I want to build a named entity linking system for resumes using an ontology of occupations and skills about how many annotations would I need? The ontology has about 20,000 entities.
As a lower bound I'm guessing I would need about 10 examples per entity and maybe 3 different annotators to label each mention so ~600K annotations. Does that make sense?
It depends on the signal-to-noise in the dataset. The amount of data to perform named entity disambiguation will depend on the tf-idf score of the occupation and skills, rare occupations and skills will require less data to build a performant model.
For example, that the sentence "I am a cook that multitasks well." "Cook" is an occupation and "multitask" is a relevant skill. In a similar sentence, "I multitasked while I cooked." "Cook" is no longer an occupation and "multitask" is no longer a relevant skill. However, the phrase "saturation diver" is less frequent than "cook", thus much easier to build a model to identify as an occupation and find relevant skills.
Annotator performance is easier to measure. Cohen's kappa is a common method of judging inter-rater reliability. Again, the number of needed raters depends on their agreement on the task. If task performance is easy, the number of raters and the number of items per rater can be lower. It is best to benchmark your system and then decide how much data you need to raise the benchmark scores.
One way to automatically create ontologies from a text is the TextRank algorithm.
You could be describing a variation of named-entity recognition (NER). You have labels/ categories for tokens. Given a corpus (resumes), you want a NER tagger to classify tokens as belonging to one of the labels or not.
You need to create a training set of ground-truth / "gold" labels of tokens and labels. Since you are only dealing with nouns, you can run a standard Part-Of-Speech (POS) Tagger then only custom tag the noun phrases.
It best to take an active learning approach. Active learning makes tagging the training set part of the entire machine learning pipeline, thus greatly reducing the number of annotations. "Deep Active learning for named entity recognition" is the current state-of-the-art.
Once you have a set of labels you can train NER classifier. The common options are Stanford Named Entity Recognizer (NER) and spaCy NER. A detailed example for Stanford Core NLP can be found here.
