If I want to build a named entity linking system for resumes using an ontology of occupations and skills about how many annotations would I need? The ontology has about 20,000 entities.

As a lower bound I'm guessing I would need about 10 examples per entity and maybe 3 different annotators to label each mention so ~600K annotations. Does that make sense?

  • $\begingroup$ There is a lot of terminology in this question. Can you define them clearly or avoid using them where it is not necessary please. Entities are instances I presumed? But then you stated that you would need 10 examples per entity. So I'm utterly confused. What are annotations? What are entities? What is a mention? $\endgroup$ – JahKnows May 15 '18 at 4:11
  • $\begingroup$ I think it would be best to describe the data you have access to, and tell us what you hope to achieve. Then go into the details of your method using the jargon. $\endgroup$ – JahKnows May 15 '18 at 4:11
  • $\begingroup$ @JahKnows, see en.wikipedia.org/wiki/Named-entity_recognition for some background. A named entity is a phrase like "New York Times". A mention is an instance of that in a sentence. An annotation is something entered by a human to hand-label a sentence. $\endgroup$ – D.W. May 15 '18 at 23:05
  • $\begingroup$ Ontology of occupations and skills has been built many times. One of the most popular ones is onetcenter.org/database.html. Either just use that or use it to benchmark your method. $\endgroup$ – Brian Spiering May 16 '18 at 0:45
  • $\begingroup$ I don't want to build the ontology. I want to link mentions in a corpus to the ontology. $\endgroup$ – 2daaa May 16 '18 at 13:03

It depends on the signal-to-noise in the dataset. The amount of data to perform named entity disambiguation will depend on the tf-idf score of the occupation and skills, rare occupations and skills will require less data to build a performant model.

For example, that the sentence "I am a cook that multitasks well." "Cook" is an occupation and "multitask" is a relevant skill. In a similar sentence, "I multitasked while I cooked." "Cook" is no longer an occupation and "multitask" is no longer a relevant skill. However, the phrase "saturation diver" is less frequent than "cook", thus much easier to build a model to identify as an occupation and find relevant skills.

Annotator performance is easier to measure. Cohen's kappa is a common method of judging inter-rater reliability. Again, the number of needed raters depends on their agreement on the task. If task performance is easy, the number of raters and the number of items per rater can be lower. It is best to benchmark your system and then decide how much data you need to raise the benchmark scores.

One way to automatically create ontologies from a text is the TextRank algorithm.

|improve this answer|||||
  • $\begingroup$ I'm just trying to estimate a lower bound by thinking about it as a classification problem. Assuming I want to reliably link mentions to entities in my ontology with ~10^4 nodes what's the order of magnitude of the number of annotations I would need? Does ~10^5 make sense? ~10^6? $\endgroup$ – 2daaa May 16 '18 at 13:06

You could be describing a variation of named-entity recognition (NER). You have labels/ categories for tokens. Given a corpus (resumes), you want a NER tagger to classify tokens as belonging to one of the labels or not.

You need to create a training set of ground-truth / "gold" labels of tokens and labels. Since you are only dealing with nouns, you can run a standard Part-Of-Speech (POS) Tagger then only custom tag the noun phrases.

It best to take an active learning approach. Active learning makes tagging the training set part of the entire machine learning pipeline, thus greatly reducing the number of annotations. "Deep Active learning for named entity recognition" is the current state-of-the-art.

Once you have a set of labels you can train NER classifier. The common options are Stanford Named Entity Recognizer (NER) and spaCy NER. A detailed example for Stanford Core NLP can be found here.

|improve this answer|||||
  • $\begingroup$ Sure, thanks for the pointers but the question isn't how to train such a model. The question is how to estimate a reasonable lower bound for the number of annotations needed. I can find these links easily but what is not so readily available is practical knowledge about how to estimate the cost of building something useful for a business. $\endgroup$ – 2daaa May 19 '18 at 16:16
  • $\begingroup$ The short answer is "It depends". If you take an Agile or Active Learning approach, just start the task and see what which annotations are easy and which ones are hard. Based on empirical evidence, improve estimations. Making an a priori lower bound estimate on a research and development project will not be meaningful. Best of luck! $\endgroup$ – Brian Spiering May 19 '18 at 16:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.