I am trying to build and NER model that can name entities in a "Job description."

The entities are:

  1. Mandatory skills (Must have skills like java, python, c++ etc.)
  2. Nicetohave skills (the candidate "may" or "may not" have this, either is fine.)
  3. Degree (bachelors, masters...)
  4. Certification
  5. Job title

I prepared the dataset, annotated them and trained a spacy V2.0 model. The model performs well (in the range of 70-90s) on all labels, except "nice to have." It is either misclassifying or not predicting at all. So I am trying to catch the sentences at with nicetohave skills and "manually re-classify them, based on certain heuristics.

However, there are some complications within it. For eg:

Required skills: Knowledge in python, c++ Must have knowledge on matlab Nice to have skills: Knowledge in CAD is a plus Must have knowledge on CATIA V5

In the above example, skill from Required skills till the word matlab are mandatory and from nice to have: till CATIA V5 everything has to be nicetohave. There is also a chance that in some Job description the order may change (Nicetohave is mentioned first, followed by mandatory skills)

Or it is also possible that Mandatory skills are given first, followed by nice to have, followed by mandatory again. And many more combinations like that.

So how do I approach this scenario? I am open to any ideas, suggestions..


1 Answer 1


Assuming that there are almost always clear markers at the beginning of the enumeration, i.e. either "required skills" or "nice to have" (or any variant of these two), I would suggest trying to add custom features for example:

  • last marker seen from the current position, a categorical value for either "Nicetohave" or "mandatoryskill" (actually two boolean features with OHE)
  • distance in number of words from the current position to the last marker seen

Obtaining values for these features would require a preprocessing step where the markers are extracted and/or labelled, probably with some simple string matching (assuming that there are not too many variants of the markers).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.