I'm more-or-less new to NLP so assume little existing knowledge! But I have strong coding skills in R and to a lesser extent Python.

We're interested in extracting key information about objectives, activities, and risks from the project documents of a few thousand education projects carried out by the World Bank. The documents are fairly structured with headings and then tables for all of the variables which we're interested in, but they go back to the 1990s and the actual format and appearance of the documents has changed a lot over time. We need the output for each variable to be fairly general (so objectives would only have a few possible categories, like "access", "learning", and so on).

Roughly what methods should we be using? Our original plan was to handlabel a subset of them, and use a supervised learning approach (perhaps text classification?) to automatically label the rest. But upon reading more this doesn't seem too well-suited. Thanks in advance!


1 Answer 1


I'd suggest starting not with manually labelling directly but with a manual case study: observe what are the clues that the system would use, how much diversity there is, if needed define exactly the different types of information that should be extracted, etc.

Based on the initial study, choose the method from basic regular expression search (if there's only a few clearly defined cases) to complex approaches, possibly involving text segmentation or named entity recognition. You should also take into account how robust the system needs to be and how much effort can be put into annotating data.

  • $\begingroup$ Thanks Erwan, this is helpful. So do you think it would still be possibly useful to use manual annotation (we do have capacity for this to some extent) and then using a supervised learning approach? If so is there a good reference for the particular method or algorithm we would use? Or more generally is there a good reference text to understand what would be the best method given the system clues, diversity etc? $\endgroup$
    – Rory
    Aug 30, 2022 at 10:03
  • $\begingroup$ @Rory the first thing to understand about annotation is that the rules to follow when annotating must be very clear and precise. If there's a lot of subjectivity, there's a risk of obtaining inconsistent or noisy data, leading to bad models. This is why I suggest starting with a preliminary study, it could help specifying the exact annotation process. Your task is probably too specific to find useful references for it (like most tasks in NLP), so you'll probably have to design most of it from scratch and by trial and error. At least I'm not aware of any ref for this part, but maybe there is. $\endgroup$
    – Erwan
    Aug 30, 2022 at 16:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.