Given different long documents of the same type, e.g. certain type of report, I need to identify certain items within the report, such as certain item's amount, the name of the certain person etc. How should I frame this problem under nlp? And what are the general approaches? I think the key challenge here is the same type of information will be in different part of the document in different documents. And the documents are 30-40 pages long.
1 Answer
$\begingroup$
$\endgroup$
2
To me this looks like Named Entity Recognition (NER), more generally a sequence labeling problem. The typical approach is to train a custom NER model using a large sample of annotated data.
- Pro: this is a very well known problem, there are multiple libraries which implement this.
- Cons: based on the description of your problem it's not sure that annotating a large sample of documents is doable, especially if there are very few target terms.
-
$\begingroup$ Aha, that's what I didn't think about before! As my docs are long, there can be many person names but only 1 particular name can be the CEO etc. In this case, how can NER handles that? Or other techniques needed? $\endgroup$– WongMar 27 at 1:42
-
$\begingroup$ @Wong it's a matter of amount and quality of training data: if you can have many annotated examples of the CEO name and/or if there are clear clues in the context (e.g. there's almost always the same sequence like "the CEO is John Smith"), then it can work well. If on the contrary there are few examples and the patterns are diverse, the model will probably miss a lot of cases. $\endgroup$– ErwanMar 27 at 11:19