0
$\begingroup$

I am building a named entity recognition model and it is having BERT+BiLSTM+CRF in it. Now I am planning to introduce an attention layer. My question is - what type of attention I should use here and why?

Data: It is an invoice data - I am trying to extract Company and person names from the invoices.

$\endgroup$
3
  • $\begingroup$ BERT is a Transformer encoder, which is specifically based on attention layers, so you already have attention there. $\endgroup$
    – noe
    Mar 4, 2021 at 13:39
  • $\begingroup$ So, according to you, we do not need any additional attention? $\endgroup$ Mar 5, 2021 at 4:49
  • $\begingroup$ Actually, using just BERT is the typical approach for NER. No need for BiLSTMs on top or CRFs. $\endgroup$
    – noe
    Mar 5, 2021 at 7:42

1 Answer 1

1
$\begingroup$

BERT is a Transformer encoder, which is specifically based on attention layers, so you already have attention there, no need for extra attention. Actually, using just BERT is the typical approach for NER. No need for BiLSTMs on top, or CRFs.

I suggest you start with standard approaches, like finetuning BERT. For that, there are libraries like tner, which is built on top of HuggingFace Transformers, that makes it really easy to finetune NER models.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.