1
$\begingroup$

I have a corpus of about one billion sentences, in which I am attempting to resolve NER conflicts (when two terms overlap in a sentence). My initial plan is to have an SME label the correct tag in each of a large number of conflicts, then use those labels to train either an NER model or a binary classification model (like GAN-ALBERT), to identify the correct choice when two NER tags conflict.

The problem is, about 5% of these sentences contain conflicts, and I don't think that I have the computational resources to run BERT or ALBERT prediction on 50 million sentences in a reasonable amount of time. So, my hope is to use the ALBERT model to generate a large number of labels (perhaps one million) for a computationally cheaper model.

So, I'm wondering if there is a model, 10 to 100 times cheaper at prediction than BERT, that could be trained to do a reasonable job of replicating the ALBERT model's performance, given a large amount of training data generated by said model.

$\endgroup$

1 Answer 1

1
$\begingroup$

There are several smaller BERT models, including bert-tiny. Bert-tiny is a distillation of the full BERT model.

$\endgroup$
1
  • $\begingroup$ Just what I needed, thank you! $\endgroup$
    – Zorgoth
    Oct 24, 2022 at 14:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.