I have a corpus of about one billion sentences, in which I am attempting to resolve NER conflicts (when two terms overlap in a sentence). My initial plan is to have an SME label the correct tag in each of a large number of conflicts, then use those labels to train either an NER model or a binary classification model (like GAN-ALBERT), to identify the correct choice when two NER tags conflict.
The problem is, about 5% of these sentences contain conflicts, and I don't think that I have the computational resources to run BERT or ALBERT prediction on 50 million sentences in a reasonable amount of time. So, my hope is to use the ALBERT model to generate a large number of labels (perhaps one million) for a computationally cheaper model.
So, I'm wondering if there is a model, 10 to 100 times cheaper at prediction than BERT, that could be trained to do a reasonable job of replicating the ALBERT model's performance, given a large amount of training data generated by said model.