Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I want to use Bert for an nlp task. But I also have additional features that I would like to include.
From what I have seen, with fine tuning, one only changes the labels and retrains the classification layer.
Is there a way to used pre-trained Bert models and include additional features?
To add additional features using BERT, one way is to use the existing WordPiece vocab and run pre-training for more steps on the additional data, and it should learn the compositionality. The WordPiece vocabulary can be basically used to create additional features that didn't already exist before.
Another approach to include additional features would be to add more vocab while training. Following approaches are possible:
"[unusedX]" tokens with your vocabulary. Since these were not used they are effectively randomly initialized. vocab_size parameter in bert_config.json. Later, write a script which generates a new checkpoint that is identical to the pre-trained checkpoint, but with a bigger vocab where the new embeddings are randomly initialized (for initialized we used tf.truncated_normal_initializer(stddev=0.02)). This will likely require mucking around with some tf.concat() and tf.assign() calls.Please note that I haven't tried any of these approaches myself.
