I want to use Bert for an nlp task. But I also have additional features that I would like to include.

From what I have seen, with fine tuning, one only changes the labels and retrains the classification layer.

Is there a way to used pre-trained Bert models and include additional features?

  • 1
    $\begingroup$ Hars to say with no detail. BERT retraining often involves only the last layer, you could feed this last layer with both BERT previous layers and your new features. $\endgroup$ – Robin Mar 5 '19 at 14:38
  • $\begingroup$ @debzsud, Want to make that an answer so we can upvote it? $\endgroup$ – D.W. Mar 9 '19 at 23:59

To add additional features using BERT, one way is to use the existing WordPiece vocab and run pre-training for more steps on the additional data, and it should learn the compositionality. The WordPiece vocabulary can be basically used to create additional features that didn't already exist before.

Another approach to include additional features would be to add more vocab while training. Following approaches are possible:

  1. Just replace the "[unusedX]" tokens with your vocabulary. Since these were not used they are effectively randomly initialized.
  2. Append new vocabulary words to the end of the vocab file, and update the vocab_size parameter in bert_config.json. Later, write a script which generates a new checkpoint that is identical to the pre-trained checkpoint, but with a bigger vocab where the new embeddings are randomly initialized (for initialized we used tf.truncated_normal_initializer(stddev=0.02)). This will likely require mucking around with some tf.concat() and tf.assign() calls.

Please note that I haven't tried any of these approaches myself.

  • 1
    $\begingroup$ Could you elaborate on what needs to be changed with tf.concat and tf.assign calls and where? I'm looking to do the same as above. Also, as a side note, when increasing the vocab I'm seeing some variable names that appear to be new and seem to be related to the optimizer, and are absent in the original checkpoint. Do you know what these are? ('bert/encoder/layer_8/output/dense/kernel/adam_m', [3072, 768]), ('bert/encoder/layer_8/output/dense/kernel/adam_v', [3072, 768]), $\endgroup$ – D.S. Mar 1 '20 at 16:25
  • 1
    $\begingroup$ The answer is from this source: github.com/google-research/bert/issues/9 . But I wonder where to find this kind of script. $\endgroup$ – Pankaj Kumar Mar 3 '20 at 9:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.