I have currently fine tuned the BERT model on some custom data and I want to conduct some more experiments to increase the accuracy.

My original dataset consists of a pair of sentences (like MRPC dataset). I want to increase the accuracy of classification by adding some numerical features (which I will separately calculate). I wanted to know if I could train bert using this after I have already fine tuned it ?

I have read about some solutions people have proposed in the past like : 'Extracting word embeddings (extract_features.py on Bert GITHUB) and combining that with my custom data to feed a single layer CNN network.' I dont want to lose out on the accuracy the BERT network is providing me by only extracting features from the pretrained model.

So is there any way I can create a kind of hybrid model which first fine tunes BERT and then I add my features and it feeds into another model for an improved classification ?


As I am new to tensorflow and Deep learning, please let me know if there is something fundamentally wrong in my understanding. Thank you for your help

  • $\begingroup$ did you find out how to do this? I am also interested $\endgroup$ – echan00 Sep 12 '19 at 19:53

With BERT I am assuming you are using finally the embeddings for your task.

Solution 1: Once you have embeddings, you can use them as features and with your other features and then build a new model for the task.

Solution 2: Here you will play with the network.

Suggested Network

Now here left one is the normal BERT, in the right we have another MLP network to deal with other numeric features, on top we are combining embeddings from both and then passing through one more MLP to give the final output. Here the advantage is, while backpropagation, the BERT weight updates will have some contribution because of the loss due to numeric features.

As you mentioned you are new to TF and Bert, refer https://towardsdatascience.com/bert-in-keras-with-tensorflow-hub-76bcbc9417b to see how BERT can be implemented, and with this only doing the tweak to create suggested network will be easy. Feel free to reach out to me in case of any further questions.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.