How can I add custom numerical features for training to BERT fine tuning?

Question

I have currently fine tuned the BERT model on some custom data and I want to conduct some more experiments to increase the accuracy.

My original dataset consists of a pair of sentences (like MRPC dataset). I want to increase the accuracy of classification by adding some numerical features (which I will separately calculate). I wanted to know if I could train bert using this after I have already fine tuned it ?

I have read about some solutions people have proposed in the past like : 'Extracting word embeddings (extract_features.py on Bert GITHUB) and combining that with my custom data to feed a single layer CNN network.' I dont want to lose out on the accuracy the BERT network is providing me by only extracting features from the pretrained model.

So is there any way I can create a kind of hybrid model which first fine tunes BERT and then I add my features and it feeds into another model for an improved classification ?

P.S

As I am new to tensorflow and Deep learning, please let me know if there is something fundamentally wrong in my understanding. Thank you for your help

Answer 1

With BERT I am assuming you are using finally the embeddings for your task.

Solution 1: Once you have embeddings, you can use them as features and with your other features and then build a new model for the task.

Solution 2: Here you will play with the network.

Now here left one is the normal BERT, in the right we have another MLP network to deal with other numeric features, on top we are combining embeddings from both and then passing through one more MLP to give the final output. Here the advantage is, while backpropagation, the BERT weight updates will have some contribution because of the loss due to numeric features.

As you mentioned you are new to TF and Bert, refer https://towardsdatascience.com/bert-in-keras-with-tensorflow-hub-76bcbc9417b to see how BERT can be implemented, and with this only doing the tweak to create suggested network will be easy. Feel free to reach out to me in case of any further questions.