# Combining textual and numeric features into pre-trained Transformer BERT

I have a dataset with 3 columns:

1. Text
2. Meta-data (intending to extract features from it, then use those i.e., numerical features)
3. Target label

Question 1: How can I use a pre-trained BERT instance on more than the text?

One theoretical solution suggests having BERT fed the text and another neural network with the numerical features fed into this one, then aggregating their output, into another neural network.

• Is that the most efficient approach?

Question 2: How can you connect neural networks?

• You get the output from each, but then what?

• You get classification output from BERT, you get classification output from MLP based on numerical features.

• You concatenate these and feed them to another MLP, and you get the final prediction? Wouldn't that last prediction be less robust ?

• In other words, does the last MLP encapsulate the other 2 networks?

• If so, what happens if BERT predicts on 90%, but the first MLP just 50%, will we get a lesser outcome?

Question 3: Any tips on how to implement this in pytorch?

Torch has a function torch.cat for this purpose.