1
$\begingroup$

I have a dataset with 3 columns:

  1. Text
  2. Meta-data (intending to extract features from it, then use those i.e., numerical features)
  3. Target label

Question 1: How can I use a pre-trained BERT instance on more than the text?

One theoretical solution suggests having BERT fed the text and another neural network with the numerical features fed into this one, then aggregating their output, into another neural network.

  • Is that the most efficient approach?

Question 2: How can you connect neural networks?

  • You get the output from each, but then what?

  • You get classification output from BERT, you get classification output from MLP based on numerical features.

  • You concatenate these and feed them to another MLP, and you get the final prediction? Wouldn't that last prediction be less robust ?

  • In other words, does the last MLP encapsulate the other 2 networks?

  • If so, what happens if BERT predicts on 90%, but the first MLP just 50%, will we get a lesser outcome?

Question 3: Any tips on how to implement this in pytorch?

$\endgroup$
1
$\begingroup$

When they are talking about aggregating their outputs, they mean the final embeddings (just before the classification layer) not the output of the network itself.

You take the embeddings from both of the networks and concatenate them vertically. This concatenated embedding you use to predict the final output you wanted.

Torch has a function torch.cat for this purpose.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.