1
$\begingroup$

I want to create sequence classification bert model. The input of model will be 2 sentence. But i want to fine tuning the model with large context data which consists of multiple sentences(which number of tokens could be exceed 512). Is it okay if the size of the training data and the size of the actual input data are different?

Thanks

$\endgroup$

1 Answer 1

1
$\begingroup$

There is a limiting factor here, which is the positional embeddings.

In BERT, positional embeddings are trainable (not sinusoidal) and support a maximum of 512 positions. To exceed such a sequence length, you would need to extend the positional embedding table and have the extra entries be trained during the fine-tuning. This, however, would probably lead to performance degradation. So, technically possible but probably not Ok.

One option would be to keep only the first (or the last) 512 tokens of the sequences as input to BERT and see if the resulting performance is fine for your purposes.

As an alternative, you may use pre-trained long-context transformers like the LongFormer.

$\endgroup$
4
  • $\begingroup$ Thanks a lot! Actually i was truncate the data when tokens exceed 512. What i confused is does the model works well even if the length of data is different for training and it's actual input. For example, Normally when creating sentence classification model, we usually training with 1 labeled sentence (suppose on average, 10 tokens constitute sentence) and actual input will be almost same size (10 tokens). But in my case, the training data will be around 512 tokens but the actual input will be 10~15 tokens. In this case, does BERT can capture similar aspects of input and trained data? $\endgroup$
    – yyouyki
    Mar 25 at 8:41
  • $\begingroup$ I think I don't understand what you mean by "training data" and "actual input" and why they are not the same concept $\endgroup$
    – noe
    Mar 25 at 9:37
  • $\begingroup$ @yyouyki can you clarify? $\endgroup$
    – noe
    Mar 28 at 12:04
  • $\begingroup$ Yeah Thanks! I misunderstood some concept about bert fine tuning. What i want to do was fine tuning with compresssed sentence which express all meanings of multiple sentence (for example, if paragraph is consist of sentence a, b, c, then i want to create [1,768] size embedding which combine those 3 sentences, not for each sentence, i know it's weird) and use it for fine tuning. I just thought that it doens't matter if i could convert input to same size of tensor that i use for fine tuning. $\endgroup$
    – yyouyki
    Mar 28 at 16:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.