0
$\begingroup$

Now I want to use my own domain data to train NLP model like BERT. The following is the details of my data:

  1. data length distribution: over 70% of my data has the length shorter than 5 and the largest length is 14;
  2. data format: the data is a list of number representing the AS-PATH of BGP announcement.

I've tried to use HuggingFace transformer package to define BERT model and train, but the MLM loss is quite high. I think this is the cause of data length.

So I want to know that is there any way to deal with these short data?

Improve this question
$\endgroup$

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.