Now I want to use my own domain data to train NLP model like BERT. The following is the details of my data:

  1. data length distribution: over 70% of my data has the length shorter than 5 and the largest length is 14;
  2. data format: the data is a list of number representing the AS-PATH of BGP announcement.

I've tried to use HuggingFace transformer package to define BERT model and train, but the MLM loss is quite high. I think this is the cause of data length.

So I want to know that is there any way to deal with these short data?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.