Now I want to use my own domain data to train NLP model like BERT. The following is the details of my data:
- data length distribution: over 70% of my data has the length shorter than 5 and the largest length is 14;
- data format: the data is a list of number representing the AS-PATH of BGP announcement.
I've tried to use HuggingFace transformer package to define BERT model and train, but the MLM loss is quite high. I think this is the cause of data length.
So I want to know that is there any way to deal with these short data?