I have a custom dataset with custom tag for each token in the text. I want to train a BERT model for classifying each token into its corresponding category. To do the same I used a pretrained BERT model as base and tried to train it for the mentioned custom task. But the results that I found were not up to the mark. The model predicted only those tokens that it had seen during its previous task it was trained on (like DATE, TIME, LOCATION, ORG, etc) which are totally different from my tokens. On further researching I found that, to train model on our custom tags, the model needs to be pre-trained on similar sentences(like how tequila bottles are rinsed with tequila before filling them with tequila). I found some articles which seemed to point that Masked Language Modelling(MLM) is a way to pretrain the model on custom data. Can anybody please confirm this? Also can someone please direct me to articles and blogs in which the whole process of pretraining and fine tuning the pretrained model is explained along with the code(preferably in Tensorflow)?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.