what are the steps to train BertForMaskedLM model on custom corpus and load it again and test it on new sentences?
- I followed the instructions shared in BER github page to train a
- i ran these 2 file create_pretraining_data.py ,python run_pretraining.py as mentioned in the instruction ,and there was a sample_text.txt corpus file on which the model was trained on.
after running above 2 python file i got below list of files in the folder output_dir=/tmp/pretraining_output
- List item
1.what should i do now to test and predict masked words for a sentences which is inside sample_text.txt corpus file on which the model was trained .
For example if there was a sentence in sample_text.txt corpus like
"He went to space.He brought a moon"
if i want to test my pretrained BertForMaskedLM to check if it correctly predicts the masked word in sentences" He went to [Mask] .He brought a gallon [Mask]
so the model must predict the same words which was in sample_text.txt corpus "space","moon" rather than other words like "store","water" since it was trained on this sample_text.txt corpus .im expecting this behavior .Is this possible to build language model using bert ?
2.how to load the BertForMaskedLM model that was trained and stored in folder /tmp/pretraining_output for this task?