what are the steps to train BertForMaskedLM model on custom corpus and load it again and test it on new sentences?

  • I followed the instructions shared in BER github page to train a
    language model
  • i ran these 2 file create_pretraining_data.py ,python run_pretraining.py as mentioned in the instruction ,and there was a sample_text.txt corpus file on which the model was trained on.

after running above 2 python file i got below list of files in the folder output_dir=/tmp/pretraining_output

  • checkpoint
  • List item
  • eval
  • eval_results.txt
  • events.out.tfevents.1575545953.26760d2fc979
  • graph.pbtxt
  • model.ckpt-0.data-00000-of-00001
  • model.ckpt-0.index
  • odel.ckpt-0.meta
  • model.ckpt-20.data-00000-of-00001
  • model.ckpt-20.index
  • model.ckpt-20.meta

1.what should i do now to test and predict masked words for a sentences which is inside sample_text.txt corpus file on which the model was trained .

For example if there was a sentence in sample_text.txt corpus like
"He went to space.He brought a moon"

if i want to test my pretrained BertForMaskedLM to check if it correctly predicts the masked word in sentences" He went to [Mask] .He brought a gallon [Mask]

so the model must predict the same words which was in sample_text.txt corpus "space","moon" rather than other words like "store","water" since it was trained on this sample_text.txt corpus .im expecting this behavior .Is this possible to build language model using bert ?

2.how to load the BertForMaskedLM model that was trained and stored in folder /tmp/pretraining_output for this task?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.