0
$\begingroup$

what are the steps to train BertForMaskedLM model on custom corpus and load it again and test it on new sentences?

  • I followed the instructions shared in BER github page to train a
    language model
    "https://github.com/google-research/bert#pre-training-with-bert"
  • i ran these 2 file create_pretraining_data.py ,python run_pretraining.py as mentioned in the instruction ,and there was a sample_text.txt corpus file on which the model was trained on.

after running above 2 python file i got below list of files in the folder output_dir=/tmp/pretraining_output

  • checkpoint
  • List item
  • eval
  • eval_results.txt
  • events.out.tfevents.1575545953.26760d2fc979
  • graph.pbtxt
  • model.ckpt-0.data-00000-of-00001
  • model.ckpt-0.index
  • odel.ckpt-0.meta
  • model.ckpt-20.data-00000-of-00001
  • model.ckpt-20.index
  • model.ckpt-20.meta

1.what should i do now to test and predict masked words for a sentences which is inside sample_text.txt corpus file on which the model was trained .

For example if there was a sentence in sample_text.txt corpus like
"He went to space.He brought a moon"

if i want to test my pretrained BertForMaskedLM to check if it correctly predicts the masked word in sentences" He went to [Mask] .He brought a gallon [Mask]

so the model must predict the same words which was in sample_text.txt corpus "space","moon" rather than other words like "store","water" since it was trained on this sample_text.txt corpus .im expecting this behavior .Is this possible to build language model using bert ?

2.how to load the BertForMaskedLM model that was trained and stored in folder /tmp/pretraining_output for this task?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.