5
$\begingroup$

Hi i downloaded the BERT pretrained model (https://storage.googleapis.com/bert_models/2018_10_18/cased_L-12_H-768_A-12.zip) from here and saved to a directory in gogole colab and in local .

when i try to load the model in colab im getting "We assumed '/content/drive/My Drive/bert_training/uncased_L-12_H-768_A-12/config.json" . tried to laod the model in local machine and getting same error .

this is how i loaded the model: from transformers import BertForMaskedLM BertNSP=BertForMaskedLM.from_pretrained('/content/drive/My Drive/bert_training/uncased_L-12_H-768_A-12/')

is this the correct way of loading model from the directory when i have downloaded the pretrained model ? Im getting error " '/content/drive/My Drive/bert_training/uncased_L-12_H-768_A-12/config.json' " the downloaded model had these naming conventions where file name start with bert_ but the BertForMaskedLM class is expecting the file name to be config.json .

bert_config.json bert_model.ckpt.data-00000-of-00001 bert_model.ckpt.index vocab.txt bert_model.ckpt.meta

FULL ERROR: Model name '/content/drive/My Drive/bert_training/uncased_L-12_H-768_A-12/' was not found in model name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc, bert-base-german-dbmdz-cased, bert-base-german-dbmdz-uncased). We assumed '/content/drive/My Drive/bert_training/uncased_L-12_H-768_A-12/config.json' was a path or url to a configuration file named config.json or a directory containing such a file but couldn't find any such file at this path or url.

when i renamed the above 4 files by removing bert from all 4 file names , i get this error even though the "model.ckpt.index" files exist

ERROR: "OSError: Error no file named ['pytorch_model.bin', 'tf_model.h5', 'model.ckpt.index'] found in directory /content/drive/My Drive/bert_training/uncased_L-12_H-768_A-12/ or from_tf set to False"

Improve this question
$\endgroup$
2
  • $\begingroup$ Did you unzip the downloaded file ? $\endgroup$
    – Astariul
    Dec 9, 2019 at 2:21
  • $\begingroup$ @Astraiul ,yes i have unzipped the files and below are the files present and my path is pointing to these unzipped files folder .bert_config.json bert_model.ckpt.data-00000-of-00001 bert_model.ckpt.index vocab.txt bert_model.ckpt.meta $\endgroup$
    – star
    Dec 9, 2019 at 9:36

2 Answers 2

Reset to default
2
$\begingroup$

You are using the Transformers library from HuggingFace.

Since this library was initially written in Pytorch, the checkpoints are different than the official TF checkpoints. But yet you are using an official TF checkpoint.

You need to download a converted checkpoint, from there.

Note : HuggingFace also released TF models. But I'm not sure if it works without conversion from official TF checkpoints. If you want to use the TF API of HuggingFace, you need to do :

from transformers import TFBertForMaskedLM

Improve this answer
$\endgroup$
9
  • $\begingroup$ @ Astariul im getting ImportError:. I want to train the bert masked language model on custom corpus ,i followed the step shared in BERT githhub "github.com/google-research/bert#pre-training-with-bert" . i ran the 2 .py files create_pretraining_data.py,run_pretraining.py and i got these files in output_dir "checkpoint, eval, eval_results.txt, graph.pbtxt, 3 files starting with words model.ckpt". how to load model which got saved in output_dir inorder to test and predict the masked words for sentences in custom corpus that i used for training this model. $\endgroup$
    – star
    Dec 10, 2019 at 11:17
  • $\begingroup$ I never did it before, but I think you should convert the TF checkpoint your created into a checkpoint that HuggingFace can read, using this script. $\endgroup$
    – Astariul
    Dec 10, 2019 at 23:53
  • $\begingroup$ I followed the instruction and create a PyTorch model using this pyhton code ->convert_bert_original_tf_checkpoint_to_pytorch.py Save PyTorch model to /content/drive/My Drive/BMaskLang the BMaskLang file was 402 MB size and it did not have any file extension ,now when i tired to load this pytorch model i get an error from transformers import BertForMaskedLM model = BertForMaskedLM.from_pretrained("/content/drive/My Drive/BMaskLang") Error: UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte $\endgroup$
    – star
    Dec 11, 2019 at 7:36
  • $\begingroup$ Sorry I never used this conversion script. If you have difficulties with the script, you can receive more help by opening an issue on HuggingFace's repo or open a new question. If I solved your initial question, please accept my answer :) $\endgroup$
    – Astariul
    Dec 11, 2019 at 8:05
  • $\begingroup$ Thank for your help .! but are you able to impot this model ? from transformers import TFBertForMaskedLM im not able to do it $\endgroup$
    – star
    Dec 11, 2019 at 12:35
4
$\begingroup$

You can import the pre-trained bert model by using the below lines of code:

pip install pytorch_pretrained_bert

from pytorch_pretrained_bert import BertTokenizer, BertModel, BertForNextSentencePrediction

BERT_CLASS = BertForNextSentencePrediction

# Make sure all the files are in same folder, i.e vocab , config and bin file
PRE_TRAINED_MODEL_NAME_OR_PATH = '/path/to/the/files/containing/models/files'

model = BERT_CLASS.from_pretrained(PRE_TRAINED_MODEL_NAME_OR_PATH, cache_dir=None)
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.