I recently downloaded Camembert Model to fine-tune it for my purpose.

Upon unzipping the file the contents are: enter image description here

Upon loading the model.pt file using pytorch:

import torch
model = torch.load(model_saved_at)

I saw that model was in OrderedDict format containing the following keys:


As the name suggests most of them are OrderedKeys themselves with the exception of args which belongs to a class argsparse.Namespace. Using vars() we can see args only contains some hyperparameters and values which are to be passed from the command-line.

model["model"] contains the weights which I want to load and use as my base model. A small part of it is as shown below:

for ans in model["model"].keys():
        print(ans, "\t" ,model["model"][ans].size())
        print(ans, type(ans))
decoder.sentence_encoder.embed_tokens.weight     torch.Size([32005, 768])
decoder.sentence_encoder.embed_positions.weight      torch.Size([514, 768])
decoder.sentence_encoder.layers.0.self_attn.in_proj_weight   torch.Size([2304, 768])
decoder.sentence_encoder.layers.0.self_attn.in_proj_bias     torch.Size([2304])
decoder.sentence_encoder.layers.0.self_attn.out_proj.weight      torch.Size([768, 768])
decoder.sentence_encoder.layers.0.self_attn.out_proj.bias    torch.Size([768])
decoder.sentence_encoder.layers.0.self_attn_layer_norm.weight    torch.Size([768])
decoder.sentence_encoder.layers.0.self_attn_layer_norm.bias      torch.Size([768])
decoder.sentence_encoder.layers.0.fc1.weight     torch.Size([3072, 768])
decoder.sentence_encoder.layers.0.fc1.bias   torch.Size([3072])
decoder.sentence_encoder.layers.0.fc2.weight     torch.Size([768, 3072])
decoder.sentence_encoder.layers.0.fc2.bias   torch.Size([768])

However, I cannot use load_state_dict() since I have no instance of this class. How am I suppose to load the weights and optimization parameters without creating an instance? I thought of using sentence.bpe.model but they are for tokenization purposes.


If you are open to using huggingface transformer for fine tuning which is really popular, here is a code sample:

import transformers
class Model(nn.Module):
    def __init__(self):
        self.Bert = transformers.CamemBertModel.from_pretrained('camembert-base')
        self.fc0 = nn.Linear(768,1)

        nn.init.normal_(self.fc0.weight,std= 0.1)
        nn.init.normal_(self.fc0.bias ,0.)
    def forward(self,input_ids,attention_mask):
        hid= self.Bert(input_ids,attention_mask = attention_mask)
        hid= hid[0][:,0]
        x = self.fc0(hid)
        return x

You can change your last layer as you need.It is just a sample. It will load the pretrained weights from huggingface.You need not provide it.You can install transformers by the following line in terminal.

pip3 install transformers
  • $\begingroup$ Thanks for the answer but I downloaded this seperately just to avoid using huggingface, it definitely would be easy but it would also restrict a lot of what I can and cannot do. $\endgroup$ – Übermensch Aug 5 '20 at 9:57
  • $\begingroup$ without the class file,which you won't probably have, you cannot optimize the camembert layer.All you can do is use it as a encoder layer,extract features and train your last layers which is similar as above.Rather in above you can optimizer camembert as well $\endgroup$ – SrJ Aug 5 '20 at 9:59
  • $\begingroup$ Is there any way to just load the initial model and not optimize those and only optimize the layers that I would be adding after these? $\endgroup$ – Übermensch Aug 5 '20 at 11:44
  • $\begingroup$ Yes. Just like the code i have provided. Instead of transform layer use your model layer to get output and send them to your layers $\endgroup$ – SrJ Aug 5 '20 at 13:16
  • $\begingroup$ Could you make an edit showing how would one go on from first layer to the last? $\endgroup$ – Übermensch Aug 5 '20 at 14:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.