# Loading a Model with weights and optimizers without creating an instance in PyTorch

I recently downloaded Camembert Model to fine-tune it for my purpose.

Upon unzipping the file the contents are:

Upon loading the model.pt file using pytorch:

import torch
model = torch.load(model_saved_at)


I saw that model was in OrderedDict format containing the following keys:

args
model
optimizer_history
extra_state
last_optimizer_state


As the name suggests most of them are OrderedKeys themselves with the exception of args which belongs to a class argsparse.Namespace. Using vars() we can see args only contains some hyperparameters and values which are to be passed from the command-line.

model["model"] contains the weights which I want to load and use as my base model. A small part of it is as shown below:

for ans in model["model"].keys():
try:
print(ans, "\t" ,model["model"][ans].size())
except:
print(ans, type(ans))

decoder.sentence_encoder.embed_tokens.weight     torch.Size([32005, 768])
decoder.sentence_encoder.embed_positions.weight      torch.Size([514, 768])
decoder.sentence_encoder.layers.0.self_attn.in_proj_weight   torch.Size([2304, 768])
decoder.sentence_encoder.layers.0.self_attn.in_proj_bias     torch.Size([2304])
decoder.sentence_encoder.layers.0.self_attn.out_proj.weight      torch.Size([768, 768])
decoder.sentence_encoder.layers.0.self_attn.out_proj.bias    torch.Size([768])
decoder.sentence_encoder.layers.0.self_attn_layer_norm.weight    torch.Size([768])
decoder.sentence_encoder.layers.0.self_attn_layer_norm.bias      torch.Size([768])
decoder.sentence_encoder.layers.0.fc1.weight     torch.Size([3072, 768])
decoder.sentence_encoder.layers.0.fc1.bias   torch.Size([3072])
decoder.sentence_encoder.layers.0.fc2.weight     torch.Size([768, 3072])
decoder.sentence_encoder.layers.0.fc2.bias   torch.Size([768])


However, I cannot use load_state_dict() since I have no instance of this class. How am I suppose to load the weights and optimization parameters without creating an instance? I thought of using sentence.bpe.model but they are for tokenization purposes.

## 1 Answer

If you are open to using huggingface transformer for fine tuning which is really popular, here is a code sample:

import transformers
class Model(nn.Module):
def __init__(self):
super(Model,self).__init__()
self.Bert = transformers.CamemBertModel.from_pretrained('camembert-base')
self.fc0 = nn.Linear(768,1)

nn.init.normal_(self.fc0.weight,std= 0.1)
nn.init.normal_(self.fc0.bias ,0.)

def forward(self,input_ids,attention_mask):
hid= self.Bert(input_ids,attention_mask = attention_mask)
hid= hid[0][:,0]
x = self.fc0(hid)
return x


You can change your last layer as you need.It is just a sample. It will load the pretrained weights from huggingface.You need not provide it.You can install transformers by the following line in terminal.

pip3 install transformers

• Thanks for the answer but I downloaded this seperately just to avoid using huggingface, it definitely would be easy but it would also restrict a lot of what I can and cannot do. – Übermensch Aug 5 '20 at 9:57
• without the class file,which you won't probably have, you cannot optimize the camembert layer.All you can do is use it as a encoder layer,extract features and train your last layers which is similar as above.Rather in above you can optimizer camembert as well – SrJ Aug 5 '20 at 9:59
• Is there any way to just load the initial model and not optimize those and only optimize the layers that I would be adding after these? – Übermensch Aug 5 '20 at 11:44
• Yes. Just like the code i have provided. Instead of transform layer use your model layer to get output and send them to your layers – SrJ Aug 5 '20 at 13:16
• Could you make an edit showing how would one go on from first layer to the last? – Übermensch Aug 5 '20 at 14:40