0
$\begingroup$

I'm reading a program that use the pre-trained Roberta model (roberta-base). The code first extracts word embeddings from each caption in the batch, using the last hidden state of the Roberta model. Then, the model is trained to align these word embeddings with the image features (pixels) of the image through a type of attention mechanism. Then the models are updated using attention loss function. This iterative process continues until the training is complete, so I guess the word embeddings will be different after each epoch ? This is a multi-modal problem.

When I compare the Roberta model after training with the pre-trained model (roberta-base), I notice that every parameters the trained Roberta model are different, seems like the new model has updated the parameters. I'm not sure whether this is a form of fine-tuning or just feature extraction or both ?

$\endgroup$

1 Answer 1

1
$\begingroup$

Feature extraction does not modify the model's weights, it just uses the model in inference mode to get its outputs and hidden states. Given that you are updating the model's weights, you are fine-tuning it.

$\endgroup$
8
  • $\begingroup$ The contextualized embeddings in the first epoch are feature extraction because it takes directly from the pre-trained Roberta, but the contextualized embeddings for rest of epochs are different because the model has updated ? $\endgroup$
    – user159173
    Mar 9 at 10:12
  • $\begingroup$ Yes, the contextual embeddings are the result of applying the model's parameters to the input. As the model parameters are changed during fine-tuning, the result of applying your model will be different as soon as the first training step ends. $\endgroup$
    – noe
    Mar 9 at 10:41
  • $\begingroup$ So it's fine-tuning overall, right ? $\endgroup$
    – user159173
    Mar 9 at 17:58
  • $\begingroup$ Yes, that's it. $\endgroup$
    – noe
    Mar 9 at 20:39
  • 1
    $\begingroup$ That's right, you should not create a new tokenizer to fine-tune RoBERTa. $\endgroup$
    – noe
    Mar 10 at 19:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.