0
$\begingroup$

I want to use the Greek BERT which can be found here https://huggingface.co/nlpaueb/bert-base-greek-uncased-v1

However I am confused about which model should I use and which are the differences.

The tokenizer is the same

tokenizer = AutoTokenizer.from_pretrained('nlpaueb/bert-base-greek-uncased-v1')

but we have two models

model = AutoModel.from_pretrained("nlpaueb/bert-base-greek-uncased-v1")
model = AutoModelWithLMHead.from_pretrained("nlpaueb/bert-base-greek-uncased-v1")

Which one should I use?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

The main difference between the two models is that the first one is a general BERT model without a specific head that simply returns the raw hidden states of the model, whereas the second one has an architecture with a language modelling head on top. You can find some more info in the huggingface documentation, compare for example the explanation of the BertModel versus the BertLMHeadModel.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you so much for your answer. So If I want to finetune the bert model for a classification task which model should I use? From your answer I think these are the same models where if I use the first one (AutoModel.from_pretrained) I must construct the classification layer, while if I use the second one (AutoModelWithLMHead) there is a function for the classification task. Is that right? $\endgroup$
    – John Smith
    Commented Mar 26, 2022 at 18:41
  • $\begingroup$ The first model simply returns the last hidden state from the model, the second one returns the scores for each token in the vocabulary. $\endgroup$
    – Oxbowerce
    Commented Mar 27, 2022 at 16:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.