1
$\begingroup$

My goal is to have a language model that understands the relationships between words and can fill the masks in a sentence related to a specific domain. At first, I thought about pretraining or even training a language model(like BERT) from scratch, but unfortunately, my data isn't that big to help the previous model learn new connections, let alone learn the embeddings from scratch.

Now what I have in mind is creating a transformer model with my own vocabulary which consists of words in my domain-specific data (after separating them with spaces and not using transformer tokenizers). This way the vocab size would be smaller and the positions and relations would be learned faster and more easily. Although I'm a bit confused about implementation.

Can I use this architecture (that is for NMT) and give plain text for both the input and output? or should I mask some tokens in the input and give the complete sentence as the label?

Any other suggestions?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

First I suggest reading the transformers paper. Couple of quick notes is that this model consists of an encoder and a decoder, and the original task the paper is trained on is machine translation. Datasets (benchmarks) they used to train and evaluate this model from scratch were WMT 2014 Engligh-to-German, WMT 2014 English-to-French (section 5.1 of the paper). The conclusion is that you cannot train transformers from scratch unless you have your sentence pairs in 2 languages.

MLM on the other hand is something that BERT used for training. So if you want to go in this direction, you can use the pre-trained BERT and fine-tune it with Masked language model head on top of your BERT model using your own dataset. You need to stick to the tokenizer that BERT was originally trained on but at least you are taking advantage of some general context that was learned during training BERT from scratch. If you want to train BERT either from scratch or fine-tune with MLM head you can follow this tutorial from hugging face

Improve this answer
$\endgroup$
3
  • $\begingroup$ Hi @FatemehRahimi, thanks for the reply. I actually tried the second approach and could pre-train the lite Persian BERT(ALBERT) model and upload the model on HuggingFace. I chose to go with this one as its vocabulary is a bit smaller and won't parse words that much. $\endgroup$
    – mitra mirshafiee
    Apr 27, 2021 at 7:07
  • $\begingroup$ @mitramirshafiee what do you mean by won't parse words that much? (Do you mean the tokenization is not what you expected it to be?) $\endgroup$
    – Fatemeh Rahimi
    Apr 27, 2021 at 16:03
  • $\begingroup$ I meant that this version of BERT has a smaller vocabulary (30,000 instead of 42000 of BERT) and can learn representations of important words faster by not chunking them in very small parts. $\endgroup$
    – mitra mirshafiee
    Apr 28, 2021 at 4:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.