My goal is to have a language model that understands the relationships between words and can fill the masks in a sentence related to a specific domain. At first, I thought about pretraining or even training a language model(like BERT) from scratch, but unfortunately, my data isn't that big to help the previous model learn new connections, let alone learn the embeddings from scratch.
Now what I have in mind is creating a transformer model with my own vocabulary which consists of words in my domain-specific data (after separating them with spaces and not using transformer tokenizers). This way the vocab size would be smaller and the positions and relations would be learned faster and more easily. Although I'm a bit confused about implementation.
Can I use this architecture (that is for NMT) and give plain text for both the input and output? or should I mask some tokens in the input and give the complete sentence as the label?
Any other suggestions?