I wish to train two domain-specific models:

  • Domain 1: Constitution and related Legal Documents
  • Domain 2: Technical and related documents.

For Domain 1, I've access to a text-corpus with texts from the constitution and no question-context-answer tuples. For Domain 2, I've access to Question-Answer pairs.

Is it possible to fine-tune a light-weight BERT model for Question-Answering using just the data mentioned above?

If yes, what are the resources to achieve this task?

Some examples, from the huggingface/models library would be mrm8488/bert-tiny-5-finetuned-squadv2, sshleifer/tiny-distilbert-base-cased-distilled-squad, /twmkn9/albert-base-v2-squad2.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.