I wish to train two domain-specific models:
- Domain 1: Constitution and related Legal Documents
- Domain 2: Technical and related documents.
For Domain 1, I've access to a text-corpus with texts from the constitution and no question-context-answer tuples. For Domain 2, I've access to Question-Answer pairs.
Is it possible to fine-tune a light-weight BERT model for Question-Answering using just the data mentioned above?
If yes, what are the resources to achieve this task?
Some examples, from the huggingface/models library would be