I am familiar with ML and Deep Learning concepts and have had a look at Dolly and even got the pretrained model running on a Jupyter lab notebook on Databricks.
However when I take a look at their training dataset format, they are all in instruction and response format.
My specific question is that if I have a super large dump of general text that is not labelled in form of instruction and response, can I just train Dolly as an autoregressive language model that will take a piece of text as an input to the generate function later once trained, and just generate text ?
Suggestions would be really appreciated. Thanks