0
$\begingroup$

I am familiar with ML and Deep Learning concepts and have had a look at Dolly and even got the pretrained model running on a Jupyter lab notebook on Databricks.

However when I take a look at their training dataset format, they are all in instruction and response format.

My specific question is that if I have a super large dump of general text that is not labelled in form of instruction and response, can I just train Dolly as an autoregressive language model that will take a piece of text as an input to the generate function later once trained, and just generate text ?

Suggestions would be really appreciated. Thanks

Improve this question
$\endgroup$
4
  • $\begingroup$ Hi, Did you figure it out? I have the same question $\endgroup$
    – Khayam Gondal
    Commented May 30, 2023 at 20:50
  • $\begingroup$ Yes we can. Check the issues in their GitHub page. I posted the same question and the authors of Dolly replied back. $\endgroup$
    – London guy
    Commented May 31, 2023 at 21:22
  • $\begingroup$ Can you share code or explain a little bit how you did it? $\endgroup$
    – Khayam Gondal
    Commented Jun 1, 2023 at 22:09
  • $\begingroup$ Sure will write here tomorrow $\endgroup$
    – London guy
    Commented Jun 3, 2023 at 0:14

1 Answer 1

Reset to default
0
$\begingroup$

You can. If you look at the training script, you will see that it just turns the input into strings anyway. Autoregressive models' input are just strings. So you can modify this to simply return one big string in your input.

However the result will just learn to continue strings like your input. It won't necessarily answer questions or follow instructions. You may have to also fine-tune for your task.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.