2
$\begingroup$

this is more a curiosity query than anything else. The git repo for dolly gives us an easy way to swap, the training dataset , to train custom models, as long as we follow the format. I however have been through tons of content online BUT am unable to find ANY github repo which shows me exactly how the "instruction" and "context" are parsed in order to generate the final response. Here's my best guess

  • use a pre trained model (gpt2 / gptj etc ) and create the mirror architecture. Now copy the weights from the pre trained model, hence initializing your custom model
  • now pick the question & input text ( aka instruction and context ), convert them using a tokenizer ( choosing any HF based api ) and MOST IMP concatenate the "instrn" and "context" using some sort of separator ( this is what i read on using distilbert to solve QnA datasets )
  • for the target / answer, we again, tokenize this thing and ensure that the final layer of the custom model outputs the same dimensionality as the temporal dimension of the output. Meaning if the "answer" is curtailed to 128 words, then the penultimate dimension of the output will be 128.
  • try some version of cross entropy loss to compare these 2 and then we have the usual loss minimization exercise

i even read the InstructGPT paper and their git repo is basically an empty shell. Any pointers to git repos which show how this works, will be deeply appreciated.

$\endgroup$
1
  • $\begingroup$ "MOST IMP concatenate the "instrn" and "context" using some sort of separator" - I'm concerned about this step since when a user asks a question, they won't be separating the context and question using a "specific" separator. It will probably be a newline or a space or something. Additionally, in a real question, either the context or the question can come first. Would this matter when fine tuning the model to follow instructions? $\endgroup$
    – dhruvbird
    Jul 5, 2023 at 4:19

1 Answer 1

2
$\begingroup$

Dolly is just fine-tuned Pythia.

The instruction inputs are just collapsed into one long string here: https://github.com/databrickslabs/dolly/blob/master/training/trainer.py#L108

Yes it's just run through the Pythia tokenizer during processing. Yes, that's about it, it's learning to continue the strings and measuring some loss comparing the actual next word to predicted probabilities.

$\endgroup$
2
  • $\begingroup$ just to be super clear, if "context" is true, then i basically collapse context, inp and response and then mask random words just like in mlm and then reproduce the whole string? this way during the inference when the model is provided "context" and "instruction", it should ideally generate the response, token by token? apologize if this sounds too dumb but thats what i got from the code and your comments $\endgroup$ Apr 25, 2023 at 1:46
  • 1
    $\begingroup$ This is a causal language model, so I think you'd more naturally fine tune that way, not as a masked language model. Or it would take more work to retrain another way. $\endgroup$
    – Sean Owen
    Apr 26, 2023 at 20:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.