2
$\begingroup$

Can anyone fine-tune the GPT-3 model on commodity hardware without GPU?

  • What I meant is can we fine tune available GPT-3 equivalent models?
  • For example, we have only access to GPT-J.
  • Can we fine-tune GPT-J with commodity hardware or lets say with only basic GPU such as 1with one RTX 3080.

Can we fine-tune these models (not training from scratch)?

Or will it need high-end infrastructure with GPUs?

$\endgroup$

1 Answer 1

4
$\begingroup$

The weights of GPT-3 are not public. You can fine-tune it but only through the interface provided by OpenAI. In any case, GPT-3 is too large to be trained on CPU.

About other similar models, like GPT-J, they would not fit on a RTX 3080, because it has 10/12Gb of memory and GPT-J takes 22+ Gb for float32 parameters. It should possible to fine-tune some special versions that use int8 precision, like this one.

$\endgroup$
2
  • $\begingroup$ @neo sorry for my ambiguous question. Updated with further detail. $\endgroup$
    – Exploring
    Nov 16, 2022 at 22:54
  • 2
    $\begingroup$ I updated my answer with the new information you provided. $\endgroup$
    – noe
    Nov 16, 2022 at 23:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.