2
$\begingroup$

CLIP: https://openai.com/research/clip

They use a small text encoder and suggest that the simpler the model the better. Is there any reason why that is? Has anyone tried using a pretrained LLM as the text encoder and then fine tuning? I would assume the LLM would learn a lot better representation of the meaning of the text.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

The link you provide states multiple reason why, namely:

"We know from GPT-2 and 3 that models trained on such data can achieve compelling zero shot performance; however, such models require significant training compute. To reduce the needed compute, we focused on algorithmic ways to improve the training efficiency of our approach."

CLIP is meant to show that good zero-shot performance is achievable without requiring large and expensive datasets or long and costly compute. Seeing if an LLM outperforms CLIP could be an interesting research question, but then it would definitely need to factor in the difference in compute and data required.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Ok, I assumed they were trying to maximize performance in which case it would've made more sense to me to fine-tune an LLM vs train a smaller model from scratch $\endgroup$
    – JobHunter69
    Commented Dec 29, 2023 at 18:04
  • $\begingroup$ Maybe or maybe not. This paper clearly states the bottleneck of labeled data. It is not a given that an LLM would outperform CLIP given the same dataset $\endgroup$
    – Valentin Calomme
    Commented Dec 29, 2023 at 20:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.