I want to train a LLM (prefered Llama-2-13b) to learn the entropy of german texts - to be specific sports news. I use perplexity as training metric and want to check the training success after the training. I want to use the fine-tuned model for RAG and I hope, that the fine-tuned model understand the query and context better to give a better answer compared to the orignal model How can an experiment look like to compare the fine-tuned Llama-2-13b related to the original Llama-2-13b preferably automated for this use case?

  • 2
    $\begingroup$ Tell us about the loss function, that is, the entropy of each sports article. How exactly are you evaluating the output of one model against another? $\endgroup$
    – J_H
    Apr 16 at 20:48
  • $\begingroup$ That's a good question, because in natrual language you have ambiguousness. I would evaluate the answer by hand or with kind of RAGAS $\endgroup$ Apr 18 at 11:12


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.