Reinforcement learning for ensemble models

Question

afaik RLHF has been consistently been associated with Gen AI tasks. The reasoning is that since gen AI is stochastic and can generate multiple responses ( based on small changes in prompts, parameters like temperature etc ) its a good candidate for alignment. This is done by giving it enough samples of pairs of (preferred , not so preferred) and then it minimizes the distance between the distributions generated by the under-training policy and the preferred options. This is done over batches and averaged over an epoch. So far so good ?

I have an ensemble model ( lets say a BERT and a LayoutLM ) whose job is to extract key value pairs from a document. Since both models are trained end to end jointly , it is capable of generating different options for the same underlying input ( since the input is text generated via OCR and OCR itself can generate slight differences based on the quality of the document ). Can i use the feedback generated by the human agent ( who corrects the output generated via a UX ) to train a policy ? The reason i am so confused by this approach is i am unable to convince myself as to why a reward function + policy training is at all needed for this since i can very simply fine tune the model with the corrections made by the human operator. Will i be missing something with this approach where i am passing the model JUST the correct answers and NOT giving it an option to UNDERSTAND the difference between a correct and a wrong answer ?

Answer 1

I believe going through this link may help alleviate some of your concerns. https://superannotate.com/blog/rlhf-for-llm

My two cents on this would be that you can essentially directly fine-tune the LLM with preferred responses by creating a preferred-dataset with better response target values. But to expand on the quote in the article

Reinforcement learning (RL) comes into play when you have a complex and not strictly defined task. Imagine trying to teach someone a game without explicitly telling them the rules but instead rewarding them for good moves. That's the essence of RL—it's about guiding the model towards making a series of decisions that lead to the best outcome, even when the "best" isn't clearly defined from the start.

Your separate RL model is learning what the "preference" of the output is and is dictating the further training data that goes into re-training/fine-tuning the LLM later on whilst also improving with continuous feedback of it's own. If the learned policy is good, you are updating LLM responses on the fly with scored outputs, or something different utilizing the same principle.

With direct Human-Feedback re-training/fine-tuning, I think the process becomes a lot more manual along with increasing the number of fine-tuning runs of the LLM, with the agent just learning to score a response accurately, you can use it to score the output of the LLM that is quite powerful in however you would like to use the value it provides.