How to prepare code data to fine tune a code LLM in an unsupervised way or is it even possible?

For example: Task: Code summarization with custom code base (with no summaries) Let's assume that this code base is unique, and a pre-trained model is giving unsatisfactory results. Now to fine tune there are three options,

  1. Manually prepare summaries for a portion of the code and fine tune
  2. Find a similar code base which has the labels (docstring) and fine tune
  3. Mask some portions of the code randomly and give as input and output will be the masked portions

Options 1 and 2 don't seem feasible for a production environment.

The reasoning behind option 3 is that with no availability labels, the model will learn the patterns in the code base and provide a better summarization with its pre-trained knowledge.

I tried the option 3 with [CodeT5+ fine tuning] (https://github.com/salesforce/CodeT5/blob/main/CodeT5%2B/tune_codet5p_seq2seq.py). The format of input and output was as follows Input:

    def __init__(self, text, font):
        self._text = text
        self._font = font

    def get_text(self):

    def set_text(self, value):
        self._text = value```

```return self._text```


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.