Benchmarking LLMs on technical questions

Question

There are several existing benchmark sets to evaluate the performance of large language models on natural-language comprehension tasks, such as CoQA, LAMBDA, HELLASWAG, LogiQA.

I'm interested in benchmarking large language models on technical tasks such as writing code, where questions might consist of something like 'Write a Python program to print the first ten prime numbers, one per line' and the output would be considered correct if feeding it to the Python interpreter does indeed produce the first ten prime numbers.

Are there any existing benchmark sets of that nature yet?

Answer 1

Just to take a couple of examples, GPT-4 performance was evaluated on HumanEval : https://arxiv.org/abs/2303.08774, LeetCode: https://leetcode.com/, and CodeForces. "One such metric is pass rate on the HumanEval dataset [43], which measures the ability to synthesize Python functions of varying complexity. We successfully predicted the pass rate on a subset of the HumanEval dataset by extrapolating from models trained with at most 1, 000× less compute"

CodeT : https://arxiv.org/pdf/2207.10397.pdf used HumanEval, MBPP, APPS, and CodeContests for performance evaluation; The github for HumanEval is here: https://github.com/openai/human-eval. hth.