I was wondering what the best way is to perform API calls to ChatGPT, and in general, with LLMs API providers, to achieve the highest throughput in a batch inference context.

Suppose you have 1000 input prompts ready to be processed. What is the best approach to perform the requests, obtaining the highest possible throughput (I don't care too much about latency)? Is it better to batch prompts together and send fewer but bigger payload requests, or send single small requests in parallel? Or is there any other way?

In both cases, we need to respect the TPM and RPM limits, so I imagine we also need some control logic to slow down the process if needed.

Whatever the selected approach, is there any library/tool/platform to support and ease the handling of requests?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.