How to select the optimal beam size for beam search?

Question

Most Text Generation Models use beam search to select the optimal output candidate. How does one choose the optimal beam size? It would probably vary from task to task, dataset to dataset, and model to model. But given it all these parameters are fixed, how do we choose the optimal beam size? Theoretically scores(beam_size) > scores(beam_size -1) but practically that may not be the case when evaluating for metrics like ROUGE, or BLEU. So is it experimentally determined, for example, to run it for all beam sizes and report the one with the best beam size? I am particularly curious about two aspects:

1. In research projects, do people tune the beam size parameter or do they just take the largest reasonable beam size that fits whatever GPU they have?
2. When these models are deployed in the real world, how is the beam size determined given the incoming distribution of inputs may be wildly different from the training dataset such that the empirically validated beam size? Or is this not a significant enough concern for resources to be deployed for this optimization?

Answer 1

Large beam sizes do not lead to improvements but to degradation in the generated text quality, as described in the article Empirical Analysis of Beam Search Performance Degradation in Neural Sequence Models. Also, article On NMT Search Errors and Model Errors: Cat Got Your Tongue? gives a nice insight into the problems of beam search for translation.

In general, the beam size is a hyperparameter that must be tuned. However, given that the "good values" are in a small range, the same value is normally used (for each task) instead of exploring different options. In the first mentioned article, you can find a table with an analysis of different beam sizes for different tasks:

In machine translation, the typical value used in most papers is 4-5. For instance, in the article that proposed the Transformer architecture (Attention is all you need), the beam size was 4.

Note that it is possible to mitigate the problems of large beam sizes by e.g. normalizing scores by sentence length. For instance, in Six Challenges for Neural Machine Translation we can see that, in machine translation for some language pairs (see German-English, in Figure 10), there is no degradation with increasingly large beam sizes.

Normally, the beam size value is not changed for different input distributions (e.g. different domains). At least there seems to be no literature on the matter.

Nevertheless, there are other popular decoding strategies apart from beam search. For instance, in LLMs, it's typical to use temperature sampling (the generated token is sampled from the probability distribution after applying a temperature factor $\alpha$, which can either flatten the distribution or sharpen it) or top_p/nucleus sampling (you sample from the probability distribution, but only consider the top probability tokens that add up to a specific cumulative probability p). You can check this answer for more decoding strategies.