Have a fine-tuned a summarization model following the Hugging Face seq2seq guide (starting from sshleifer/distilbart-xsum-12-6).
We are interested in using AWS elastic inference for deployment for cost reduction. (e.g. similar to this)
Wondering whether there’s any examples or any suggested way to use the beam searching logic in BartForConditionalGeneration with model inference from a torchscript model. Most of the examples for torchscript found are with classification tasks where this isn’t necessary.