I may have the wrong stack exchange. If that's the case, could someone point me to a stack that could help with this. Anyways...

My backend employs a sentence transformer model from HuggingFace. Since the number of requests per day is small, deploying a dedicated instance for serving inference requests is not cost-effective. Hence, I was thinking of AWS Lambda whereby I could pay per inference.

However, I need to serve each request fast, which necessitates using a GPU, which AWS Lambda does not offer.

Is there a deployment solution (not necessarily at AWS) whereby I would be able to use a GPU and pay per inference?

  • $\begingroup$ Have you considered doing inference on CPU with a lightweight framework like ggml? $\endgroup$
    – noe
    Commented Aug 30, 2023 at 13:45
  • $\begingroup$ @noe This is a good idea, but translating a ready model into ggml is not trivial. There is this project, but it does not take care of this, which is one of the models I am using. $\endgroup$ Commented Sep 2, 2023 at 17:21


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.