Similar question here Deploy ML Model on AWS, but more detail provided below:

If I understand correctly, models such as llama and stable diffusion, are all types of Artificial Neural Networks (ANNs). These ANNs all have different ways to interact with them, and all produce different outputs (e.g. text vs images).

One thing these ANNs have in common is that they can can be hosted on Hugging Face's 'Inference Endpoints': https://ui.endpoints.huggingface.co/ and then using them is as simple as calling api endpoints from, e.g. a web app built in NodeJs. (I've done this).

Inference Endpoints, I have discovered, seem to be wrappers around services such as aws EC2 instances, and they also provide the services for interacting with these via api (presumably by deploying some form of API Gateway setup for AWS or something).

Inference Endpoints are therefore quite expensive, as one is paying for a 'constantly on' server which is always running in the cloud. As someone who builds serverless stuff, I'd like to figure out whether it is possible to host Models in a similar way.

Is is possible to host a model in such a way so that it is not on an 'always on' server, such as on an AWS Lambda-like structure? I have started looking into SageMaker, however these seem to also be instances of servers.

I am aware that I will probably have to figure out some way of creating my own api for interacting with these Models too. Is there a service out there which provides something similar to 'Inference Endpoints' but on shared instances like ChatGPT?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.