1
$\begingroup$

We are building an ML pipeline on AWS, which will obviously require some heavy-compute components including preprocessing and batch training.

Most the the pipeline is on Lambda, but Lambda is known to have time limits on how long a job can be run (~15mins). Thus for the longer running jobs like batch training of ML models we will need(?) to access something like EC2 instances. For example a lambda function could be invoked and then kick off an EC2 instance to handle the training.

Are there any alternatives to using EC2 for the heavy compute? Is there a way to still host/run the job on AWS without leveraging any EC2 to do the compute?

The idea is to avoid the extra management that comes with EC2 since we’re not currently using it. Keeping everything ad close to Lambda-like as possible is ideal.

$\endgroup$

1 Answer 1

1
$\begingroup$

For batch training i have been utilizing sagemaker though it's a bit expensive then ec2 but it's easy to setup and get started.

  • Make a docker container and push it to ecr then start the training and track the metrics using any monitoring tool like wandb
  • If your use case don't require any custom packages then you can also utilize HuggingFace DLC it which can make it more easy to start training.

References:

$\endgroup$
3
  • $\begingroup$ Excellent, thank you. $\endgroup$
    – Cybernetic
    Jun 29, 2021 at 15:43
  • 1
    $\begingroup$ I would think that SageMaker is just using EC2 behind the scenes anyway, but I guess with SageMaker they are managing it, whereas with EC2 you must manage everything yourself. I also see that SageMaker Python SDK is open source....is this a cheaper way to use SageMaker? $\endgroup$
    – Cybernetic
    Jun 29, 2021 at 15:52
  • 2
    $\begingroup$ sagemaker sdk is utilized for working with sagemaker so they are not different. Basically sagemaker helps you concentrate more on model building and training. $\endgroup$ Jun 29, 2021 at 16:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.