I am looking for best way to deploy my Deep Learning application which does the following:

  1. Fetches data from BigQuery
  2. Finds anomaly (based on trained model)
  3. Throws the output in a mysql database.

I want this program to run everyday on a specific time like a cronjob.

Any lead on how can I achieve this deployment, other than using a cronjob because it will be unnecessary to pay for the time and resources when it will not be in use.

Thanks in Advance.

  • $\begingroup$ What stops you from running it as a standalone application from a cron job, depending on your application server setup? $\endgroup$ – mapto Jun 27 '18 at 12:11
  • $\begingroup$ @mapto- Thanks for your comment. I was looking for more something where ML and DL models can be deployed. Ofcourse cronjob can run the script which will evoke the model as well, but then I guess I need to set up GPUs and higher RAM in cron in that case. Might just turn out to be more expensive. What's your thought in it? $\endgroup$ – Jazz Jun 27 '18 at 12:16
  • 1
    $\begingroup$ Have a cron job running on a tiny server that boots the bigger server? $\endgroup$ – kbrose Jul 31 '18 at 13:56

There are two steps necessary, you need a trained model and you need to apply this model regularly to new data.

Step 1: After training your model you need to serialize this in some way. This could be via the tooling in your deep learning framework, or by saving the parameters somewhere and the code somewhere else or by pickling the whole object. After storing this model artifact we can reuse this fitted model in step 2.

Step 2: Schedule some job that loads this model artifact, loads the data from BigQuery (in chunks or the full scope of the rows for the last time period), runs these through your model and saves the results. The schedule only kicks off some script, how to properly run these predictions depends a bit on the scale. If it is a very big job you would likely have to distribute this via something like Spark. If it is managable you could just do this on a single machine, maybe by chunking the data.


other than using a cronjob because it will be unnecessary to pay for the time and resources when it will not be in use

It sounds like you want to use cloud compute.

I would suggest looking at the Events offered by AWS in their CloudWatch tool. You can set up a job and then schedule it to run at any time you like - they even have a cron-job interface to specify something like: every hour.

Have a look at relevant AWS documentation for Events

You could probably set this up based on a single AWS instance, checking that you meet three requirements for your use case:

  1. It has enough network bandwidth to receive your BigQuery results in a reasonable time
  2. It has the required hardware to execute your model (e.g. is a GPU necessary?)
  3. It can hold or link to your own mySQL database

There are probably even nice AMI (Amazon Machine Image - with pre-installed software) that will be ready to go for your use case!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.