0
$\begingroup$

as a self learner Data Scientist I am working on deploying my model in order to get regular predictions. And I need your guidance about deployment section, in which I know nothing about and there is nobody around to guide me. There is not any line of code etc, so I am just asking for your theoretical opinions.

My process will go on like this:

1- Every 2 hours, system will run a query and save a csv file to a specific location. It will overwrite of course. And this task is on me, I am not asking about it. This csv file is the prediction dataset actually, which will be retrieved by ML model periodically to make predictions on it.

2- I have to make predictions, lets say, every 2 hour. My service should retrieve this csv file automatically from the local path I will define.

3- Then it should make predictions based on the model I will define for it in pickle form.

4- Finally it should download the results or final dataframe in csv format again into the same path as predictions.csv for example. It should overwrite of course.

For this process, I am making my research on Flask Web Services. But I must ask you, is this process possible with flask applications? As you can see, there is not any manual trigger in this process, everything is automatic, no one needs to visit a web page or hit a run button etc. If you can suggest me any resources or web pages that you think may be beneficial for me, it is highly welcome.

Have a nice day

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

This seems more like an ETL job than an API one. Of course you can have both, but in your specific case I would design it as a series of tasks where each task you extract, transform and load data.

Airflow is an amazing tool for that: https://github.com/apache/airflow In airflow, each ETL is defined as a DAG (directed acyclic graph) with a series of tasks that are customized by you.

So, for your specific case, you would create a DAG with following tasks:

  • Retrieve data, prep if necessary.
  • Make predictions
  • Save new dataset predictions.csv.

You can schedule it like a cron job to run at each 2 hours. I highly recommend you to look at.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Also, feel free to contact me if you have any further questions and need help. I am starting to learn this kind of stuff and I think I can help you out at this beginning. $\endgroup$
    – Victor Oliveira
    Commented Aug 13, 2020 at 12:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.