0
$\begingroup$

our team of 4 data scientists has been exploring options to automate our ETL, data storage and model updates and are looking for a hassle-free platform that can help with this.

We've had several consultancies trying to sell us expensive and complicated options like Synapse Analytics, which are an overkill. Our workflows consist mostly of data import, cleaning and export as well as model training, evaluation and generating predictions. We do the explorative analysis on local machines but once our models are finalized, we need to set them up so that the ETL and the subsequent training and predicting happen automatically. Our datasets are rather small, not more than 20-30 K observations and usually <100 columns. We use Python (80%), SQL (15%) and R (5%) and GitHub as our code sharing platform. All our production code is stored in scripts, not notebooks, which we only use for exploration.

Currently, we have two of our models saving outputs to Azure datalake but we're looking for something that can also do the computation automatically based on a schedule, without it being too complicated to administer or too expensive for our small team (we're not expected to grow).

Are there any other good options there?

$\endgroup$

1 Answer 1

0
$\begingroup$

You can always use the almighty task scheduler. Windows can periodically run a script that saves the output data. You can also schedule a script to receive new accuracy reports. Also you can set the script to send you a warning if something does not go as expected. So yeah, you just need a working computer.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.