I will soon deal with multiple projects in python. Some of them have to run regularly (many times a day), they can take some time (many days) and they use/produce some data coming from and going to other servers.

I would like to know what tools you use to handle this kind of processes, knowing that they run on python. These tools could focus on:

  • workflow management (what task should be executed when ?)
  • data synchronization (database updates/synchronization)
  • resources monitoring (space on disk, RAM and CPU ?)
  • versioning (when these input/output arrived, should I treat them, or is there a new version ?)
  • visualization/monitoring (easy estimate the state of the processes, be alerted if an error occurred)

As an example, luigi tackle some of these problems.

Share links to articles or to tutorials if you have some in mind. I think this kind of challenges are faced by many people starting machine learning projects for production purposes.

  • $\begingroup$ Great question! Did you by any means find a set of useful tools? $\endgroup$ – Pieter Feb 8 '18 at 13:49
  • $\begingroup$ I only found out that Jupyter Notebook was working in this direction (see the JupyterCon videos on youtube, many interesting work in progress, but not enough "production" oriented to my opinion). But I actually decided to create my own set of tools, with the ambition to release a public version as soon as it is stable. It's working with flask and adapts ideas of web development (like Django) but specifically focused on highly computational processes and large data pipelines. I'll answer my own question the day I release the project on GitHub. $\endgroup$ – Robin Feb 8 '18 at 13:55
  • $\begingroup$ I'm looking forward to it :) $\endgroup$ – Pieter Feb 8 '18 at 14:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.