As a data scientist who recently joined a new team, I wanted to ask the community how they share data and models among their colleagues. Currently I have to resort to storing data in some central server or location where all of us can access (which means unix permissions etc). For models I also tend to send a weights file over to my colleague and share my github. Both I've found pretty cumbersome. What have some of you done?
-
2$\begingroup$ file server + weights files here :p for non-data scientist colleagues I make models accessible through web APIs (mostly using cherrypy). $\endgroup$– stmaxMar 17, 2017 at 18:52
-
2$\begingroup$ cherrypy docs - interesting, never heard of that before $\endgroup$– Martin ThomaMar 17, 2017 at 19:53
-
$\begingroup$ If you have a model / dataset you want to publish, I can recommend zenodo.org $\endgroup$– Martin ThomaMar 17, 2017 at 19:53
6 Answers
You can try using dvc
, which stands for data version control.
https://dvc.org/
Honestly once you get to something serious (big or evolving) the main problem is about sharing and updating data. Once a solution is devised for data, it is not really hard to adapt it to models.
Depending on the volume and the usage, the data can be stored, exchanged and acessed in a wide range of solutions. It might be old historical/external solution with associated langages (SAS), internal data bases on linux server with a 'lab' interface, clouds solutions or just csv file for tinier data sets. Once this is arranged and the solution to access those data is devised is it rather easy to adapt the solution to models. Sometimes it demand a bit of involvment to handle the rights properly but it shouldn't be too much of a hassle if the architecture is set with that goal in mind.
So, regarding models, depending on the context, the solution might be sharing SAS programs on a shared workspace, pickled python models on a linux server or simply Excel files trough mail. More recently, new 'tools' appeared and result in new solution : share experiments trough notebooks (Jupyter notebooks or R Markdown) and use Git for versionning models.
Listening to the podcast Partially Derivative Episode "Data Science On The Silicon Beach" the host interviews Maksim Percherskiy, Chief Data Officer for the City of San Diego.
Talking about the stack he uses for the City of San Diego: (08:50) "The way we move data around ... we use Airflow [...] and Airflow is just Python." Percherskiy continues characterizing the data sharing problem in the context of city government.
-
1$\begingroup$ airflow is for moving data around computational pipelines, not the office :) $\endgroup$– EmreApr 27, 2017 at 16:53
-
$\begingroup$ Honestly, I haven't used Airflow, and offered this because the interview explicitly discusses sharing datasets (~06:00 - 11:00). So, I if Airflow is not "the solution" then at least its in the ballpark. $\endgroup$– xtianApr 29, 2017 at 17:48
-
$\begingroup$ No, it's not. Airflow is for sharing data between tasks, not people: Use airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress, and troubleshoot issues when needed. Does this sound like what you'd use to share your data or model with your coworkers? $\endgroup$– EmreApr 29, 2017 at 21:10
For big files, I use cloud storage (Google, Amazon, Microsoft, or whichever ecosystem your company's on), with folders named after the issue/project ticket name/number. These services support file versioning, by the way. Small files I just attach to the ticket. If have to share something small and transient with a handful of people I can use email or our corporate chat application.
-
$\begingroup$ thanks! sounds like you use quite a few different approaches. Any reason you wouldn't just stick to one? $\endgroup$ Apr 27, 2017 at 3:08
-
$\begingroup$ Because I'm at a startup and I'm empowered to do whatever I want to get the job done ASAP. And I think I don't use that many approaches; only two or three. $\endgroup$– EmreApr 27, 2017 at 4:20
To store and share the data amongst colleagues, cloud storage is the option we use (s3, google storage) where you can just have a folder structure to store all your datasets. While there is no specific way to share models, it totally depends on the model type, one thing that's used is making a binary of model (pickle in python) and share that file which you can also encrypt in case you are floating around sensitive data.
In case it is an unsupervised learning model you can directly share the codebase.
I'd say this question is much broader, than simple sharing files. How do you perform research with team in agile fashion? Sadly, there's just a few solutions on the market. As mentioned before, most of the people hack it using already available services. Some time ago I've stumbled upon neptune.ml. It looks nice, though is quite pricey. Most of the time I'm doing something similar. I try to stick to git-flow
convention and I have separate folder in repo named research
, next to vanilla git-flow
branches. Also keep your data in separate repo, so you know against which data version you ran your experiments.