8
$\begingroup$

As a data scientist who recently joined a new team, I wanted to ask the community how they share data and models among their colleagues. Currently I have to resort to storing data in some central server or location where all of us can access (which means unix permissions etc). For models I also tend to send a weights file over to my colleague and share my github. Both I've found pretty cumbersome. What have some of you done?

$\endgroup$
  • 2
    $\begingroup$ file server + weights files here :p for non-data scientist colleagues I make models accessible through web APIs (mostly using cherrypy). $\endgroup$ – stmax Mar 17 '17 at 18:52
  • 2
    $\begingroup$ cherrypy docs - interesting, never heard of that before $\endgroup$ – Martin Thoma Mar 17 '17 at 19:53
  • $\begingroup$ If you have a model / dataset you want to publish, I can recommend zenodo.org $\endgroup$ – Martin Thoma Mar 17 '17 at 19:53
1
$\begingroup$

You can try using dvc, which stands for data version control. https://dvc.org/

| improve this answer | |
$\endgroup$
0
$\begingroup$

Listening to the podcast Partially Derivative Episode "Data Science On The Silicon Beach" the host interviews Maksim Percherskiy, Chief Data Officer for the City of San Diego.

Talking about the stack he uses for the City of San Diego: (08:50) "The way we move data around ... we use Airflow [...] and Airflow is just Python." Percherskiy continues characterizing the data sharing problem in the context of city government.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ airflow is for moving data around computational pipelines, not the office :) $\endgroup$ – Emre Apr 27 '17 at 16:53
  • $\begingroup$ Honestly, I haven't used Airflow, and offered this because the interview explicitly discusses sharing datasets (~06:00 - 11:00). So, I if Airflow is not "the solution" then at least its in the ballpark. $\endgroup$ – xtian Apr 29 '17 at 17:48
  • $\begingroup$ No, it's not. Airflow is for sharing data between tasks, not people: Use airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress, and troubleshoot issues when needed. Does this sound like what you'd use to share your data or model with your coworkers? $\endgroup$ – Emre Apr 29 '17 at 21:10
0
$\begingroup$

For big files, I use cloud storage (Google, Amazon, Microsoft, or whichever ecosystem your company's on), with folders named after the issue/project ticket name/number. These services support file versioning, by the way. Small files I just attach to the ticket. If have to share something small and transient with a handful of people I can use email or our corporate chat application.

| improve this answer | |
$\endgroup$
  • $\begingroup$ thanks! sounds like you use quite a few different approaches. Any reason you wouldn't just stick to one? $\endgroup$ – asampat3090 Apr 27 '17 at 3:08
  • $\begingroup$ Because I'm at a startup and I'm empowered to do whatever I want to get the job done ASAP. And I think I don't use that many approaches; only two or three. $\endgroup$ – Emre Apr 27 '17 at 4:20
0
$\begingroup$

To store and share the data amongst colleagues, cloud storage is the option we use (s3, google storage) where you can just have a folder structure to store all your datasets. While there is no specific way to share models, it totally depends on the model type, one thing that's used is making a binary of model (pickle in python) and share that file which you can also encrypt in case you are floating around sensitive data.

In case it is an unsupervised learning model you can directly share the codebase.

| improve this answer | |
$\endgroup$
0
$\begingroup$

I'd say this question is much broader, than simple sharing files. How do you perform research with team in agile fashion? Sadly, there's just a few solutions on the market. As mentioned before, most of the people hack it using already available services. Some time ago I've stumbled upon neptune.ml. It looks nice, though is quite pricey. Most of the time I'm doing something similar. I try to stick to git-flow convention and I have separate folder in repo named research, next to vanilla git-flow branches. Also keep your data in separate repo, so you know against which data version you ran your experiments.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.