23
$\begingroup$

I would like to set up a server which could support a data science team in the following way: be a central point for storing, versioning, sharing and possible also executing Jupyter notebooks.

Some desired properties:

  1. Different users can access the server and open and execute notebooks that were stored by them or by other team members. The interesting question here is what would be the behavior if user X executes cells in a notebook authored by user Y. I guess the notebook should NOT be changed:
  2. Solution should be self-hosted.
  3. Notebooks should be stored either on the server or on Google drive or on self-hosted instance of owncloud.
  4. (Bonus) Notebooks will be under git versioning control (git may be self-hosted. Cannot be bounded to GitHub or something of that sort).

I looked into JupyterHub and Binder. With the former, I didn't understand how to allow cross users access. The latter seems to only support GitHub as the storage of the notebooks.

Do you have experience with either of the solutions?

$\endgroup$
5
  • $\begingroup$ Kaggle announced a system that might be what you want. $\endgroup$ – Ricardo Magalhães Cruz Nov 11 '16 at 15:07
  • 2
    $\begingroup$ JupiterHub is exactly for this propose. $\endgroup$ – dannyeuu Nov 15 '16 at 14:24
  • $\begingroup$ @dannyeuu I assume you meant JupyterHub, right? Is it possible to have cross users notebooks sharing? $\endgroup$ – Dror Atariah Nov 16 '16 at 7:08
  • 1
    $\begingroup$ No, each user has a separate instance of Jupyter spawned. AFAIK you can't easily share notebooks. $\endgroup$ – Lukasz Tracewski Dec 2 '16 at 20:06
  • $\begingroup$ Google Colaboratory definitely meets requirement 3. $\endgroup$ – Leponzo Apr 11 '19 at 21:02
3
$\begingroup$

JupyterHub does not provide version control system nor facilitates sharing of Notebooks. You mentioned yourself limitation of Binder.

Try Zeppelin. Version 0.7 should be released within a few next days.

  • As you can see from the roadmap, this version delivers "enterprise" features which are exactly about collaboration.
  • Version control system (git) is integrated.
  • It's self-hosted.

In essence, I think it meets all requirements you posted. On top of that it delivers richer visualisation capabilities and plethora of other features (works with Shiro, Knox, Kerberos - secure Spark anyone?).

$\endgroup$
2
$\begingroup$

Airbnb recently open sourced their internal data science knowledge repository: https://github.com/airbnb/knowledge-repo

From its readme, it seems it could loosely fit your use case:

The Knowledge Repository project is focused on facilitating the sharing of knowledge between data scientists and other technical roles using data formats and tools that make sense in these professions. It provides various data stores (and utilities to manage them) for "knowledge posts", with a particular focus on notebooks (R Markdown and Jupyter / iPython Notebook) to better promote reproducible research.

There's also a blog post commenting on its motivation.

$\endgroup$
1
$\begingroup$

What I found - sharing notebooks for data scientists is a not a desirable format for communication. Many of them prefer IDE like Spider/RStudio or just a text editors (I know a few data scientists who use vi).

You might just share code by your source control and data by cloud storages. It will increase flexibility.

I've recently open sourced a tool which combines code, data, and the dependencies between data and code to a single environment and makes your data science project reproducible: DVC or dataversioncontrol.com (there is a tutorial).

With DVC tool you can just share your project by Git, sync data to S3 by a single DVC command. If some of your data scientists decide to change the code at any stage of your project then the final result could be easily reproduced by a single command dvc repro data/target_metrics.txt.

$\endgroup$
0
$\begingroup$

The only self-hosted solution I know is the paid Anaconda Enterprise cloud setup, https://anaconda.org/about. The other solutions I am aware of are not self-hostable!

$\endgroup$
1
  • $\begingroup$ I'm confused. Doesn't cloud mean that it's not self-hosted? $\endgroup$ – Paul Brannan Jan 29 at 21:35
0
$\begingroup$

Isn't this solution good enough ?

You can protect the access with ssh, and the hosted files could be the git repository you want, with different linux (or whatever) user access. You'll need your own server.

$\endgroup$
0
$\begingroup$

Domino Data Lab offers premises, SaaS, and VPC-based notebook hosting (Jupyter, Zeppelin, RStudio), git integration, scalable compute, environment templates, and a bunch of other useful things. The premises/ VPC offerings may be overkill and too pricey if you're a small team, but the SaaS plans are pretty reasonably priced.

[ Full disclosure: I'm a former Domino employee ]

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.