2
$\begingroup$

Does anyone have any experience organizing large data analysis projects? It seems that the majority of project organization tools for data science (DVC, datmo, mlflow, etc) assume a very model-centric approach.

On the other hand, I am currently dealing with a project that doesn't need any predictive models, but contains lots and lots of data to be explored, analyzed, summarized, and re-organized (rich geographic datasets can be quite multidimensional and messy). As the result the project directory looks approximately like this:

./DS-project
|
|__ data/
|
|__notebooks/
|    |
|    |__thing_to_analyze_1/
|    |    |
|    |    |__01_aa_analysis1.ipynb
|    |    |__01.1_aa_analysis1.ipynb
|    |    |__02_aa_analysis2.ipynb
|    |    ...
|    |
|    |__thing_to_analyze_2/
|    ...
|
|__src/
|    |
|    |__utility_package_1/
|    |    |
|    |    |__README.rst
|    |    |__setup.py
|    |    ...
|    |
|    |__utility_package_2/
|    ...
|
|__README.md
|__environement.yml

Currently the biggest pain is caused by the notebooks. We are following notebook naming convention suggested here, but due to the sheer number of thing_to_analyze_x folders and the number of notebooks in them, it can be quite hard to find previous research (even when notebook summaries are documented in some sort of README file).

Does anyone have experience dealing with such projects? Date based project/notebook naming seems quite interesting (e.g.: 2019-03-05_thing-to-analyze-1), but I'm afraid it might make it harder to understand which stuff is old and which stuff has been recently revisited.

$\endgroup$
0
$\begingroup$

I think you could take a look at neptune.ml which is a pretty cool tool (for non-organizations) that lets you track and share your data science work of whether it is just running experiments or exploration.

For example, check out some model exploration or post-competition write-up that I put together.

There is a lot of work on improving wiki to be more interactive with user mentions comments etc. Anyhow, I figured it could be something interesting for you.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.