Does anyone have any experience organizing large data analysis projects? It seems that the majority of project organization tools for data science (DVC, datmo, mlflow, etc) assume a very model-centric approach.
On the other hand, I am currently dealing with a project that doesn't need any predictive models, but contains lots and lots of data to be explored, analyzed, summarized, and re-organized (rich geographic datasets can be quite multidimensional and messy). As the result the project directory looks approximately like this:
./DS-project | |__ data/ | |__notebooks/ | | | |__thing_to_analyze_1/ | | | | | |__01_aa_analysis1.ipynb | | |__01.1_aa_analysis1.ipynb | | |__02_aa_analysis2.ipynb | | ... | | | |__thing_to_analyze_2/ | ... | |__src/ | | | |__utility_package_1/ | | | | | |__README.rst | | |__setup.py | | ... | | | |__utility_package_2/ | ... | |__README.md |__environement.yml
Currently the biggest pain is caused by the notebooks. We are following notebook naming convention suggested here, but due to the sheer number of
thing_to_analyze_x folders and the number of notebooks in them, it can be quite hard to find previous research (even when notebook summaries are documented in some sort of README file).
Does anyone have experience dealing with such projects? Date based project/notebook naming seems quite interesting (e.g.:
2019-03-05_thing-to-analyze-1), but I'm afraid it might make it harder to understand which stuff is old and which stuff has been recently revisited.