Python Machine Learning/Data Science Project Structure

Question

I'm looking for information on how should a Python Machine Learning project be organized. For Python usual projects there is Cookiecutter and for R ProjectTemplate.

This is my current folder structure, but I'm mixing Jupyter Notebooks with actual Python code and it does not seems very clear.

.
├── cache
├── data
├── my_module
├── logs
├── notebooks
├── scripts
├── snippets
└── tools

I work in the scripts folder and currently adding all the functions in files under my_module, but that leads to errors loading data(relative/absolute paths) and other problems.

I could not find proper best practices or good examples on this topic besides some kaggle competition solutions and some Notebooks that have all the functions condensed at the start of such Notebook.

Answer 1

I don't think there are best practices in this developing area yet but, in addition to cookiecutter, there were some interesting ideas shown in a tutorial at the SciPy 2016 conference.

Personally, I try to minimize the number of sub-folders in a project unless I have a really good way to distinguish them and have good reason to keep them separate. Bad organization is nearly as bad as no organization. I think best practice might depend on use case -- not every project needs the same amount of boilerplate.

Answer 2

The University of Washington has released a project template for small scientific python projects (including data science projects) called shablona. Is that more or less what you're looking for?