Data Science Project Data Workflow Structure

Question

I'm in the middle of a project of marketing regarding the sales prediction with promotions. The client has very complex business processes and so the data needs a lot of preprocessing (joins, filters, etc.). I have organize the code in different jupyter notebooks: one for the client dimension, one for the products dimension, one for the sales, etc. I have the notebooks ordered so that it is possible to run them correctly.

At first, I considered that a good idea, trying to follow the "modularity of the code" explained in the OOP principles, but the problem is that I have to load/save the data in every notebook. We're talking of millions of rows that take a while to load and save, and so now I'm wondering if breaking the code in different notebooks is really a good practice.

One-Notebook	Multiple-Notebooks
preprocess all the data every time I want to run it ⚠️	run just what is needed ✅
load/save the data just once ✅	take a long time loading/saving the data in every step ⚠️

Is there any solution for that? A good practice to organize the code so that it can keep its modular form while not having to load/save multiple times? I was reading about project structures like cookiecutter, Makefile, etc. but I'm not sure how to implement them so that I can solve my problem. Any resource that explain how to approach this?

Thanks

Answer 1

This is a tradeoff that you need to decide for yourself, taking into account the frequency and time cost of each operation.

My suggestion, and what I consider to be the middle-of-the-road approach is to save intermediate data.

It does not only offer modularity of the code on the conceptual level, but also modularity in run-times. This would mean for example that if your preprocessing is considered to be ok, and you want to see changes after changing something to the prediction model, you can load the saved preprocessed data (not the original) and work with that. This is the only way you can avoid always running every step.

Reading the preprocessed data will almost always be faster than the combination of reading the original data and running the preprocessing routines. This is also more efficient for memory (you don't need to have every intermediate step loaded in memory at once). So, if speed is your concern, I think that is the best solution. It also can be life-saving if an error occurs somewhere in the middle, in which case you will at least have saved data up until the previous process.

To make this more helpful, (in case you are not already doing it) try to separate the code from the experiments. You shouldn't be thinking Jupyter Notebooks in an OOP way (the client module, the product module) but in terms of processes (the client data preprocessing module), since that's what a Jupyter Notebook does, executes some processes. This could reduce the number of intermediate steps that need to save and load data.
Create python classes/functions (even if they are just a simple wrapper in some cases) in separate modules that do each of the steps required, and then use the jupyter notebooks only to run stuff, as needed.

Another advantage of this kind of modularity is that you could run things in parallel, if for example you are doing two different preprocessing routines to create different feature sets.

If on the other hand you find yourself running everything most of the time, then the single notebook might be the best way to go since you simply save time by not saving/loading for no reason, but in my experience I have found out that it is almost never the case.

Answer 2

Based on your question and comments, I would suggest not creating multiple python notebooks. If you are looking for modularity, instead of using multiple notebooks to segregate your code, use a single notebook, segregate your code into functions and then use it. This has the added advantage of making your code more readable.

Another advantage is that you can use Multiprocesing/Multithreadding for parallel processing. If for example you want to run 2 functions at the same time, conventionally you would run function A first, wait for it to complete and then execute function B. But using Multithreadding for example, you can run both of them simultaneously.

Also just an FYI, processing the data usually takes longer than saving or loading the data (depending on the tasks and data length).