I'm in the middle of a project of marketing regarding the sales prediction with promotions. The client has very complex business processes and so the data needs a lot of preprocessing (joins, filters, etc.). I have organize the code in different jupyter notebooks: one for the client dimension, one for the products dimension, one for the sales, etc. I have the notebooks ordered so that it is possible to run them correctly.
At first, I considered that a good idea, trying to follow the "modularity of the code" explained in the OOP principles, but the problem is that I have to load/save the data in every notebook. We're talking of millions of rows that take a while to load and save, and so now I'm wondering if breaking the code in different notebooks is really a good practice.
|preprocess all the data every time I want to run it ⚠️
|run just what is needed ✅
|load/save the data just once ✅
|take a long time loading/saving the data in every step ⚠️
Is there any solution for that? A good practice to organize the code so that it can keep its modular form while not having to load/save multiple times? I was reading about project structures like cookiecutter, Makefile, etc. but I'm not sure how to implement them so that I can solve my problem. Any resource that explain how to approach this?