Skip to main content
added another example/advantage
Source Link
liakoyras
  • 636
  • 4
  • 15

This is a tradeoff that you need to decide for yourself, taking into account the frequency and time cost of each operation.

My suggestion, and what I consider to be the middle-of-the-road approach is to save intermediate data.

It does not only offer modularity of the code on the conceptual level, but also modularity in run-times. This would mean for example that if your preprocessing is considered to be ok, and you want to see changes after changing something to the prediction model, you can load the saved preprocessed data (not the original) and work with that. This is the only way you can avoid always running every step.

Reading the preprocessed data will almost always be faster than the combination of reading the original data and running the preprocessing routines. This is also more efficient for memory (you don't need to have every intermediate step loaded in memory at once). So, if speed is your concern, I think that is the best solution. It also can be life-saving if an error occurs somewhere in the middle, in which case you will at least have saved data up until the previous process.

To make this more helpful, (in case you are not already doing it) try to separate the code from the experiments. You shouldn't be thinking Jupyter Notebooks in an OOP way (the client module, the product module) but in terms of processes (the client data preprocessing module), since that's what a Jupyter Notebook does, executes some processes. This could reduce the number of intermediate steps that need to save and load data.
Create python classes/functions (even if they are just a simple wrapper in some cases) in separate modules that do each of the steps required, and then use the jupyter notebooks only to run stuff, as needed.

Another advantage of this kind of modularity is that you could run things in parallel, if for example you are doing two different preprocessing routines to create different feature sets.

If on the other hand you find yourself running everything most of the time, then the single notebook might be the best way to go since you simply save time by not saving/loading for no reason, but in my experience I have found out that it is almost never the case.

This is a tradeoff that you need to decide for yourself, taking into account the frequency and time cost of each operation.

My suggestion, and what I consider to be the middle-of-the-road approach is to save intermediate data.

It does not only offer modularity of the code on the conceptual level, but also modularity in run-times. This would mean for example that if your preprocessing is considered to be ok, and you want to see changes after changing something to the prediction model, you can load the saved preprocessed data (not the original) and work with that. This is the only way you can avoid always running every step.

Reading the preprocessed data will almost always be faster than the combination of reading the original data and running the preprocessing routines. This is also more efficient for memory (you don't need to have every intermediate step loaded in memory at once). So, if speed is your concern, I think that is the best solution. It also can be life-saving if an error occurs somewhere in the middle, in which case you will at least have saved data up until the previous process.

To make this more helpful, (in case you are not already doing it) try to separate the code from the experiments. You shouldn't be thinking Jupyter Notebooks in an OOP way (the client module, the product module) but in terms of processes (the client data preprocessing module), since that's what a Jupyter Notebook does, executes some processes. This could reduce the number of intermediate steps that need to save and load data.
Create python classes/functions (even if they are just a simple wrapper in some cases) in separate modules that do each of the steps required, and then use the jupyter notebooks only to run stuff, as needed.

If on the other hand you find yourself running everything most of the time, then the single notebook might be the best way to go since you simply save time by not saving/loading for no reason, but in my experience I have found out that it is almost never the case.

This is a tradeoff that you need to decide for yourself, taking into account the frequency and time cost of each operation.

My suggestion, and what I consider to be the middle-of-the-road approach is to save intermediate data.

It does not only offer modularity of the code on the conceptual level, but also modularity in run-times. This would mean for example that if your preprocessing is considered to be ok, and you want to see changes after changing something to the prediction model, you can load the saved preprocessed data (not the original) and work with that. This is the only way you can avoid always running every step.

Reading the preprocessed data will almost always be faster than the combination of reading the original data and running the preprocessing routines. This is also more efficient for memory (you don't need to have every intermediate step loaded in memory at once). So, if speed is your concern, I think that is the best solution. It also can be life-saving if an error occurs somewhere in the middle, in which case you will at least have saved data up until the previous process.

To make this more helpful, (in case you are not already doing it) try to separate the code from the experiments. You shouldn't be thinking Jupyter Notebooks in an OOP way (the client module, the product module) but in terms of processes (the client data preprocessing module), since that's what a Jupyter Notebook does, executes some processes. This could reduce the number of intermediate steps that need to save and load data.
Create python classes/functions (even if they are just a simple wrapper in some cases) in separate modules that do each of the steps required, and then use the jupyter notebooks only to run stuff, as needed.

Another advantage of this kind of modularity is that you could run things in parallel, if for example you are doing two different preprocessing routines to create different feature sets.

If on the other hand you find yourself running everything most of the time, then the single notebook might be the best way to go since you simply save time by not saving/loading for no reason, but in my experience I have found out that it is almost never the case.

Source Link
liakoyras
  • 636
  • 4
  • 15

This is a tradeoff that you need to decide for yourself, taking into account the frequency and time cost of each operation.

My suggestion, and what I consider to be the middle-of-the-road approach is to save intermediate data.

It does not only offer modularity of the code on the conceptual level, but also modularity in run-times. This would mean for example that if your preprocessing is considered to be ok, and you want to see changes after changing something to the prediction model, you can load the saved preprocessed data (not the original) and work with that. This is the only way you can avoid always running every step.

Reading the preprocessed data will almost always be faster than the combination of reading the original data and running the preprocessing routines. This is also more efficient for memory (you don't need to have every intermediate step loaded in memory at once). So, if speed is your concern, I think that is the best solution. It also can be life-saving if an error occurs somewhere in the middle, in which case you will at least have saved data up until the previous process.

To make this more helpful, (in case you are not already doing it) try to separate the code from the experiments. You shouldn't be thinking Jupyter Notebooks in an OOP way (the client module, the product module) but in terms of processes (the client data preprocessing module), since that's what a Jupyter Notebook does, executes some processes. This could reduce the number of intermediate steps that need to save and load data.
Create python classes/functions (even if they are just a simple wrapper in some cases) in separate modules that do each of the steps required, and then use the jupyter notebooks only to run stuff, as needed.

If on the other hand you find yourself running everything most of the time, then the single notebook might be the best way to go since you simply save time by not saving/loading for no reason, but in my experience I have found out that it is almost never the case.