2
$\begingroup$

Having working on data sets, sometimes we want to keep track of mtiple models with different architectures which work on the same data set on which we have made some transformations and preprecessing of data has been done.

So I would like to know what is the elegant way to work on multiple models which use the same data set? Because having multiple models on the same notebook is cumborsome and recreating the same data preprocessing nd transformations on separate notebooks is also a lot of copy-paste which I think can be solved with some existing solutions which I am not aware of. What is the industry standard when doing such tasks?

Any help is appreciated.

$\endgroup$

1 Answer 1

2
$\begingroup$

If I was in your situation, I would approach it this way:

  • Create a "preprocessing"/"data" module

That module can either be a simple data access layer that can be shared across notebooks, or it can also include preprocessing steps that you can add. This enforces a common data access layers across notebooks without necessarily duplicating code.

So you could do something like this:

from data_layer import Data, Preprocessor

preprocessor = Preprocessor(**kwargs)
data = Data(preprocessor = preprocessor, **kwargs)

In an ideal world, you should be able to define your data access layer as data itself. For instance, a JSON document containing the data source, and the preprocessing options.

  • Create a different notebook per model

And in this case, if your models share components, I'd also create modules they can all import from. The same way you'd use sklearn or torch component.

$\endgroup$
3
  • $\begingroup$ Yeah this sounds doable. Is this how people in the industry approach any problem? $\endgroup$ Commented Jun 27, 2020 at 11:43
  • $\begingroup$ Well, people in the industry should only use notebooks for development/experiments anyway. If you want to put anything to production, you have to write actual software. The way I work is I write my code as a Python package and import it in my notebook. $\endgroup$ Commented Jun 27, 2020 at 12:53
  • $\begingroup$ Yeah, that sounds more like an industrial approach, Comng from a software development background i was inquisite about how things go in datascience industry. $\endgroup$ Commented Jun 27, 2020 at 15:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.