I want to perform a global optimization of the entire model development pipeline. I have several stages of development, each of which can be performed automatically: preprocessing, removal of outliers/anomalies, oversampling, feature selection, hyperparameters tuning. Each of these steps has many different implementations. That is, anomalies can be removed by dozens of different algorithms, etc. Stages can be repeated. For example, you can tuning hyperparameters , remove 10% of the most unused functions, repeat the search for hyperparameters. Train several models and combine them into an ensemble. Remove anomalies with 1 algorithm, then another. Remove anomalies after feature selection or before, etc. I also want to do all these steps for different input datasets (all within the 1 task of choosing the best data source)
That is, there are many different combinations of these steps. But it is too expensive to run one big function to find all the hyperparameters of all the algorithms. So I want to expertly select different combinations.
I need a Pipeline execution framework that is ml-oriented, so that I can conveniently create different configurations in it like a lego constructor. So that this framework automatically determines, since the anomaly removal model has already been trained on these inputs and saved to disk, then it doesn't need to be trained again. And that you can easily get a final comparison of all approaches and choose the best one.
I tried to use kedro, but it seems that there you need to generate for each option separate entries in yaml files using modular pipelines. This is a very cumbersome and inconvenient approach.