0
$\begingroup$

I want to perform a global optimization of the entire model development pipeline. I have several stages of development, each of which can be performed automatically: preprocessing, removal of outliers/anomalies, oversampling, feature selection, hyperparameters tuning. Each of these steps has many different implementations. That is, anomalies can be removed by dozens of different algorithms, etc. Stages can be repeated. For example, you can tuning hyperparameters , remove 10% of the most unused functions, repeat the search for hyperparameters. Train several models and combine them into an ensemble. Remove anomalies with 1 algorithm, then another. Remove anomalies after feature selection or before, etc. I also want to do all these steps for different input datasets (all within the 1 task of choosing the best data source)

That is, there are many different combinations of these steps. But it is too expensive to run one big function to find all the hyperparameters of all the algorithms. So I want to expertly select different combinations.

I need a Pipeline execution framework that is ml-oriented, so that I can conveniently create different configurations in it like a lego constructor. So that this framework automatically determines, since the anomaly removal model has already been trained on these inputs and saved to disk, then it doesn't need to be trained again. And that you can easily get a final comparison of all approaches and choose the best one.

I tried to use kedro, but it seems that there you need to generate for each option separate entries in yaml files using modular pipelines. This is a very cumbersome and inconvenient approach.

$\endgroup$

2 Answers 2

1
$\begingroup$

Depending at which stage you are in your project, I'd suggest using an experiment tracking system for machine learning, like MLflow, Metaflow, Weights & Biases, etc., in combination maybe with DVC.

There you can log your experiments, including: algorithms chosen, their hyperparameters, evaluation metrics, you can log plots, artifacts, any other custom "thing" that's important to you.

I personally use MLflow for this, there's a bit of an overhead in terms of writing code (you have to call something.log(..) a lot) but it has a nice dashboard functionality that enables you to visually inspect an experiment and compare different experiment runs. You have the concept of a project there, and you can specify multiple entrypoints, which you can logically split as: data pre-processing, hyperparemeter optimisation, algorithm selection, evaluation, etc.

The actual algorithm selection step, the hyperparameter search space, the data pre-processing steps, etc. are written in whatever library you use (sklearn, lightgbm, etc.) and its logic is defined by you. I use Optuna to do hyperparameter optiimsation, pandas for data manipulation, sklearn for data pre-processing and model evaluation, lightgbm for modeling, etc.

If you have access to cloud, I think Sagemaker is like an AWS solution for this kind of problems, but not 100% sure, never worked with it.

To sum up, my advice is, instead of writing one big function to handle all your steps, write many small experiments that handle one logical step within your pipeline/project, use something like MLflow to figure out what works best, and iterate, a lot.

$\endgroup$
0
$\begingroup$

"And that you can easily get a final comparison of all approaches and choose the best one." This comes from an experiment tracking tool very easily. I use CometML and the graphing capabilities for comparing each training run to each other and then also graphs for digging deeper into an individual experiment are awesome. As said above, there are a number of these tools, but they're not hard to learn and simply modeling life a lot.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.