0
$\begingroup$

I have a question regarding hyperparameter optimization in scikit learn. I am most familiar with tensorflow where you first split your data into three sets: Train, validation and test. Hyperparameters are optimized using the train and validation sets, and then the model is finally evaluated using the test set. All data is normalized using statistics from the test set, and as far as I have understood the rationale behind this is that the model has not "seen" the data from validation and test and therefore can not use their stats in any way.

Anyways, my question is related to the optimizers in scikit. As far as I can see they all use cross-validation. So I have two questions:

  • Is there a way to set the validation set explicitly in the scikit optimizers?
  • If not, how do I deal with the normalization? Should I just feed the optimizer with both the train and validation sets, wouldn't that just mix the sets, and somehow mess up the rationale behind initially using the training set for normalization?

Side note: I want to be able to compare models made using both scikit and tensorflow, so I want to follow procedures that are as similar as possible when comparing models.

$\endgroup$

2 Answers 2

1
$\begingroup$

I think it will suit you sklearn.model_selection.PredefinedSplit. From the user manual User guide :

For some datasets, a pre-defined split of the data into training- and validation fold or into several cross-validation folds already exists. Using Pre defined Split it is possible to use these folds e.g. when searching for hyperparameters.

For example, when using a validation set, set the test_fold to 0 for all samples that are part of the validation set, and to -1 for all other samples.

I can also recommend good libraries for optimizing hyperparameters, very easy to learn, but with great functionality. They have algorithms for selecting hyperparameters not only by brute force or by chance, but also conditional hyperparameters. For example, in if parameter1 appears only if there is parameter2 . These are optuna and hyperopt.

In them, you can more flexibly specify the learning function, rather than the sklearn pipeline with all its limitations.

$\endgroup$
0
$\begingroup$

It seems to me that you simply want to use a fixed validation set for hyperparameter tuning. This is a very cardinal procedure, especially in the scenario as you mentioned, that one wants to ensure that the results are comparable across models.

What I would try, if there aren't a whole lot of hyperparameters, is to manually set grids for those parameters and perform brute-force grid search. Sklearn has a nice feature sklearn.model_selection.ParameterGrid that helps you iterate all possible combinations of hyperparmaters.

If there are indeed a lot of hyperparameters, or the values should be sampled from a continuous distribution, then I would suggest using hyperparameter techniques practiced in deep learning field. Essentially, try using Weights & Biases, which is very versatile yet easy to use.

See this SO question (https://stackoverflow.com/questions/54126811/order-between-using-validation-training-and-test-sets) also if you want some clarifications on hyperparam tuning strategies.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.