1
$\begingroup$

I'm working on optimizing the hyperparameters for several ML models (FFN, CNN, LSTM, BiLSTM, CNN-LSTM) at the moment, and running this alongside another experiment examining which word embeddings are best to use on the task of binary text classification.

My question is: should I decide on which embeddings to use before I tune the hyperparameters, or can I decide on the best hyperparameters and then experiment with the embeddings? The task remains the same in both cases.

In other words, is hyperparameter tuning more affected by the task (which is constant) or by the input data?

$\endgroup$
0
$\begingroup$

In other words, is hyperparameter tuning more affected by the task (which is constant) or by the input data?

It's correct that the task is constant, but hyper-parameters are usually considered specific to a particular learning algorithm, or to a method in general. In a broad sense the method may include what type of algorithm, its hyper-parameters, which features are used (in your case which embeddings), etc.

The performance depends on both the data and the method (in a broad sense), and since hyper-parameters are parts of the method, there's no guarantee that the optimal parameters stay the same when any part of the method is changed even if the data doesn't change.

So for optimal results it's better to tune the hyper-parameters for every possible pair of ML model and words embeddings. You can confirm this experimentally: it's very likely that the selected hyper-parameters will be different when you change any part of the method.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.