3 votes
Accepted

How to select the optimal beam size for beam search?

Large beam sizes do not lead to improvements but to degradation in the generated text quality, as described in the article Empirical Analysis of Beam Search Performance Degradation in Neural Sequence ...
noe's user avatar
  • 26.6k
2 votes
Accepted

Changing model architecture doesn't impact results

there are several aspects that impact the model results. The architecture is one of them, with a high influence. My guess is that your model is still too shallow (2 layers of 16 neurons each is very ...
Danfper's user avatar
  • 46
1 vote

which hyperparameters are returned as best in cross validation?

Sometimes these values are not used anymore, they are just used to calculate the best model for each fold and its prediction error (on the validation folds). And if the error satisfies you, at each ...
skan's user avatar
  • 185
1 vote

Is hyperparameter tuning done on training or validation data set?

Tuning hyperparameters on the training set is generally not as critical as training on the test set, but overall increases the risk of overfitting. Alternatively, use cross-validation to tune the ...
Martin Benes's user avatar
1 vote
Accepted

Why is it so common to focus only validation performance during hyper-parameter optimization

The simple answer might be that we want our model to be the best in a real-world application. Because of that, in most cases we don't care about training loss as it's not a good sample of real-world ...
Tomasz Witkowski's user avatar
1 vote

Hyperparameter tuning

This is called data leakage. She should choose hyper parameters based upon the training data. And then report performance when evaluated using the test data. Being able to make good predictions on ...
J_H's user avatar
  • 1,045
1 vote

Add tuning stage to DVC pipeline

I'm from the DVC team. This is a great question. Do you want to save each trial in your search as its own experiment? It could be overkill and might make sense to start with saving the entire search ...
Dave Berenbaum's user avatar
1 vote

Ordering of Train/Val/Test set use in hyperparameter tuning

You are asking about a model's ability to generalize to unseen examples. Consult Abu-Mostafa, et al., Learning From Data, § 5.3 Data Snooping. If a data set has affected any step in the learning ...
J_H's user avatar
  • 1,045

Only top scored, non community-wiki answers of a minimum length are eligible