1
$\begingroup$

If I want to do a hyperparameter optimisation on a dataset using e.g. hyperband or random search, I note that some of the models being randomly chosen seem to have rather good R2 scores, MSE etc.

I then get excited and create a neural network of the same description and train it on the same thing, same number of epochs, all seemingly same way etc. and I get quite a bit worse values.

Is there any good reason for this behaviour (e.g. are randomly chosen weights to start off that important for the final output? (???)) (and if this is true what causes might there be for this that explains it as "expected behaviour"?), or is it entirely unexpected and I should be thinking I've done something wrong? (and if this is true, what causes might there be?)

Thanks!


Edit: code layout

#build hypermodel 
class MyHyperModel(kt.HyperModel): 
  def build_model(self, hp): model = Sequential() ... 
  #do hyperband search 
  tuner = kt.tuners.Hyperband( MyHyperModel().build_model, objective='loss', ...) 
  tuner.search(x=X_train, y=y_train, epochs=...) 
  #get best 
  best_model = tuner.get_best_models(num_models=1) 
  #build best model and fit - same data 
  best_model.fit(x=X_train, y=y_train, epochs=...) 
$\endgroup$

1 Answer 1

1
$\begingroup$

Imho the most likely option is overfitting.

Hyper-parameter tuning is a kind of training, in the sense that we try to find the values which give the best results on a particular dataset. Like any training process overfitting can happen, and it's more likely to happen if the dataset is not representative enough (in particular too small), or if there are too many possible combinations of parameters.

Basically if one tries "too hard" to find the best hyper-parameters, then one is more likely to end up with hyper-parameters values which just happen to work by chance on this dataset.

$\endgroup$
7
  • $\begingroup$ Ah thanks. I mean that if I literally replicate the hyperparameter optimisation training, I get a different result. I.e. rather than train on a different dataset/check a score to one which it hasn't seen, I mean that the scores for training (and then checking validation score) on the same two datasets are different if hyperband/random hyperparameter optimisation is doing the training (and finding the best hyperparams) or if I manually use those best hyperparams myself, to that same data. I.e. in my mind the same models, same data, different scores. $\endgroup$
    – Socorro
    Commented Jun 25, 2022 at 17:25
  • $\begingroup$ @Socorro Do you mean that the dataset is exactly the same, as in exactly the same set of instances? i.e. not even a random splitting? In my hypothesis I was assuming that the second training is not applied to exactly the same subset of data. $\endgroup$
    – Erwan
    Commented Jun 25, 2022 at 21:16
  • $\begingroup$ Yes. Here's my overview of code: #build hypermodel class MyHyperModel(kt.HyperModel): def build_model(self, hp): model = Sequential() ... #do hyperband search tuner = kt.tuners.Hyperband( MyHyperModel().build_model, objective='loss', ...) tuner.search(x=X_train, y=y_train, epochs=...) #get best best_model = tuner.get_best_models(num_models=1) #build best model and fit - same data best_model.fit(x=X_train, y=y_train, epochs=...) The R2 values the model achieved, same data, in tuner.search aren't loosely the same as .fit. MSE's are more robust. $\endgroup$
    – Socorro
    Commented Jun 26, 2022 at 2:55
  • $\begingroup$ Layout of the code above isn't as good as it could be ... apologies for that. $\endgroup$
    – Socorro
    Commented Jun 26, 2022 at 2:56
  • $\begingroup$ @Socorro hope you don't mind: I copied the code inside the question, more readable this way. $\endgroup$
    – Erwan
    Commented Jun 26, 2022 at 8:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.