I happily created my first NN and performed hyperparameter optimization through GridSearchCV. I just don't know what to do next. Do I have to fit it again with the best parameters GridSearchCV() revealed? is there an elegant way to do so? Otherwise, how to proceed?

def create_model(...

    model.compile(loss='mean_squared_error', optimizer=optimizer, metrics=['accuracy'])            
    return model

model = KerasRegressor(build_fn=create_model, verbose=0)

> hypparas
{'batch_size': [2, 6], 'optimizer': ['Adam', 'sgd'], 'opt_par': [0.5, 0.8]}

                    , param_distributions=hypparas 
                    , n_jobs=1   
                    , n_iter=20
                    , cv=3 

grid_result = grid_obj.fit(X_train1, y_train1, callbacks = [time_callback])

print("Best: %f using %s" %  (grid_result.best_score_, grid_result.best_params_), "\n")

means = grid_result.cv_results_['mean_test_score']
stds = grid_result.cv_results_['std_test_score']
params = grid_result.cv_results_['params']

for mean, stdev, param in zip(means, stds, params):
    print("%f (%f) with: %r" % (mean, stdev, param))

    Best: -0.941568 using {'optimizer': 'Adam', 'opt_par': 0.8, 'batch_size': 2} 

    -1.725617 (0.620383) with: {'optimizer': 'Adam', 'opt_par': 0.5, 'batch_size': 2}
    -1.595137 (0.224487) with: {'optimizer': 'sgd', 'opt_par': 0.5, 'batch_size': 2}
    -0.941568 (0.149151) with: {'optimizer': 'Adam', 'opt_par': 0.8, 'batch_size': 2}
    -1.338372 (0.523434) with: {'optimizer': 'sgd', 'opt_par': 0.8, 'batch_size': 2}
    -1.094907 (0.121018) with: {'optimizer': 'Adam', 'opt_par': 0.5, 'batch_size': 6}
    -1.588476 (0.569475) with: {'optimizer': 'sgd', 'opt_par': 0.5, 'batch_size': 6}
    -1.443133 (0.342028) with: {'optimizer': 'Adam', 'opt_par': 0.8, 'batch_size': 6}
    -1.275414 (0.331939) with: {'optimizer': 'sgd', 'opt_par': 0.8, 'batch_size': 6}

You can use grid_obj.predict(X) or grid_obj.best_estimator_.predict(X) to use the tuned estimator. However, I suggest you to get this _best_estimator and train it again with the full set of data, because in GridSearchCV, you train with K-1 folds and you lost 1 fold to test. More data, better estimates, right?

  • $\begingroup$ Thanks a lot! And again, what follows after fitting again with 'best_estimator_' ? :) Should I do some evaluation or other stuff or just going straight over to prediction? $\endgroup$
    – Ben
    Oct 16 '19 at 14:04
  • 1
    $\begingroup$ You can always go over to prediction. But depending on your purpose to fit a ML model, you may have to assess your algorithm with other evaluation plots. $\endgroup$ Oct 16 '19 at 14:06
  • $\begingroup$ Thanks! I've seen various procedures and also different evaluation/validation curves and so on and also specific libraries for that (have some bugs atm which prevent me from installing them..), could you therefore provide 1-2 useful sources where I can follow some useful guidelines? $\endgroup$
    – Ben
    Oct 16 '19 at 14:15
  • 1
    $\begingroup$ Andrew Ng lessons on ML (lecture 10) is a very good source. $\endgroup$ Oct 16 '19 at 14:19
  • $\begingroup$ Thank you! As I'm roaming in this field in general atm, I stumbled over dkopczyk.quantee.co.uk/hyperparameter-optimization . There, they use/fit CV differently, as far as I understand. Given 5 folds, they fit four times, instead of two (as it should be?). $\endgroup$
    – Ben
    Oct 17 '19 at 7:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.