2
$\begingroup$

I happily created my first NN and performed hyperparameter optimization through GridSearchCV. I just don't know what to do next. Do I have to fit it again with the best parameters GridSearchCV() revealed? is there an elegant way to do so? Otherwise, how to proceed?

def create_model(...

    model.compile(loss='mean_squared_error', optimizer=optimizer, metrics=['accuracy'])            
    return model

model = KerasRegressor(build_fn=create_model, verbose=0)

> hypparas
{'batch_size': [2, 6], 'optimizer': ['Adam', 'sgd'], 'opt_par': [0.5, 0.8]}

GridSearchCV(estimator=model 
                    , param_distributions=hypparas 
                    , n_jobs=1   
                    , n_iter=20
                    , cv=3 
                    )

grid_result = grid_obj.fit(X_train1, y_train1, callbacks = [time_callback])


print("Best: %f using %s" %  (grid_result.best_score_, grid_result.best_params_), "\n")

means = grid_result.cv_results_['mean_test_score']
stds = grid_result.cv_results_['std_test_score']
params = grid_result.cv_results_['params']

for mean, stdev, param in zip(means, stds, params):
    print("%f (%f) with: %r" % (mean, stdev, param))

    Best: -0.941568 using {'optimizer': 'Adam', 'opt_par': 0.8, 'batch_size': 2} 

    -1.725617 (0.620383) with: {'optimizer': 'Adam', 'opt_par': 0.5, 'batch_size': 2}
    -1.595137 (0.224487) with: {'optimizer': 'sgd', 'opt_par': 0.5, 'batch_size': 2}
    -0.941568 (0.149151) with: {'optimizer': 'Adam', 'opt_par': 0.8, 'batch_size': 2}
    -1.338372 (0.523434) with: {'optimizer': 'sgd', 'opt_par': 0.8, 'batch_size': 2}
    -1.094907 (0.121018) with: {'optimizer': 'Adam', 'opt_par': 0.5, 'batch_size': 6}
    -1.588476 (0.569475) with: {'optimizer': 'sgd', 'opt_par': 0.5, 'batch_size': 6}
    -1.443133 (0.342028) with: {'optimizer': 'Adam', 'opt_par': 0.8, 'batch_size': 6}
    -1.275414 (0.331939) with: {'optimizer': 'sgd', 'opt_par': 0.8, 'batch_size': 6}
$\endgroup$

1 Answer 1

1
$\begingroup$

You can use grid_obj.predict(X) or grid_obj.best_estimator_.predict(X) to use the tuned estimator. However, I suggest you to get this _best_estimator and train it again with the full set of data, because in GridSearchCV, you train with K-1 folds and you lost 1 fold to test. More data, better estimates, right?

$\endgroup$
5
  • $\begingroup$ Thanks a lot! And again, what follows after fitting again with 'best_estimator_' ? :) Should I do some evaluation or other stuff or just going straight over to prediction? $\endgroup$
    – Ben
    Commented Oct 16, 2019 at 14:04
  • 1
    $\begingroup$ You can always go over to prediction. But depending on your purpose to fit a ML model, you may have to assess your algorithm with other evaluation plots. $\endgroup$ Commented Oct 16, 2019 at 14:06
  • $\begingroup$ Thanks! I've seen various procedures and also different evaluation/validation curves and so on and also specific libraries for that (have some bugs atm which prevent me from installing them..), could you therefore provide 1-2 useful sources where I can follow some useful guidelines? $\endgroup$
    – Ben
    Commented Oct 16, 2019 at 14:15
  • 1
    $\begingroup$ Andrew Ng lessons on ML (lecture 10) is a very good source. $\endgroup$ Commented Oct 16, 2019 at 14:19
  • $\begingroup$ Thank you! As I'm roaming in this field in general atm, I stumbled over dkopczyk.quantee.co.uk/hyperparameter-optimization . There, they use/fit CV differently, as far as I understand. Given 5 folds, they fit four times, instead of two (as it should be?). $\endgroup$
    – Ben
    Commented Oct 17, 2019 at 7:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.