0
$\begingroup$

Is there any thumb of rule which scoring function should be used for e.g. the validation_curve?

Atm I try to study the difference between several optimizers:

validation_curve(grid_best 
                                          , X_train 
                                          , y_train
                                          , param_name = 'Adam', 'SGD', 'RMSprop', 'Adagrad', 'Adadelta', 'Adamax', 'Nadam']
                                          , param_range = param_range 
                                          , cv=tscv
                                          , scoring="explained_variance" 
                                          , verbose = 1
                                          , n_jobs = n_jobs
                )

enter image description here

I use the explained_variance but I think the function has to be minimized cause the values are mostly below zero. That's why I think the explained variance does not make sense here?

edit:

When I use r2 I get the following curve:

nn

Is that normal?

$\endgroup$
1
$\begingroup$

I'd use r2 for validating regression. It'd be much easier to compare different models. Here's wiki, here's sklearn.

$\endgroup$
  • $\begingroup$ Any explanation why R^2 over others? :) $\endgroup$ – Ben Nov 21 '19 at 11:22
  • 1
    $\begingroup$ It's not r square / r^2, it's just r2. It's much easier to compare this metric to other splits/experiments, as it converges to 1, 0 is constant answer, below 0 you're just bad. $\endgroup$ – Piotr Rarus - Reinstate Monica Nov 22 '19 at 9:07
  • $\begingroup$ sure, but why not e.g. rmse? $\endgroup$ – Ben Nov 22 '19 at 9:25
  • $\begingroup$ It's just easier to compare values that limits to 1. With rmse it's harder to interpret results in a meaningful way. $\endgroup$ – Piotr Rarus - Reinstate Monica Nov 22 '19 at 10:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.