1
$\begingroup$

Which specific performance evaluation metrics are used in training, validation and testing and why? I am thinking error metrics (RMSE, MAE, MSE) are used in validation, and testing should use a wide variety of metrics? I don't think performance is evaluated in training, but not 100% sure.

Specifically I am actually after deciding when to use (i.e. in training, validation or testing) correlation coefficient, RMSE, MAE and others for numeric data (e.g. Willmott's Index of Agreement, Nash-Sutcliffe coefficient, etc.)

Sorry about this being broad - I have actually been asked to define it generally (i.e. not for a specific dataset). But datasets I have been using have all numeric continuous values with supervised learning situations.

Generally I am using performance evaluation for environmental data where I am using ANN to model. I have continuous features and am predicting a continuous variable.

$\endgroup$
  • 2
    $\begingroup$ It's a very broad Question and the short answer is, it depends on the problem at hand... $\endgroup$ – Aditya Apr 14 '18 at 13:12
  • $\begingroup$ Sorry about it being broad - I have actually been asked to define it generally (i.e. not for a specific dataset). But datasets I have been using are numeric continuous values with supervised learning situations $\endgroup$ – user9645302 Apr 14 '18 at 13:19
  • 1
    $\begingroup$ Take a look at here. $\endgroup$ – Media Apr 14 '18 at 13:52
  • $\begingroup$ Thanks for that - sorry, I am actually after deciding when to use specifically correlation coefficient, RMSE, MAE and others for numeric data (e.g. Willmott's Index of Agreement, Nash-Sutcliffe coefficient, etc.) $\endgroup$ – user9645302 Apr 14 '18 at 13:57
  • $\begingroup$ Depending on your task they may vary. For regression tasks you can use symmetric cost functions like RSS if you have evenly important concepts. $\endgroup$ – Media Apr 14 '18 at 14:19