1
$\begingroup$

I am comparing 2 neural network models. I have used the model to make predictions on unseen data.

  • One model returns an error of 20.9% for y1, 36.6% for y2, 4.53% for y3 on unseen data, and a CV(RMSE) of 19.3.

  • The other returns an error of 15.5% for y1, 33.8% for y2 and 4.83% for y3 on unseen data, and a CV(RMSE) of 31.5.

I'm struggling to interpret the results. A lower CV(RMSE) is better, yet why do I get a much higher error on unseen data?

$\endgroup$
2
  • $\begingroup$ Can you explain how you define "error"? Is that RMSE too or a different error function? And is that error function also your loss function? Also, what do y1, y2 and y3 refer to? $\endgroup$
    – Jonathan
    Commented Apr 2, 2020 at 14:59
  • $\begingroup$ error is calculated as ((measured-predicted)/measured))*100. The loss function was MAE. Y1,2,3 are the predicted outputs from the network. Referring to energy, temperature and relative humidity $\endgroup$
    – milo204
    Commented Apr 2, 2020 at 15:53

1 Answer 1

0
$\begingroup$

When you calculate RMSE error on multi-output models you will normally end up with 3 RMSE's. You can average those errors into one by specifying weights. They basically reflects of bad error on y1, y2 and y3 are for your application.

Even if the outputs are averaged equally, keep in mind that RMSE is "scale sensitive". If y1 value is way higher than y2 and y3 values, it will also produce bigger RMSE.

If you want to get rid of this problem I recommend you to use RMSE% wich is a relative error metric (scale invariant) as mentionned here : https://stats.stackexchange.com/questions/190868/rmse-is-scale-dependent-is-rmse

$\endgroup$
1
  • $\begingroup$ great thank you for your help! $\endgroup$
    – milo204
    Commented Apr 3, 2020 at 18:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.