0
$\begingroup$

I have created multiple regression models and wanted to choose the best one. One common metric would be RMSE, as you know.

When I looked at the results, the second model (RMSE = 0.15) was better able to detect some of the peaks than the first one (RMSE = 0.1). For example, look at the two results:

Best model (RMSE = 0.1): enter image description here

Here is the second model's result(RMSE = 0.15): enter image description here The model which is able to detect all the peaks, as much as possible (although its RMSE is not the least among all of the models), is more preferable to a model which has less RMSE but is not able to detect peaks. I searched through the web but didn't find what I was looking for.

Can anyone suggest to me a code to evaluate the results of the models, based on their ability to detect peaks better?

Suppose the results are simply 2 arrays:

import numpy as np
np.random.seed(10)
predicted_1 = np.random.rand(10, 1)
predicted_2 = np.random.rand(10, 1)
original = np.random.rand(10, 1)
$\endgroup$
4
  • $\begingroup$ Are you interested in all of the peaks, or only the largest peaks? What about troughs - what should the metric indicate then? $\endgroup$
    – bradS
    Apr 24, 2019 at 11:02
  • $\begingroup$ The model which predicts all of the peaks is more preferable. My model is a weather forecasting one. Hence, it's ability to detect peaks is of high importance for me. It should be able to detect all of the peaks. Large and small ones. The more it detects the peaks, the less the error should be. $\endgroup$
    – hyTuev
    Apr 25, 2019 at 9:44
  • $\begingroup$ Hw about estimating correlation? See here stackoverflow.com/questions/6157791/…. Perhaps this can be used along with your RMSE somehow. $\endgroup$ Apr 25, 2019 at 10:11
  • $\begingroup$ @TwinPenguins Thanks.Using R-squared along with RMSE seems interesting. as discussed here: stats.stackexchange.com/questions/38631/…. $\endgroup$
    – hyTuev
    Apr 25, 2019 at 11:17

1 Answer 1

0
$\begingroup$

I had this problem in my work before and used weighted RMSE and assigned higher weights to the peaks.

Some ideas I took from here https://stats.stackexchange.com/questions/230517/weighted-root-mean-square-error

$\endgroup$
3
  • $\begingroup$ This seems a suitable approach to my problem. However, I'm not sure what type of weight function to use. The first weight function that came to my mind is to divide the current Y_original by sum of all Y_originals. Could you please explain your weight function? $\endgroup$
    – hyTuev
    Apr 25, 2019 at 10:51
  • 1
    $\begingroup$ Yes, you could try Y_original by the sum of all Y_originals, but you need to be aware that these weights favor peaks with bigger Y value. You could try to make a numerical derivative of the signal and give bigger weights to the points with a derivative close to zero. You could do smoothing first if tiny peaks do not matter. The derivative has a known effect of increasing noise. It depends. In my previous work, I used threshold, that assigns weights to the values bigger than the threshold, but the signal was completely different. $\endgroup$
    – Tatyana
    Apr 25, 2019 at 11:46
  • $\begingroup$ Thanks. I got the main idea. $\endgroup$
    – hyTuev
    Apr 25, 2019 at 15:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.