0
$\begingroup$

I have created multiple regression models and wanted to choose the best one. One common metric would be RMSE, as you know.

When I looked at the results, second model (RMSE = 0.15) was better able to detect some of the peaks than the first one (RMSE = 0.1). For example look at the two results:

Best model (RMSE = 0.1): enter image description here

Here is the second model's result(RMSE = 0.15): enter image description here The model which is able to detect all the peaks, as much as possible (although its RMSE is not the least among all of the models), is more preferable than a model which has less RMSE but is not able to detect peaks. I searched through the web but didn't find what I was looking for.

Can anyone suggest me a code to evaluate the results of the models, based on their ability to detect peaks better?

Suppose the results are simply 2 arrays:

import numpy as np
np.random.seed(10)
predicted_1 = np.random.rand(10, 1)
predicted_2 = np.random.rand(10, 1)
original = np.random.rand(10, 1)
$\endgroup$
  • $\begingroup$ Are you interested in all of the peaks, or only the largest peaks? What about troughs - what should the metric indicate then? $\endgroup$ – bradS Apr 24 '19 at 11:02
  • $\begingroup$ The model which predicts all of the peaks is more preferable. My model is a weather forecasting one. Hence, it's ability to detect peaks is of high importance for me. It should be able to detect all of the peaks. Large and small ones. The more it detects the peaks, the less the error should be. $\endgroup$ – hyTuev Apr 25 '19 at 9:44
  • $\begingroup$ Hw about estimating correlation? See here stackoverflow.com/questions/6157791/…. Perhaps this can be used along with your RMSE somehow. $\endgroup$ – TwinPenguins Apr 25 '19 at 10:11
  • $\begingroup$ @TwinPenguins Thanks.Using R-squared along with RMSE seems interesting. as discussed here: stats.stackexchange.com/questions/38631/…. $\endgroup$ – hyTuev Apr 25 '19 at 11:17
0
$\begingroup$

I had this problem in my work before and used weighted RMSE and assigned higher weights to the peaks.

Some ideas I took from here https://stats.stackexchange.com/questions/230517/weighted-root-mean-square-error

| improve this answer | |
$\endgroup$
  • $\begingroup$ This seems a suitable approach to my problem. However, I'm not sure what type of weight function to use. The first weight function that came to my mind is to divide the current Y_original by sum of all Y_originals. Could you please explain your weight function? $\endgroup$ – hyTuev Apr 25 '19 at 10:51
  • 1
    $\begingroup$ Yes, you could try Y_original by the sum of all Y_originals, but you need to be aware that these weights favor peaks with bigger Y value. You could try to make a numerical derivative of the signal and give bigger weights to the points with a derivative close to zero. You could do smoothing first if tiny peaks do not matter. The derivative has a known effect of increasing noise. It depends. In my previous work, I used threshold, that assigns weights to the values bigger than the threshold, but the signal was completely different. $\endgroup$ – Tatyana Apr 25 '19 at 11:46
  • $\begingroup$ Thanks. I got the main idea. $\endgroup$ – hyTuev Apr 25 '19 at 15:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.