I am estimating water depth with satellite data (predicted value) and would like to validate my result using bathymetry lidar data collected on the field and believed to be more accurate (observed value). I have different observations at each water depth. For example, number of observations at water depth range of 0-10 m are 300, where as values at deeper depth range (10 - 20 m) are less (~50 points). I have been using RMSE (as I would like to penalize larger error) to measure my accuracy but wondering if there is a better error metric out that is not sensitive to number of observations. In other words, for water depth 10 - 20 m with 50 points, I have RMSE of ~6m, and I was thinking the value could be lower if I have more observations. Where as for shallow water depth (0-10 m), my RMSE are much lower, perhaps because I have lots of observation.
$\begingroup$
$\endgroup$
3
-
$\begingroup$ Why do you think that your metric depends on the number of observations? $\endgroup$– DaveCommented May 10, 2020 at 3:40
-
$\begingroup$ @Dave for water depth 25-30 m, I have ~50 observations, and for other lower depth ranges I have as high as 300 - I have an increment of 5m (i.e., 0-5, 5-10, 10-15...). RMSE is ~ 1m for other depth ranges, but ~6m for depth range 25-30. I know error is overall and will depend on number of points that are predicted well. I want a metric that will take the large error of ~6m at 25-30 m depth range into account. $\endgroup$– ChrisCommented May 10, 2020 at 4:02
-
3$\begingroup$ RMSE is a MEAN error. It divides by the number of observations: $$RMSE=\sqrt{\dfrac{\sum_{i=1}^N\big(y_i-\hat{y}_i\big)^2}{N}}$$As far as wanting to penalize large errors at shallow depths more than large errors as deep depths, that makes more sense. Mean absolute percentage error sounds like a good place to start. Its known issues are discussed in its Wikipedia article, and alternatives are proposed. $\endgroup$– DaveCommented May 10, 2020 at 4:17
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Since you have a regression problem, you can use many possible evaluation metrics. Other regression evaluation metrics include mean squared error (MSE), mean absolute value (MAE), and mean absolute percentage error (MAPE).
However, it sounds like the issue is variance across different target values. If you switch to Bayesian Regression, it will better model variance and uncertainty across the range of target values.
answered Feb 26, 2021 at 2:50