2
$\begingroup$

We have MSE and RMSE as evaluation metrics for regression problems. I have for some problems people use Weighted Mean Squared Error (WMSE) as the evaluation metrix.

Below is the WMSE formula: enter image description here Can anyone explain me the real need of WMSE and when not to use MSE.

Thanks in advance.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
4
$\begingroup$

Weighting MSE is a way to give more importance to some prediction errors than to others in the overall score. This is useful if you are using MSE as a performance metric for your model, especially during the model training (loss function) or validation (hyper-parameter setting).

In the case you cite as example, more importance is given to cases with more clicks. If you use this WMSE as performance metric for validation, your model will tend to be better on cases with high number of clicks, than if you had used MSE.

Just a note on your example: the used squared error ($(predictedClicks - observedClicks)^2$) is an absolute squared error (I would have expected a relative error), and therefore already increases with the number of clicks. So in this case, weighting that way reinforces the performance on cases with high number of clicks.

Improve this answer
$\endgroup$
0
1
$\begingroup$

I am running into exactly this problem. I am looking at correlations between load and throughput. Literally (load,throughput) pairs. But if you are measuring a real system that is usually at load 1000, you may never get data for load in low values like load 1,2,3, etc. (ie: what is Google throughput at 1 user only? We will never get data for it.) So, what I want to do is to properly weigh all the data that I DO have. In the observed data, every observation has a weight, which is basically the number of times to include it in the MSE. I can't make up observations for "Google throughput at load 1", because that never happens.

It means that the equation for MSE needs proper weighting. If you think about it, this is generally the case for doing statistics! If you measure f(20)=100 1000x, but only measure f(5)=2.3 2x, and f(1) is never actually measured... So, in the rates. ie: say it's bytes/sec... keep separate byte and duration counts. You must weigh 2.3 byte/sec as 2 measurements if it happened twice.

Improve this answer
$\endgroup$
1
  • $\begingroup$ note that the 1/N term is dividing by the weight, where there is an assumption that everything is weight 1. But if I am comparing the model to the data (X[n] - X_n)^2 ... but the observation happened 3 times... it actually adds (X[n] - X_n)^2 * w[n]. Note that even works if you have repeated measurements, like: [(2,3.5),5],[(2,3.4)] ... where f(2)=3.5 in 5 observations, and f(2)=3.4 in 4 observations. The length of time during a byte/sec rate could be the weight. When you do 1/N, that's dividing by the weight. If you track a total weight W, you can divide by that. $\endgroup$
    – Rob
    Commented Jan 29, 2021 at 1:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.