0
$\begingroup$

I'm trying to build my first models for regression after taking MOOCs on deep learning. I'm currently working on a dataset whose labels are between 0 and 2. Again, this is a regression task, not classification.

The low y values imply that the loss for each sample is quite low, always < 1. My question is then about the relevance of mse as a metric in such a case : since the loss is < 1, squaring it will result in an even smaller value, making the metric value drop very rapidly. In this case, would it be more relevant to use mae ? Or should I multiply the y values so that the order of magnitude of a sample loss would be > 1.

I found this nice article about regression metrics, but didn't find the answer in it. Thanks for your help.

Improve this question
$\endgroup$
4
  • $\begingroup$ What is your objection to having each squared error be lower than the absolute error? If your error is $0.25$, your squared error is $0.0625$; if your error is $0.5$, your squared error is $0.25$. As usual, doubling the error magnitude results in four times the penalty from square error. I do not see any issue except maybe rounding issues when you do math on a computer. $\endgroup$ – Dave Nov 24 '20 at 19:53
  • $\begingroup$ The problem I see is that when you use MSE to asses the model, it is small even if the model fit is not a good one. Best to use rRMSE as I mentioned below for cases like this. $\endgroup$ – Suren Nov 24 '20 at 20:12
  • $\begingroup$ I should have mentioned in the question that I also use mse as a loss function. I thought that maybe the loss value was artificially small and that it could have a bad influence on the model's ability to converge. But I'm quite a noob so my reasoning could also just be wrong. $\endgroup$ – Gwalchaved Nov 25 '20 at 20:40
  • $\begingroup$ See how the convergence is measured in your model. $\endgroup$ – Suren Nov 25 '20 at 21:19
0
$\begingroup$

I'd use relative RMSE $\sqrt{\frac{1}{n} \sum \frac{(Preicted - True)^2}{True^2}}$. In this case, close to 0 implies a good model, regardless of the scale of the true values.

Similarly, you can try relative MAE.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks, I'll try this. Yet another hyperparameter to test ! I'm experiencing what Andrew Ng told in the courses I took: ML is a highly empirical process. $\endgroup$ – Gwalchaved Nov 24 '20 at 14:43
  • $\begingroup$ I added the ^2 to the equation. MSE/RMSE/rRMSE etc. are not hyper-parameters. They are ways to asses your model accuracy. $\endgroup$ – Suren Nov 24 '20 at 20:02
  • $\begingroup$ You're right, that's my fault: I should have also mentioned that I'm using mse as the loss function. So I was wondering if I could be facing an articificially low loss value that could hurt the model's ability to converge. But I know I'm still quite new to this, so I could just be wrong. $\endgroup$ – Gwalchaved Nov 25 '20 at 20:43
0
$\begingroup$

If your only concern is small error values, why not simply scale the output by some constant?

  • The idea would be to multiply all the actual values by some factor e.g. 10*y_actual
  • Next, train your model on the scaled values.
  • To make a prediction in the orginal rang you would have to scale back the outputs: y_scale_orginal = y_prediction / 10
Improve this answer
$\endgroup$
4
  • $\begingroup$ Yes, that's one solution I was considering. As a newbie, I wasn't sure it was a good idea. Thanks. $\endgroup$ – Gwalchaved Nov 24 '20 at 14:40
  • $\begingroup$ This would be equivalent to a unit conversion. Don't like working in light years for molecular biology? Convert to nanometers. However, I do not believe this is necessary; see my comment on the original post. $\endgroup$ – Dave Nov 24 '20 at 19:54
  • $\begingroup$ I don't see the point in this answer; is this essentially equivalent to just multiplying the MSE by a large number. $\endgroup$ – Suren Nov 24 '20 at 20:08
  • $\begingroup$ The questions express a concern about small error values (but does not explicitly express why this is a concern). If your concern is interpretation then I agree use a relative measure. If your concern is that (y-y_pred)^2 is very small which might lead to rounding errors, then this answer is perfectly fine. Although I agree with Dave it should not be necessary $\endgroup$ – Burger Nov 24 '20 at 20:56
0
$\begingroup$

MSE and Standard deviation

Mean squared error, shows us how much error we have over all our points. Indeed the goal is to reduce it, however, in your case, the error yielded would already be small.

One way to understand the relevance of your (MSE) RMSE is to compare it to the standard deviation.

Imagine having a standard deviation lower than your learned model's RMSE, therefore, if you take the mean as a value for all your predictions (X_test), it would be a better answer than trying to predict the value using your estimator.

In other words, imagine using a naive regressor, that gives all your points the mean value. If this estimator is yielding less RMSE than your model that should have learned something, then your model is very bad since the naive estimator beats it.

Start from this logic...

I would love you to think of what I said, however, if you lose hope in figuring it out check this.

Why not use MAE

MAE has its own benefits, therefore, using it randomly is useless. MAE is mostly used when we are dealing with data that has outliers or noise, therefore, we want to try to not give much importance to those spikes in magnitude.

MSE vs. MAE (L2 loss vs L1 loss) In short, using the squared error is easier to solve, but using the absolute error is more robust to outliers. But let’s understand why!

Read here

Improve this answer
$\endgroup$
4
  • $\begingroup$ I think I got it, and it's pretty interesting. It makes me think of what I saw in the courses I took about accuracy for classification in the case of skewed classes. But I don't see how it will help me with my concern of having a mse dropping artificially fast because of my y values scale. $\endgroup$ – Gwalchaved Nov 24 '20 at 14:46
  • $\begingroup$ Everything has a scale, but as long as you can minimize it even more, that would be great! And btw, if your data is between 0 and 2 and lots of your predicted answers have a big residual error, when you square it, it will add up. $\endgroup$ – ombk Nov 24 '20 at 14:48
  • $\begingroup$ @Gwalchaved do not try MAE as suggested by the answer your gave a +1, because MAE is used for robust regression. Read about it. $\endgroup$ – ombk Nov 24 '20 at 14:49
  • $\begingroup$ @ombk, you are wrong about your interpretation of MAE. MAE and MSE are two different ways of estimating model accuracy. Unlike MSE, MAE is less effected by outliers. Whether you want to use MAE or MSE (L1 or L2 error) on model selection depends on how you are going to use your model. $\endgroup$ – Suren Nov 24 '20 at 20:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.