4
$\begingroup$

Am I doing regression task with 7 dependent variables and 10000 data points. The SGD gives me 22% of mean absolute percentage error on test and train dataset. And Least Squares method using numpy scipy.optimize.least_squares give me only 58% (I have tried different settings.). I thought that Least Squares should give the same or better performance with such size dataset. What can be the reasons?

$\endgroup$
  • $\begingroup$ What is your loss function for running SGD? $\endgroup$ – Neil Slater Jun 16 '17 at 19:01
5
$\begingroup$

The reason is that the two metrics, mean absolute percentage error (MAPE) and mean square error (MSE) are optimising to different targets. Improving one can be done at the expense of the other.

As a simple example, consider this data:

x = [ 0,  1,  2,  3,  4,  5]
y = [ 3,  5, 10, 10, 11, 15]

The best fit mean squared error (MSE) for a line on this data is $\hat{y} = 2.23x + 3.43$, which has MSE of $1.18$, and a mean absolute percentage error (MAPE) of $11.0$%.

The best fit mean absolute percentage loss for a line on this data is $\hat{y} = 2.35x + 2.99$, which has a MSE of $1.24$, and a MAPE of $8.34$%.

You can see that optimising for MAPE gives a worse MSE, and vice-versa.

The difference can get extreme when there is a large range for y values (in terms of orders of magnitude covered), because optimising for MAPE will favour being more accurate on small values at the expense of larger ones. So if we change y to be:

y = [ 1,  2, 10, 10, 11, 20]

Then optimising mean abs percentage gives the line $\hat{y} = 3.78x + 1.05$ with MSE $7.09$ and MAPE $21.9$%. But optimising for mean square error gives $\hat{y} = 3.49x + 0.286$ with MSE $4.56$ and MAPE $62.9$% - this is a larger difference, and I suspect that your data has a large range of target variable causing a similar effect.


You can potentially get closer results using the Least Squares regressor by using a transformed target variable $z = log(y)$ and transforming back at the end. This still won't be quite the same, but it does reduce the difference significantly - in my last example if I try this, I get MAPE $24.2$% - compared to $21.9$% for optimising MAPE directly.

|improve this answer|||||
$\endgroup$
  • $\begingroup$ Good answer and I apreciating your help, so I will accept, but the reason was actually the other. In the begining I have to notice that we can use MAPE in the least squares not only MSE. So the reason: I have already log transformed the data and then for gradient descent I have used a trick. I have used three losses in three steps ro learn better: mean relative squared error - > abs relative error - > relative error of exp of prediction / exp of real value(to go back from log) and for least squares I have used just mean squared relative error as loss without exp. And that is important. $\endgroup$ – Brans Ds Jun 16 '17 at 20:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.