6
$\begingroup$

Maybe it is a bit general question. I am trying to solve various regression tasks and I try various algorithms for them. For example, multivariate linear regression or an SVR. I know that the output can't be negative and I never have negative output values in my training set, though I could have 0's in it (for example, I predict 'amount of cars on the road' - it can't be negative but can be 0). Rather often I face a problem that I am able to train relatively good algorithm (maybe fit a good regression line to my data) and I have relatively small average squared error on training set. But when I try to run my regression algorithm against new data I sometimes get a negative output. Obviously, I can't accept negative output since it is not a valid value. The question is - what is the proper way of working with such output? Should I think of negative output as a 0 output? Is there any general advice for such cases?

$\endgroup$
  • $\begingroup$ would you consider a function which more accurately reflects the data you have rather than a linear regression model ? $\endgroup$ – image_doctor Feb 1 '15 at 13:26
  • $\begingroup$ I tried different models, but so far all of them sometimes output negative values. This is not really wrong - if you think of linear regression as a line it is possible that it is not able to build a function which will be ALWAYS positive, or even carefully 'touch' the axis (to output 0s). But once again, whatever I tried, for new data I sometimes keep getting negatives. $\endgroup$ – Maksim Khaitovich Feb 1 '15 at 16:39
  • $\begingroup$ Hard to know for definite without sight of the data, but one could imagine a piecewise linear approximation that could be shaped not to go negative, or a high order polynomial perhaps. :) $\endgroup$ – image_doctor Feb 1 '15 at 18:58
4
$\begingroup$

The problem is your model choice, as you seem to recognize. In the case of linear regression, there is no restriction on your outputs. Often this is fine when predictions need to be non-negative so long as they are far enough away from zero. However, since many of your training examples are zero-valued, this isn't the case.

If your data is non-negative and discrete (as in the case with number of cars on the road), you could model using a generalized linear model (GLM) with a log link function. This is known as Poisson regression and is helpful for modeling discrete non-negative counts such as the problem you described. The Poisson distribution is parameterized by a single value $\lambda$, which describes both the expected value and the variance of the distribution.

This results in an approach similar to the one described by Emre in that you are attempting to fit a linear model to the log of your observations.

$\endgroup$
  • $\begingroup$ Hmmm, interesting, thank you! I will try it, are there any good links on GLM\Poisson regression articles which you could suggest? $\endgroup$ – Maksim Khaitovich Feb 3 '15 at 21:13
  • $\begingroup$ I'm not aware of any particularly outstanding general guides on Poisson regression. As a first step, I would recommend looking up GLMs in whatever programming language you are comfortable with. Most implementations provide support for Poisson regression and getting started is often as easy as plugging in your training data and specifying a "Poisson" error distribution or a "log" link function. $\endgroup$ – Ryan J. Smith Feb 4 '15 at 2:40
4
$\begingroup$

A standard trick is to estimate the logarithm of the desired quantity, then take its exponential, which is always positive. The drawback is that the error is optimized for the log, which treats differences in order of magnitude as equal. Another option is to do your regression as usual then project onto the feasible set (use the positive part of the output; $max(0, \cdot)$)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.