24
$\begingroup$

I am using a standard linear regression using scikit-learn in python. However, I would like to force the weights to be all positive for every feature (not negative), is there any way I can accomplish that? I was looking in the documentation but could not find a way to accomplish that. I understand I may not get the best solution, but I need the weights to be non-negative.

$\endgroup$
26
$\begingroup$

What you are looking for, is the Non-negative least square regression. It is a simple optimization problem in quadratic programming where your constraint is that all the coefficients(a.k.a weights) should be positive.

Having said that, there is no standard implementation of Non-negative least squares in Scikit-Learn. The pull request is still open.

But, looks like Scipy has implemented the same.

PS: I haven't tried the scipy version. I found it solely by googling around.

$\endgroup$
  • 1
    $\begingroup$ what about ridge regression where it forced to positive? $\endgroup$ – Charlie Parker Jan 27 '18 at 20:03
14
$\begingroup$

I use a workaround with Lasso on Scikit Learn (It is definitely not the best way to do things but it works well). Lasso has a parameter positive which can be set to True and force the coefficients to be positive. Further, setting the Regularization coefficient alpha to lie close to 0 makes the Lasso mimic Linear Regression with no regularization. Here's the code:

from sklearn.linear_model import Lasso
lin = Lasso(alpha=0.0001,precompute=True,max_iter=1000,
            positive=True, random_state=9999, selection='random')
lin.fit(X,y)
$\endgroup$
1
$\begingroup$

Sorry I don't have enough posts to comment yet, but why on earth do you want that?

To select weights in a linear regression is to minimize the cost function (sum of squared errors).

The equation for a linear regression is given by: $$ \hat{Y} = w_0+w_1\hat{x_1} + ... + w_k\hat{x_k} $$ Obviously each feature ($ x_i $) enters the equation linearly through a weighting ($w_i$), and are independent to each other.

If weights turn out to be negative when fitted, it means that there is a negative relationship between the associated input variable ($x_i$) and the target $Y$.

By imposing a non-negativity constraint on the weights, means that the optimal weight you use on negatives is 0. The implication of this is that you are no longer using that feature in the equation at all. - By including the feature with a positive weight, you are strictly making the prediction worse. Since this is a linear relationship, and the input variables are independent to each other, changing a weight of one variable is not going to affect any other variables, which is why you can simply change one weight to 0 without worrying about the others.

So to answer your question, what you could do is take the array of weights and set = 0 where they are < 0 (after fitting the model), and this should be fairly trivial to do in python.

Here's a brief example in python & sklearn:

import numpy as np
from sklearn.linear_model import LinearRegression

# make some data:
x1 = np.array(np.random.rand(5))
x2 = np.array(np.random.rand(5))
# stack together - to use as input
x = np.column_stack((x1,x2))
# create targets TRUE relationship
y = np.array(2*x1 - 0.5*x2)

# create & fit model...
k = LinearRegression()
k.fit(x,y)

# show the fitted weights
print(k.coef_)

# change weights to be 0 for nonzeros...
k.coef_ = [np.max([c,0]) for c in k.coef_]

# show new weights:
print(k.coef_)

But yeah, I highly advise against doing this, and really can't understand your reasoning for wanting this.

$\endgroup$
  • 1
    $\begingroup$ Hi. Welcome to the site. Nice answer :) but why on earth do you want that : Exactly. Forcing the coeffs. to be all positive doesn't make sense! $\endgroup$ – Dawny33 Apr 11 '17 at 4:27
  • 4
    $\begingroup$ Thanks! :). Pretty stoked there's a specific place for data science on stack exchange now, rather than either overflow or crossvalidated. $\endgroup$ – MaximilianP Apr 11 '17 at 4:32
  • 3
    $\begingroup$ Thank you for your answer, but simply setting the negative to zero is NOT the same as forcing them to be non-negative. You could have a coefficient to be 6 and one to be -5, but, possibly, by forcing them to be positive while calculating them, the first would be 1 and the other 0, not 6 and 0. $\endgroup$ – user Apr 11 '17 at 11:22
  • 3
    $\begingroup$ The problem is that, for physical reasons, the coefficients should be positive. I am trying to understand the total contribution of each feature, and which ones are most important. Some times there are reasons ;) $\endgroup$ – user Apr 11 '17 at 11:26
  • 16
    $\begingroup$ Definitely not a nice answer. Making assumptions about the weights is actually a standard procedure in both statistics and ML to stabilize ill systems. The very simple case of regularization (coefficients drawn from a normal distribution). The example of your coefficients being time units would require them to be positive. Picture trying to estimate read and write timings (of multiple devices) from coupled latency equations. The design matrix would be binary(and unit-less) and both coefficients and target values would be time. $\endgroup$ – Francisco Vargas May 25 '17 at 13:01
0
$\begingroup$

Here is an example of why you would want to do it (and approximately how).

I have 3 predictive models of housing prices: linear, gradient boosting, neural network.

I want to blend them into a weighted average and find the best weights.

I run linear regression, and I get a solution with weights like -3.1, 2.5, 1.5, and some intercept.

So what I do instead using sklearn is

blendlasso = LassoCV(alphas=np.logspace(-6, -3, 7),
                     max_iter=100000,
                     cv=5,
                     fit_intercept=False,
                     positive=True)

And I get positive weights that sum (very close) to 1. In my example I want the alpha that works best out-of-sample so I use LassoCV with cross-validation.

The sklearn docs state that you shouldn't set alpha to 0 for numerical reasons, however you can also use straight Lasso() and set the alpha parameter as low as you can get away with to get a reasonable answer.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.