40
$\begingroup$

I am using a standard linear regression using scikit-learn in python. However, I would like to force the weights to be all positive for every feature (not negative), is there any way I can accomplish that? I was looking in the documentation but could not find a way to accomplish that. I understand I may not get the best solution, but I need the weights to be non-negative.

$\endgroup$
31
$\begingroup$

What you are looking for, is the Non-negative least square regression. It is a simple optimization problem in quadratic programming where your constraint is that all the coefficients(a.k.a weights) should be positive.

Having said that, there is no standard implementation of Non-negative least squares in Scikit-Learn. The pull request is still open.

But, looks like Scipy has implemented the same.

PS: I haven't tried the scipy version. I found it solely by googling around.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ what about ridge regression where it forced to positive? $\endgroup$ – Charlie Parker Jan 27 '18 at 20:03
21
$\begingroup$

I use a workaround with Lasso on Scikit Learn (It is definitely not the best way to do things but it works well). Lasso has a parameter positive which can be set to True and force the coefficients to be positive. Further, setting the Regularization coefficient alpha to lie close to 0 makes the Lasso mimic Linear Regression with no regularization. Here's the code:

from sklearn.linear_model import Lasso
lin = Lasso(alpha=0.0001,precompute=True,max_iter=1000,
            positive=True, random_state=9999, selection='random')
lin.fit(X,y)
| improve this answer | |
$\endgroup$
1
$\begingroup$

There are is a constrained least squares method scipy.optimize.lsq_linear. Another option is to use an optimizing solver for Python. Here is one of the options (Gekko) that I maintain that includes coefficient constraints.

# Constrained Multiple Linear Regression
import numpy as np
nd = 100 # number of data sets
nc = 5   # number of inputs
x = np.random.rand(nd,nc)
y = np.random.rand(nd)

from gekko import GEKKO
m = GEKKO(remote=False); m.options.IMODE=2
c  = m.Array(m.FV,nc+1)
for ci in c:
    ci.STATUS=1
    ci.LOWER=0
xd = m.Array(m.Param,nc)
for i in range(nc):
    xd[i].value = x[:,i]
yd = m.Param(y); yp = m.Var()
s =  m.sum([c[i]*xd[i] for i in range(nc)])
m.Equation(yp==s+c[-1])
m.Minimize((yd-yp)**2)
m.solve(disp=True)
a = [c[i].value[0] for i in range(nc+1)]
print('Solve time: ' + str(m.options.SOLVETIME))
print('Coefficients: ' + str(a))

It uses the nonlinear solver IPOPT to solve the problem. It is a good option for problems that aren't too large because there is some waisted computational effort on calculating exact 1st and 2nd derivatives for possible nonlinear functions. It may be faster for larger problems with the APOPT solver with m.options.SOLVER=1.

| improve this answer | |
$\endgroup$
0
$\begingroup$

Here is an example of why you would want to do it (and approximately how).

I have 3 predictive models of housing prices: linear, gradient boosting, neural network.

I want to blend them into a weighted average and find the best weights.

I run linear regression, and I get a solution with weights like -3.1, 2.5, 1.5, and some intercept.

So what I do instead using sklearn is

blendlasso = LassoCV(alphas=np.logspace(-6, -3, 7),
                     max_iter=100000,
                     cv=5,
                     fit_intercept=False,
                     positive=True)

And I get positive weights that sum (very close) to 1. In my example I want the alpha that works best out-of-sample so I use LassoCV with cross-validation.

The sklearn docs state that you shouldn't set alpha to 0 for numerical reasons, however you can also use straight Lasso() and set the alpha parameter as low as you can get away with to get a reasonable answer.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Your work converts LassoCV to Linear Regression (if it not exactly, maybe approximately) $\endgroup$ – Redhwan Oct 12 at 12:47
  • $\begingroup$ @Redhwan yes, if you set the lasso alphas close enough to 0 it approaches linear regression, with weights constrained to be positive, so it's a hack, and really the only thing it adds to adarsh-chavakula's answer above is it lets you try additional alphas $\endgroup$ – Rocky McNuts Oct 13 at 17:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.