# How to force weights to be non-negative in Linear regression

I am using a standard linear regression using scikit-learn in python. However, I would like to force the weights to be all positive for every feature (not negative), is there any way I can accomplish that? I was looking in the documentation but could not find a way to accomplish that. I understand I may not get the best solution, but I need the weights to be non-negative.

What you are looking for, is the Non-negative least square regression. It is a simple optimization problem in quadratic programming where your constraint is that all the coefficients(a.k.a weights) should be positive.

Having said that, there is no standard implementation of Non-negative least squares in Scikit-Learn. The pull request is still open.

But, looks like Scipy has implemented the same.

PS: I haven't tried the scipy version. I found it solely by googling around.

• what about ridge regression where it forced to positive? – Charlie Parker Jan 27 '18 at 20:03

I use a workaround with Lasso on Scikit Learn (It is definitely not the best way to do things but it works well). Lasso has a parameter positive which can be set to True and force the coefficients to be positive. Further, setting the Regularization coefficient alpha to lie close to 0 makes the Lasso mimic Linear Regression with no regularization. Here's the code:

from sklearn.linear_model import Lasso
lin = Lasso(alpha=0.0001,precompute=True,max_iter=1000,
positive=True, random_state=9999, selection='random')
lin.fit(X,y)


Here is an example of why you would want to do it (and approximately how).

I have 3 predictive models of housing prices: linear, gradient boosting, neural network.

I want to blend them into a weighted average and find the best weights.

I run linear regression, and I get a solution with weights like -3.1, 2.5, 1.5, and some intercept.

So what I do instead using sklearn is

blendlasso = LassoCV(alphas=np.logspace(-6, -3, 7),
max_iter=100000,
cv=5,
fit_intercept=False,
positive=True)


And I get positive weights that sum (very close) to 1. In my example I want the alpha that works best out-of-sample so I use LassoCV with cross-validation.

The sklearn docs state that you shouldn't set alpha to 0 for numerical reasons, however you can also use straight Lasso() and set the alpha parameter as low as you can get away with to get a reasonable answer.