I am using a standard linear regression using scikit-learn in python. However, I would like to force the weights to be all positive for every feature (not negative), is there any way I can accomplish that? I was looking in the documentation but could not find a way to accomplish that. I understand I may not get the best solution, but I need the weights to be non-negative.
What you are looking for, is the Non-negative least square regression. It is a simple optimization problem in quadratic programming where your constraint is that all the coefficients(a.k.a weights) should be positive.
But, looks like Scipy has implemented the same.
PS: I haven't tried the scipy version. I found it solely by googling around.
I use a workaround with Lasso on Scikit Learn (It is definitely not the best way to do things but it works well). Lasso has a parameter
positive which can be set to
True and force the coefficients to be positive. Further, setting the Regularization coefficient
alpha to lie close to 0 makes the Lasso mimic Linear Regression with no regularization. Here's the code:
from sklearn.linear_model import Lasso lin = Lasso(alpha=0.0001,precompute=True,max_iter=1000, positive=True, random_state=9999, selection='random') lin.fit(X,y)
Here is an example of why you would want to do it (and approximately how).
I have 3 predictive models of housing prices: linear, gradient boosting, neural network.
I want to blend them into a weighted average and find the best weights.
I run linear regression, and I get a solution with weights like -3.1, 2.5, 1.5, and some intercept.
So what I do instead using sklearn is
blendlasso = LassoCV(alphas=np.logspace(-6, -3, 7), max_iter=100000, cv=5, fit_intercept=False, positive=True)
And I get positive weights that sum (very close) to 1. In my example I want the alpha that works best out-of-sample so I use LassoCV with cross-validation.
The sklearn docs state that you shouldn't set alpha to 0 for numerical reasons, however you can also use straight Lasso() and set the alpha parameter as low as you can get away with to get a reasonable answer.