3
$\begingroup$

I'm using sklearn.linear_model.Ridge to use ridge regression to extract the coefficients of a polynomial. However, some of the coefficients have physical constraints that require them to be negative. Is there a way to impose a constraint on those parameters? I haven't spotted one in the documentation...

As a workaround of sorts, I have tried making many fits using different complexity parameters (see toy code below) and selecting the one with coefficients that satisfy the physical constraint, but this is too unreliable to use in production.

# Preliminaries
from sklearn.linear_model import Ridge
n_alphas = 2000
alphas = np.logspace(-15,3,n_alphas)
# Perform fit
fits = {}
for alpha in alphas:
   temp_ridge = Ridge(alpha, fit_intercept=False)
   temp_ridge.fit(indep_training_data, dep_training_data)
   temp_ridge_R2 = temp_ridge.score(indep_test_data, dep_test_data)
   fits[alpha] = [temp_ridge, temp_ridge_R2]

Is there a way to impose a sign constraint on some of the parameters using ridge regression? Thanks!

$\endgroup$
1
  • 1
    $\begingroup$ I don’t think that it is possible to do this since this would impose quite some severe restrictions to the estimator. Sounds like you do some kind of causal inference? Be careful with interpreting shrunken coefs (and sign) since they might lose their causal interpretation. I‘m not really into the topic, just want to let you know. jrnold.github.io/intro-methods-notes/… $\endgroup$
    – Peter
    Aug 27 at 19:28
3
$\begingroup$

I am assuming a linear regression of the form

$$y = w_0x_0 + w_1x_1+ \ldots w_px_p + \varepsilon.$$

If we combine all output observations into a single vector $\mathbf{y}$ and represent the data matrix with an 1-column from left as $\mathbf{X}$, then we can express the linear regression

$$\mathbf{y} = \mathbf{X}\mathbf{w} + \mathbf{\varepsilon},$$

in which $\mathbf{w}=[w_0, w_1,\ldots,w_p]^T$ and $\varepsilon$ is the vector of model errors. If you apply the ridge regression loss to this model and simplify the expressions you will obtain the following loss function.

$$E(\mathbf{w}) = \dfrac{1}{2} \mathbf{w}^T\left[\mathbf{X}^T\mathbf{X} + \lambda \mathbf{I} \right]\mathbf{w} + \left[-\mathbf{X}^T\mathbf{y} \right]^T\mathbf{w}$$

Our goal is to minimize this expression with the additional constraints on the coefficients. If we assume only negative coefficients we will obtain this inequality constraint

$$\mathbf{I}\mathbf{w} \preceq \mathbf{0}.$$

Hence, we have obtained a quaratic programming formulation of the problem.

$$\text{minimize: } E(\mathbf{w}) = \dfrac{1}{2} \mathbf{w}^T\left[\mathbf{X}^T\mathbf{X} + \lambda \mathbf{I} \right]\mathbf{w} + \left[-\mathbf{X}^T\mathbf{y} \right]^T\mathbf{w}$$ $$\text{subject to: } \mathbf{I}\mathbf{w} \preceq \mathbf{0}$$

You can solve this kind of problem quite straight forward with cvxopt for Python. You can also have more complicated linear constraints (equality and inequality constriants).

Note: CVXOPT uses $\mathbf{x}$ for the unknows, which are $\mathbf{w}$ in my formulation.

$\endgroup$
2
  • $\begingroup$ does cvxopt support mixed signs for the constraints (ie, some parameters positive, some negative), or even better, does it support only constraining some parameters and leaving the others free? If so I think that's exactly what I'm looking for! $\endgroup$
    – awho
    Aug 28 at 23:57
  • $\begingroup$ You can always rewrite inequality constraints into something like $Ax \leq h$. E.g. $x > 0 => -x < 0$. $\endgroup$ Sep 2 at 14:24
2
$\begingroup$

It is possible to constrain to linear regression in scikit-learn to only positive coefficients. The sklearn.linear_model.LinearRegression has an option for positive=True which:

When set to True, forces the coefficients to be positive. This option is only supported for dense arrays.

The positive=True option is not available for ridge regression in scikit-learn.

$\endgroup$
1
  • $\begingroup$ I definitely need some parameters to be negative, as stated in my question, so this won't work. Thanks though. $\endgroup$
    – awho
    Aug 28 at 23:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.