0
$\begingroup$

Is there a way to intuitively tell if the lasso penalty for a particular feature will be small or large? Consider the following scenario:

Imagine we use Lasso regression on a dataset of 100 features $ [X_1, X_2, X_3, ..., X_{100}] $. Now, if we take one feature and scale up its values by a factor of 10, so for example $X_1 = X_1 * 10$, and then we fit lasso regression on the new dataset, which of the following might be true?

  1. X1 probably would be excluded from the fitted model
  2. X1 probably would be included in the fitted model

Someone told me that 2 is true. Somehow the operation on $X_1$ would result in less lasso penalty for $X_1$, therefore the feature is likely to be kept.

I cannot understand why. Can anyone tell if this whole statement is not correct to begin with or if there is some truth in it?

$\endgroup$
0
$\begingroup$

When you increase the scale of $X_1$ by 10, the scale of the corresponding weight in you linear regression sees its scale divided by 10 (it is exactly 10 for standard linear regression and the same order of magnitude holds for lasso loss). As a consequence, the $L_1$ penalty for this coefficient is also divided by a factor 10 and it is less likely to be set to 0.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.