For instance, why is it that it is more favourable for a weight of [0.25, 0.25, 0.25, 0.25] (for which the L2 penalty is 0.25) instead of simply [1, 0, 0, 0] (for which an L2 penalty is 1)?

In this case, both weights would give the same dot product when using W.T * X


You answered this in your question. "Prefer" means "produces a smaller penalty", and you've identified that the penalty in the first case is smaller. Why would this be a good thing? It amounts to preferring an explanation based a bit on many features, rather than one based entirely on one features. That's often a good bet to avoid overfitting.

These two weight vectors do not produce the same dot product with other vectors in general. If the vector X contained all identical values, they would.

If the 4 input features were regularly identical or nearly so, then it means they're redundant, and you may prefer to use just 1 of the features (your second case), instead of a bit of each. In this case, an L1 penalty would at least be indifferent to the two, not penalize the second one.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.