1
$\begingroup$

I have multiple datasets that I trained with ElasticNetCV (sklearn), and I noticed that many of them selected l1_ratio = 1 as the best value (which is the max value tried by the CV),

So as a test I wondered if values greater than 1 will produce a better result - and surprisingly the answer is yes... in fact you can reproduce this phenomenon with this code:

    from sklearn.linear_model import ElasticNet
    from sklearn.model_selection import train_test_split

    n = 200
    features = np.random.rand(n, 5)
    target = np.random.rand(n)+features.sum(axis=1)*5

    train_feat, test_feat, train_target, test_target = train_test_split(features, target)

    cls = ElasticNet(random_state=42, l1_ratio=1, alpha=0.1)
    cls.fit(train_feat, train_target)
    print(cls.score(test_feat, test_target), cls.score(train_feat, train_target))

    cls = ElasticNet(random_state=42, l1_ratio=1.1, alpha=0.1)
    cls.fit(train_feat, train_target)
    print(cls.score(test_feat, test_target), cls.score(train_feat, train_target))

And you will find that the l1_ratio=1.1 regressor is better on both train and test.

According to the documentation, you shouldn't use l1_ratio>1, but it does technically work. However it doesn't make much sense, as it would mean that the L2 part of the loss function becomes negative - so higher L2 values of the coefs don't punish, but in fact reward (!) the loss function.

Is there any theoretical logic behind this? Is there any reason not to expand the L1 search range to $[0,2]$ instead of $[0,1]$?

$\endgroup$
4
  • $\begingroup$ Interesting. l1_ratio > 1 should not be possible in sklearn v0.24, it is caught with a ValueError. What's your sklearn version? $\endgroup$ – Tinu Jan 20 at 11:29
  • $\begingroup$ 0.23.1, lucky that I didn't update so I could find that out... $\endgroup$ – Oren Matar Jan 20 at 11:41
  • $\begingroup$ My understanding is that $\alpha$ is a scaling parameter for the L1-Ratio and is thus defined $ \in [0,1] $. See this discussion: stats.stackexchange.com/questions/84012/… $\endgroup$ – Peter Jan 20 at 14:23
  • $\begingroup$ I think there is no logical reason to extend the range above 1. Just because it is mathematically and technically possible to find a solution doesn't make it a useful one. As you already noticed, a ratio larger 1 would make the L2 part negative, and therefore encourage large weights. This is quite the opposite of regularization and possibly the reason why this case is caught as ValueError in the latest version of sklearn. $\endgroup$ – Tinu Jan 20 at 14:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.