# On simple 1D dataset, LogisticRegressionCV selects terrible hyperparameters, resulting scores are nonsensical

I am trying to use LogisticRegressionCV to fit a logistic regression model to a simple 1D dataset. Very oddly, when given a choice, it seems to select a tiny C value, which forces my model to select a tiny theta resulting in a useless model.

I tried looking at the scores_ provided by the model, but they make no sense. For example, when I tell it to do 3-fold cross validation with 5 choices of C values, it gives me:

{1: array([[0.47058824, 0.47058824, 0.47058824, 0.47058824, 0.47058824],
[1.        , 1.        , 1.        , 1.        , 1.        ],
[0.63636364, 0.63636364, 0.63636364, 0.63636364, 0.63636364]])}


The dataset is not linearly separable, and yet it claims to be getting 100% accuracy regardless of which C values I've given it to try.

Example code below:

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
from sklearn.linear_model import LogisticRegressionCV
from sklearn.linear_model import LogisticRegression

def gen_y(x):
p1 = np.clip(x + 0.5, 0, 1)
v = np.random.uniform(0, 1)
if v < p1:
return 1
return 0

np.random.seed(6)
x_data = np.sort(np.random.normal(0, 0.3, 100))
y_data = np.array([gen_y(x) for x in x_data])

regularized_logistic_regression_model = LogisticRegressionCV(Cs = np.array([10**-8, 10**-4, 1, 10**4, 10**8]), fit_intercept = False, cv = 3)

regularized_logistic_regression_model.fit(x_data.reshape(-1, 1), y_data)

print(regularized_logistic_regression_model.C_) # yields 10^-8
print(regularized_logistic_regression_model.coef_) # yields incredibly tiny value
print(regularized_logistic_regression_model.scores_) # yields nonsensical scores
$$$$


First I did a visualization of what your code is doing (see code at the bottom) The model seems completely fine. The coefficient of the linear regression is close to 0, where it should be given how you have created the data.

You are misunderstanding

regularized_logistic_regression_model.scores_

{1: array([[0.47058824, 0.47058824, 0.47058824, 0.47058824, 0.47058824], [1. , 1. , 1. , 1. , 1. ], [0.63636364, 0.63636364, 0.63636364, 0.63636364, 0.63636364]])}

I quote from sklearn documentation:

scores_dict: dict with classes as the keys, and the values as the grid of scores obtained during cross-validating each fold

This is the prediction of one of the folds (note that if you increase the cv parameter it changes length)

The data created is sorted! If you plot the values of X vs the Index you can see You are getting those results because the data is sorted! Not just because of luck. If you shuffle the data you won´t have perfect predictions.

Your result is around 0.7 that is actually what makes sense by looking at the image I attached.

I shuffled the data, not in the most elegant way but you get now different results.

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
from sklearn.linear_model import LogisticRegressionCV
from sklearn.linear_model import LogisticRegression
plt.style.use('seaborn-whitegrid')

def gen_y(x):
p1 = np.clip(x + 0.5, 0, 1)
v = np.random.uniform(0, 1)
if v < p1:
return 1
return 0

np.random.seed(6)

x_data = np.sort(np.random.normal(0, 0.3, 100))
y_data = np.array([gen_y(x) for x in x_data])

df = pd.DataFrame(x_data.reshape(-1, 1),columns=['X'])
df['y']= y_data

plt.figure()
plt.title('Values of X vs Index')
df.X.plot()
plt.savefig('x')
plt.show()

df = df.sample(frac=1)

regularized_logistic_regression_model = LogisticRegressionCV( fit_intercept = False, cv = 3)

regularized_logistic_regression_model.fit(df['X'].values.reshape(-1, 1) , df['y'])

plt.figure()

df['results'] =regularized_logistic_regression_model.predict(df['X'].values.reshape(-1, 1))
plt.scatter(df[df['results']==0].y,df[df['results']==0].X,label='pred=0')
plt.scatter(df[df['results']==1].y,df[df['results']==1].X,label='pred=1')
plt.legend()
plt.hlines(y=regularized_logistic_regression_model.coef_.squeeze(),xmin=-0.1,xmax=1.1)
plt.savefig('ex')
plt.show()

regularized_logistic_regression_model.scores_

• +1, I think this helps clarify things. But I think there are two oddities not yet addressed: (1) How/why do the different folds score so dramatically differently (how did the splitting allow for a perfect score on one fold, and worse-than-random on another)? (2) Why are the scores independent of the hyperparameter? – Ben Reiniger Apr 13 at 12:37
• I think the answer to (1) is just bad luck, and made plausible by the small sample size. The answer to (2) appears to be a combination of using 1D data, using fit_intercept=False, using the default scoring of accuracy, and that the default cutoff (for probability to class prediction) in sklearn is 0.5: all models will score the same on a fixed set of data, unless they swap the sign on the single feature. – Ben Reiniger Apr 13 at 12:40
• Thanks Ben! I appreciate your comments a lot :) – Carlos Mougan Apr 13 at 13:42
• I think I found the reason for (1), the data was created in an order :) – Carlos Mougan Apr 13 at 14:02

Two basic mistakes:

• No intercept
• Sorted data without shuffling

The intercept gives significant boost in the expressive capacity of a logistic regression, especially in problems with only one feature, like here. There is a reason why its default setting is True - it is one of these defaults with which you'd better not mix, unless you know exactly what you are doing. Omitting the intercept in the simple univariate case like here is easy to picture: it forces the regression line to pass through the origin (0, 0) - a huge constraint.

Shuffling is especially important in such cases of artificial datasets, where at some point the values are sorted (as you do here). The reason is, while ML models can be very good at interpolating, they are extremely bad at extrapolating (predicting values outside their training range); and with sorted data, each of your validation CV folds tries to predict with data that are outside the respective training fold (and, unsurprisingly, does not do a good job).

So, just shuffling the data, and fitting with these data and fit_intercept = True, we get:

from sklearn.utils import shuffle

x_s, y_s = shuffle(x_data, y_data, random_state=0)

regularized_logistic_regression_model = LogisticRegressionCV(
Cs = np.array([10**-8, 10**-4, 1, 10**4, 10**8]), fit_intercept = True, cv = 3)

regularized_logistic_regression_model.fit(x_s.reshape(-1, 1), y_s)

print(regularized_logistic_regression_model.C_)
print(regularized_logistic_regression_model.coef_)
print(regularized_logistic_regression_model.scores_)


Results:

[10000.]
[[4.57770177]]
{1: array([[0.61764706, 0.61764706, 0.70588235, 0.67647059, 0.67647059],
[0.60606061, 0.60606061, 0.78787879, 0.84848485, 0.84848485],
[0.60606061, 0.60606061, 0.57575758, 0.72727273, 0.72727273]])}


Already much more sensible than the ones you report.

Adding a little more data (300 samples instead of just 100), gives

[1.]
[[3.57243675]]
{1: array([[0.52, 0.52, 0.72, 0.72, 0.72],
[0.52, 0.52, 0.68, 0.67, 0.67],
[0.51, 0.51, 0.67, 0.66, 0.66]])}


Final note: although shuffling is a highly recommended practice in general, here (artificial random data, by definition) you could avoid the need for it if you leave the initial data as-is (i.e. without sorting them):

x_data = np.random.normal(0, 0.3, 100) # no sorting
`

I'm leaving the verification of this as an exercise.