0
$\begingroup$

Cross-posting from Stack Overflow:

I'm running into a weird situation where my sklearn LogisticRegressionCV model is apparently getting 100% accuracy (the lack of shuffling is intentional). However, when I ask the model to report its accuracy scores, the accuracy scores per inverse regularization are much lower than 100%. What am I doing wrong?

The model:

    
    decoder = LogisticRegressionCV(
        penalty='l1',                       # want sparse parameters
        cv=KFold(n_splits=5, shuffle=False),
        Cs=np.logspace(-1, 2, 3),
        random_state=0,                     # reproducibility
        dual=False,                         # Prefer dual=False when n_samples > n_features.
        solver='liblinear',                 # for l1 penalty, must use liblinear or saga
        max_iter=1e5,                       # give plenty of time to get good performance
        fit_intercept=False,
        class_weight='balanced',
    ).fit(X, y)
    
    y_hat = decoder.predict(X)
    
    y_hat_probs = decoder.predict_proba(X)
    
    print(classification_report(y, y_hat))
    print(confusion_matrix(y, y_hat))

outputs

              precision    recall  f1-score   support

         0.0       1.00      1.00      1.00       374
         1.0       1.00      1.00      1.00       105

    accuracy                           1.00       479
   macro avg       1.00      1.00      1.00       479
weighted avg       1.00      1.00      1.00       479

[[374   0]
 [  0 105]]

Asking for decoder.scores_ yields values nowhere near 100%:

{1.0: array([[0.35416667, 0.60416667, 0.61458333],
        [0.64583333, 0.64583333, 0.65625   ],
        [0.59375   , 0.65625   , 0.64583333],
        [0.54166667, 0.57291667, 0.59375   ],
        [0.17894737, 0.61052632, 0.65263158]])}

Why is there this discrepancy?

$\endgroup$
1
  • $\begingroup$ When making the kfold, the model have access not to all the data frame whereas when making the predictions, since you are training and evaluating in the same set, the model is just "remembering" the data $\endgroup$ – Julio Jesus Dec 10 '20 at 23:57
0
$\begingroup$

You are training and evaluate on the same data.

You should split data to train set(to train) and test set(to evaluate) to get proper results. Kinda:


decoder.fit(X_train, y_train)
y_hat = decoder.predict(X_test)

print(classification_report(y_test, y_hat))
print(confusion_matrix(y_test, y_hat))

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.