2
$\begingroup$

I am using raw data set with 4 feature variables to do a Binominal Classification using Logistic Regression Algorithm.

I made sure that the class counts are balanced. i.e., an equal number of occurrences per class.

The four features are Total Cholesterol, Systolic Blood Pressure, Diastolic Blood Pressure, and Cigraeette count. The class variable name is Stroke.

Dataset description is shown below:

       TOTCHOL    SYSBP    DIABP  CIGPDAY   STROKE
count  200.000  200.000  200.000  200.000  200.000
mean   231.040  144.560   81.400    4.480    1.500
std     42.465   23.754   11.931    9.359    0.501
min    112.000  100.000   51.500    0.000    1.000
25%    204.750  126.750   73.750    0.000    1.000
50%    225.500  141.000   80.000    0.000    1.500
75%    256.250  161.000   90.000    4.000    2.000
max    378.000  225.000  113.000   60.000    2.000

SKEW is

TOTCHOL    0.369
SYSBP      0.610
DIABP      0.273
CIGPDAY    2.618
STROKE     0.000

Python + sklearn is used here. The problem is that the classification performance gets very negatively-impacted when I try to normalize the dataset using

 X=preprocessing.StandardScaler().fit(X).transform(X)

or

 X=preprocessing.MinMaxScaler().fit_transform(X)

The classification report (before) normalizing the dataset:

            precision    recall  f1-score   support

      1       0.85      0.79      0.81        28
      2       0.82      0.88      0.85        32
avg / total   0.83      0.83      0.83        60

While the classification report (After) normalizing the dataset:

         precision    recall  f1-score   support

      1       0.47      1.00      0.64        28
      2       1.00      0.03      0.06        32

  avg/total   0.75      0.48      0.33        60

Please note that the class variable includes the outcomes 1(yes) and 2(no) instead of the usual 0 and 1.

at this point, I see the preprocessing have damaged the classification accuracy instead of helping it. Is there any logical explanation for that? do I "have" to do the normalization step?

Another remark is the probability scores that I get (before) normalizing the dataset are much higher per class as shown below:

 [ 0.65929838  0.34070162]
 [ 0.40999878  0.59000122]
 [ 0.43592976  0.56407024]
 [ 0.40306785  0.59693215]
 [ 0.92748002  0.07251998]
 [ 0.74173761  0.25826239]

compared to the ones (after) normalizing the dataset

  [ 0.51636816  0.48363184]
  [ 0.5183946   0.4816054 ]
  [ 0.51410135  0.48589865]
  [ 0.50739794  0.49260206]
  [ 0.52645649  0.47354351]
  [ 0.5308564   0.4691436 ]

is there any explanation why I get such results?

$\endgroup$
1
  • $\begingroup$ Have you checked your variables for multicollinearity? If they are slightly multicollinear in their raw scales you might be making it worse by normalizing them, thereby decreasing model performance. $\endgroup$
    – bstrain
    Sep 11, 2019 at 6:34

1 Answer 1

2
$\begingroup$

I have found the answer to my own question, went back to the Python script and in the command that fits the model i.e.

   LR = LogisticRegression (C=0.1, solver = "sag",max_iter=1000).fit (X_train, y_train)

The parameter C was set to 0.001 which is a very small value (meaning lambda is very high as C=1/lambda) (C is the regularization strength and smaller values indicate stronger regularization). more on that matter can be found here and here

$\endgroup$
3
  • $\begingroup$ That was my first question while reading the post. The scaled dataset "needs" larger coefficients to produce the same overall model, but then these larger coefficients get penalized more; so you end up with a model with much-too-small coefficients, leading to your indecisive probability scores and performance loss. $\endgroup$
    – Ben Reiniger
    Feb 13, 2020 at 13:23
  • $\begingroup$ congrats on finding the answer! Please "accept" your own answer (with the little green "v" sign) so others know it's closed. $\endgroup$ Jan 5, 2021 at 7:30
  • $\begingroup$ This is why I like using LogisticRegressionCV (scikit-learn.org/stable/modules/generated/…) to automatically find C by cross-validation. It is important for the data to be normalized in this case (the regularization makes sense when the coefficients are on the same scale) $\endgroup$ Jan 5, 2021 at 7:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.