Info about dataset:

  • df.shape = (10000, 100)
  • All feature are numerical values.
  • There are few outliers in each column. The column with the most outliers has 0.7% of data as outliers.

I am trying to improve on my baseline logistic regression; however, I'm stuck.

baseline = LogisticRegression(solver='lbfgs', max_iter=100, penalty='l2')

Here are some approaches I've taken and relative results:

  1. Standard scaler - Logistic regression (similar)
  2. Robust scaler - Logistic regression (simliar)
  3. Remove outliers (IQR method) - standard scaler - Logistic regression (worse)
  4. Standard scaler - PCA (n_component=n_comp that explain 83% variance) - Logistic regression (more worse)

All approaches seem to perform worse than the baseline.

How can I improve my baseline logistic regression model or do I need to resort to nonlinear models like random forest (I've already tried it however it overfits)?

Thanks in advance.

  • $\begingroup$ Have you tried regularization ? Perhaps try lasso regression since that has the effect of eliminating features that are essentially noise to your problem objective. $\endgroup$ May 8, 2021 at 16:09
  • $\begingroup$ Are you able to share a sample of your data? So that I can work out with it? at least 500 observations randomly sampled would be enough $\endgroup$
    – Multivac
    May 8, 2021 at 16:50
  • $\begingroup$ @JayaramIyer So simply changing logistic regression paramater "penalty=l1" would do? $\endgroup$
    – haneulkim
    May 9, 2021 at 2:01
  • $\begingroup$ Use penalty = l1, solver = saga and C = 0.1 (reducing the value increases the amount of regularization applied) $\endgroup$ May 9, 2021 at 3:18
  • $\begingroup$ @JayaramIyer Thanks, indeed using lasso, c=0.1 but with solver="liblinear" improved performance and converged. $\endgroup$
    – haneulkim
    May 9, 2021 at 3:41

1 Answer 1


It appears that you are using manual trial-and-error to search for better hyperparameters.

Another approach would to be use automated hyperparameter search. Define a search space (i.e., either a range or distribution) for each hyperparameter. Then use cross-validation to find the best combination in the search space. Random search on hyperparameters is often useful.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.