How to improve baseline logistic regression in a high dimensional binary classification problem?

Question

Info about dataset:

• df.shape = (10000, 100)
• All feature are numerical values.
• There are few outliers in each column. The column with the most outliers has 0.7% of data as outliers.

I am trying to improve on my baseline logistic regression; however, I'm stuck.

baseline = LogisticRegression(solver='lbfgs', max_iter=100, penalty='l2')

Here are some approaches I've taken and relative results:

1. Standard scaler - Logistic regression (similar)
2. Robust scaler - Logistic regression (simliar)
3. Remove outliers (IQR method) - standard scaler - Logistic regression (worse)
4. Standard scaler - PCA (n_component=n_comp that explain 83% variance) - Logistic regression (more worse)

All approaches seem to perform worse than the baseline.

How can I improve my baseline logistic regression model or do I need to resort to nonlinear models like random forest (I've already tried it however it overfits)?

Thanks in advance.

Answer 1

It appears that you are using manual trial-and-error to search for better hyperparameters.

Another approach would to be use automated hyperparameter search. Define a search space (i.e., either a range or distribution) for each hyperparameter. Then use cross-validation to find the best combination in the search space. Random search on hyperparameters is often useful.