Clarifying some unclear Areas of model training, python, Machine Learning

Question

The Problem

I'm learning ML these days. I'm training a dataset with 10,000+ samples with 20+ features, the model I picked to train was "Logistic Regression", I have some problems in my mind, "Unclear areas", which are making my brain boil. So I'm trying to clear those and keep on learning what I can, but yet couldn't find where to really start solving it. So I decided to get a little community support by at least a two or one reference that will solve where I'm stuck.

Explanation of what I tried and what is this "Problem"

So, before start explaining, I will tell you what my setting so far;

1. Logistic Regression
2. Using as a classification Problem - Binary Classification(1,0)
3. 30% of testing data
4. all 100% of the data are balanced well with equal number of labels-> Binary Classification
5. Number of features selected: 14

Current workaround that I did:

• I trained the above mentioned model with above setting + getting the support of sci kit learn as well, however in the accuracy_score() function with normalization=true set, I got a performance value of 1.(The best performance as to the function standards as doc says).

Back to the problem explanation

I have these problems in my head right now;

1. Am I doing this correct? I mean, for logistic regression with 14 features(X[0..13]), and 2(y[0..1]) class labels.
2. Most Importantly I want to interpret this in a plot, where can I learn to do that, I have 14 features, I mean lets say that I had 6 features, where am I doing wrong?
3. I want to check this on decision boundaries, how can I do that? (Testing, and trained decision boundaries), I cannot do that as well since I'm having more than 2 features.
4. How can I further more verify this test is an under fit or an over fit?

Thank you for reading, hope you will provide your answer with care.

Answer 1

Do you have a single class and you're trying to predict whether or not the input is an instance of it? In this case, you're doing binary classification with logistic regression, though you only need 1 output: your model would predict the probability that the input belongs to the class, and will vary between 0 and 1. If the output of the model is >= 0.5, then the input is predicted to belong to the class, otherwise no. If you have two or more classes, then you need two or more outputs (i.e. your y[]s) and you would want to do softmax regression where your model predicts the probability of each class and then you take the predicted class with the highest probability.

For visualizing, you want to do dimensionality reduction. There are several ways to do this and it's a large subject in itself, but probably the easiest way to start is to project your 14 dimensions down to 2 and plot them. Scikit-learn has a PCA class which makes this straightforward:

pca = PCA(n_components=2)
X_projected = pca.fit_transform(X_in)

To check for overfitting or underfitting, you generally want to separate out some percentage of your data for testing purposes only, not used for training/fitting the model. If I understand correctly, you have already done this with 30% of your data? Then you check the ability of your model to predict the class of the test data it hasn't seen. Overfitting models will perform much better on training data vs. test data.

hth.