0
$\begingroup$

The Problem

I'm learning ML these days. I'm training a dataset with 10,000+ samples with 20+ features, the model I picked to train was "Logistic Regression", I have some problems in my mind, "Unclear areas", which are making my brain boil. So I'm trying to clear those and keep on learning what I can, but yet couldn't find where to really start solving it. So I decided to get a little community support by at least a two or one reference that will solve where I'm stuck.

Explanation of what I tried and what is this "Problem"

So, before start explaining, I will tell you what my setting so far;

  1. Logistic Regression
  2. Using as a classification Problem - Binary Classification(1,0)
  3. 30% of testing data
  4. all 100% of the data are balanced well with equal number of labels-> Binary Classification
  5. Number of features selected: 14

Current workaround that I did:

  • I trained the above mentioned model with above setting + getting the support of sci kit learn as well, however in the accuracy_score() function with normalization=true set, I got a performance value of 1.(The best performance as to the function standards as doc says).

Back to the problem explanation

I have these problems in my head right now;

  1. Am I doing this correct? I mean, for logistic regression with 14 features(X[0..13]), and 2(y[0..1]) class labels.
  2. Most Importantly I want to interpret this in a plot, where can I learn to do that, I have 14 features, I mean lets say that I had 6 features, where am I doing wrong?
  3. I want to check this on decision boundaries, how can I do that? (Testing, and trained decision boundaries), I cannot do that as well since I'm having more than 2 features.
  4. How can I further more verify this test is an under fit or an over fit?

Thank you for reading, hope you will provide your answer with care.

$\endgroup$

1 Answer 1

0
$\begingroup$

Do you have a single class and you're trying to predict whether or not the input is an instance of it? In this case, you're doing binary classification with logistic regression, though you only need 1 output: your model would predict the probability that the input belongs to the class, and will vary between 0 and 1. If the output of the model is >= 0.5, then the input is predicted to belong to the class, otherwise no. If you have two or more classes, then you need two or more outputs (i.e. your y[]s) and you would want to do softmax regression where your model predicts the probability of each class and then you take the predicted class with the highest probability.

For visualizing, you want to do dimensionality reduction. There are several ways to do this and it's a large subject in itself, but probably the easiest way to start is to project your 14 dimensions down to 2 and plot them. Scikit-learn has a PCA class which makes this straightforward:

pca = PCA(n_components=2)
X_projected = pca.fit_transform(X_in)

To check for overfitting or underfitting, you generally want to separate out some percentage of your data for testing purposes only, not used for training/fitting the model. If I understand correctly, you have already done this with 30% of your data? Then you check the ability of your model to predict the class of the test data it hasn't seen. Overfitting models will perform much better on training data vs. test data.

hth.

$\endgroup$
5
  • $\begingroup$ Your explanation is great, Sorry I forgot to mention more clearly about labeling and what the classification type is of my model that I'm working on. Yes it's binary and 1 and 0s labelled, I'm trying predict my prediction whether or not the input is an instance of it. As I've described in my thread, I've separated 30% of the test data 70% of training data, when I execute my trained model on accuracy_score() with test data (unseen data, yes)I get 1, so as it means 100%, so this means that the model is performing in a good level? $\endgroup$
    – OctoCat
    Apr 26, 2023 at 9:10
  • $\begingroup$ Code that I used to score my model-> `y_pred = lr.predict(X_test_std); from sklearn.metrics import accuracy_score; print('Accuracy: %.3f ' %accuracy_score(y_test, y_pred, normalize=True));`` $\endgroup$
    – OctoCat
    Apr 26, 2023 at 9:10
  • $\begingroup$ So your output of your model should only be 1 float instead of 2, and that will be 0 to indicate negative if the class is not present and positive if it is. I would start off by projecting the input data down to 2 dimensions, and plotting that, with different colors for positive and negative instances. Then you can see if the data is indeed perfectly separable. There are other ways of measuring models (confusion matrix, precision, recall, f1, etc), but 100% accuracy certainly always make me feel like I've missed something. ;) $\endgroup$ Apr 26, 2023 at 9:45
  • $\begingroup$ All you said were very useful, thank you a lots for your guidance! <3 $\endgroup$
    – OctoCat
    Apr 29, 2023 at 12:18
  • $\begingroup$ Glad to help, please feel free to accept/upvote if it's useful :-) $\endgroup$ Apr 30, 2023 at 9:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.