26 votes
Accepted

Should I use a decision tree or logistic regression for classification?

Long story short: do what @untitledprogrammer said, try both models and cross-validate to help pick one. Both decision trees (depending on the implementation, e.g. C4.5) and logistic regression ...
  • 476
24 votes

What does it mean to "share parameters between features and classes"

I will try to answer this question through logistic regression, one of the simplest linear classifiers. The simplest case of logistic regression is if we have a binary classification task ($y \in\{0,...
  • 7,758
23 votes
Accepted

Scikit-learn: Getting SGDClassifier to predict as well as a Logistic Regression

The comments about iteration number are spot on. The default SGDClassifier n_iter is 5 ...
  • 961
17 votes

Text categorization: combining different kind of features

If I understand correctly, you essentially have two forms of features for your models. (1) Text data that you have represented as a sparse bag of words and (2) more traditional dense features. If that ...
  • 800
17 votes
Accepted

Python implementation of cost function in logistic regression: why dot multiplication in one expression but element-wise multiplication in another

In this case, the two math formulae show you the correct type of multiplication: $y_i$ and $\text{log}(a_i)$ in the cost function are scalar values. Composing the scalar values into a given sum over ...
  • 28.1k
16 votes
Accepted

Linear regression with non-symmetric cost function?

If I understand you correctly, you want to err on the side of overestimating. If so, you need an appropriate, asymmetric cost function. One simple candidate is to tweak the squared loss: $\mathcal L: ...
  • 10.5k
15 votes
Accepted

Why continuous features are more important than categorical features in decision tree models?

It could be the way that you encode categorical variables. If you do One Hot Encoding (dummy) each encoded feature will only have two possible values [0,1]. Binary variables normally have less ...
14 votes

How to get p-value and confident interval in LogisticRegression with sklearn?

The short answer is that sklearn LogisticRegression does not have a built in method to calculate p-values. Here are a few other posts that discuss solutions to this, however. https://stackoverflow....
  • 1,429
14 votes

How to get p-value and confident interval in LogisticRegression with sklearn?

One way to get confidence intervals is to bootstrap your data, say, $B$ times and fit logistic regression models $m_i$ to the dataset $B_i$ for $i = 1, 2, ..., B$. This gives you a distribution for ...
  • 603
14 votes

What is the difference between SGD classifier and the Logisitc regression?

Welcome to SE:Data Science. SGD is a optimization method, while Logistic Regression (LR) is a machine learning algorithm/model. You can think of that a machine learning model defines a loss function, ...
  • 2,149
12 votes

Does scikit-learn use regularization by default?

Please take a look at the documentation. The first line shows the default parameters, which include penalty='l2' and C=1.0. You ...
  • 1,627
11 votes
Accepted

How do I implement the sigmoid function in Octave?

This will compute the sigmoid of a scalar, vector or matrix. ...
11 votes
Accepted

The differences between SVM and Logistic Regression

If you use logistic regression and the cross-entropy cost function, it's shape is convex and there will be a single minimum. But during optimization, you may find ...
  • 13.7k
11 votes

How to plot logistic regression decision boundary?

Your decision boundary is a surface in 3D as your points are in 2D. With Wolfram Language Create the data sets. ...
  • 695
11 votes

My data is highly overlapping, but when I apply logistic regression, it is giving an impressive accuracy of 79%. Why?

Decision Tree, KNN, & Random Forest (Methods that are suitable for overlapping data) This statement is false. All those methods are good when the decision surface (separating surface) has a ...
  • 4,603
10 votes
Accepted

Trying to understand Logistic Regression Implementation

There are several issues I see with the implementation. Some are just unnecessarily complicated ways of doing it, but some are genuine errors. Primary takeaways A: Try to start from the math behind ...
  • 526
10 votes
Accepted

How to plot logistic regression decision boundary?

Regarding the code You should plot the decision boundary after training is finished, not inside the training loop, parameters are constantly changing there; unless you are tracking the change of ...
  • 8,997
9 votes
Accepted

Logistic regression on biased data

Background I'll start with some background to help you research the solution yourself and then will add some specifics. What you refer to as "biased data" is more commonly known as ...
  • 6,778
8 votes
Accepted

Best or recommended R package for logit and probit regression

Unless you have some very specific or exotic requirements, in order to perform logistic (logit and probit) regression analysis in R, you can use standard (built-in ...
8 votes
Accepted

What's the relationship between an SVM and hinge loss?

They are both discriminative models, yes. The logistic regression loss function is conceptually a function of all points. Correctly classified points add very little to the loss function, adding more ...
  • 6,525
8 votes

Are linear models better when dealing with too many features? If so, why?

There is some important information missing in your question, i.e. what the standard parameters are and what kind of logistic regression you use. When you use ...
  • 7,207
7 votes

Should I use a decision tree or logistic regression for classification?

Try using both regression and decision trees. Compare the efficiency of each technique by using a 10 fold cross validation. Stick to the one with higher efficiency. It would be difficult to judge ...
7 votes

Does scikit-learn use regularization by default?

Yes, there is regularization by default. It appears to be L2 regularization with a constant of 1. I played around with this and found out that L2 regularization with a constant of 1 gives me a fit ...
7 votes
Accepted

Preprocessing text before use RNN

Welcome to the Data Science forum. Yes, data preprocessing is an important aspect of sentiment analysis for better results. What sort of preprocessing to be done largely depends on the quality of ...
7 votes
Accepted

Do logistic regression and softmax regression do the same thing?

There is a key difference: Softmax regression provides class probabilities for mutually exclusive classes. Logistic regression treats class membership for each class separately. Classes do not need ...
  • 28.1k
7 votes
Accepted

Bad classification performance of logistic regression on imbalanced data in testing as compared to training

I suspect the reason is that the class balance in your test set is different from the class balance in your training set. That will throw everything off. The fundamental assumption made by ...
  • 3,242
7 votes

Is this a good practice of feature engineering?

1) Yes, it makes sense. Trying to create features manually will help the learners (i.e. models) to graspe more information from the raw data because the raw data is not always in a form that is ...
  • 71
7 votes
Accepted

How does binary cross entropy work?

When doing logistic regression you start calculating a bunch of probabilities $p_i$ and your target is maximize the product of those probabilities (as they're considered independent events). The ...
  • 241
7 votes
Accepted

What is the difference between SVM and logistic regression?

Both logistic regression and SVM are linear models under the hood, and both implement a linear classification rule: $$f_{\mathbf{w},b}(\mathbf{x}) = \mathrm{sign}(\mathbf{w}^T \mathbf{x} + b)$$ Note ...
  • 2,101
7 votes
Accepted

Risk prediction vs classification model

I will try to answer your question as shortly as possible. Yes, if you define probability as a risk, then the probabilities are risk scores. But, there's a catch in these scenarios, you will have to ...
  • 1,138

Only top scored, non community-wiki answers of a minimum length are eligible