31

Logistic regression is regression, first and foremost. It becomes a classifier by adding a decision rule. I will give an example that goes backwards. That is, instead of taking data and fitting a model, I'm going to start with the model in order to show how this is truly a regression problem. In logistic regression, we are modeling the log odds, or logit, ...


25

Long story short: do what @untitledprogrammer said, try both models and cross-validate to help pick one. Both decision trees (depending on the implementation, e.g. C4.5) and logistic regression should be able to handle continuous and categorical data just fine. For logistic regression, you'll want to dummy code your categorical variables. As @...


23

The comments about iteration number are spot on. The default SGDClassifier n_iter is 5 meaning you do 5 * num_rows steps in weight space. The sklearn rule of thumb is ~ 1 million steps for typical data. For your example, just set it to 1000 and it might reach tolerance first. Your accuracy is lower with SGDClassifier because it's hitting iteration limit ...


23

I will try to answer this question through logistic regression, one of the simplest linear classifiers. The simplest case of logistic regression is if we have a binary classification task ($y \in\{0,1\})$and only one input feature ($x \in R$). In this case the output of logistic regression would be: $$ \hat y = σ(w \cdot x + b) $$ where $w$ and $b$ are ...


17

If I understand correctly, you essentially have two forms of features for your models. (1) Text data that you have represented as a sparse bag of words and (2) more traditional dense features. If that is the case then there are 3 common approaches: Perform dimensionality reduction (such as LSA via TruncatedSVD) on your sparse data to make it dense and ...


16

If I understand you correctly, you want to err on the side of overestimating. If so, you need an appropriate, asymmetric cost function. One simple candidate is to tweak the squared loss: $\mathcal L: (x,\alpha) \to x^2 \left( \mathrm{sgn} x + \alpha \right)^2$ where $-1 < \alpha < 1$ is a parameter you can use to trade off the penalty of ...


15

It's hard to say without knowing a little more about your dataset, and how separable your dataset is based on your feature vector, but I would probably suggest using extreme random forest over standard random forests because of your relatively small sample set. Extreme random forests are pretty similar to standard random forests with the one exception that ...


15

It could be the way that you encode categorical variables. If you do One Hot Encoding (dummy) each encoded feature will only have two possible values [0,1]. Binary variables normally have less importance in Decision trees given how it is computed the feature weight. Let's say per example, that you are trying to predict the condition of a patient at the ...


14

Welcome to SE:Data Science. SGD is a optimization method, while Logistic Regression (LR) is a machine learning algorithm/model. You can think of that a machine learning model defines a loss function, and the optimization method minimizes/maximizes it. Some machine learning libraries could make users confused about the two concepts. For instance, in scikit-...


13

In this case, the two math formulae show you the correct type of multiplication: $y_i$ and $\text{log}(a_i)$ in the cost function are scalar values. Composing the scalar values into a given sum over each example does not change this, and you never combine one example's values with another in this sum. So each element of $y$ only interacts with its matching ...


12

For low parameters, pretty limited sample size, and a binary classifier logistic regression should be plenty powerful enough. You can use a more advanced algorithm but it's probably overkill.


11

Since you are doing binary classification, have you tried adjusting the classification threshold? Since your algorithm seems rather insensitive, I would try lowering it and check if there is an improvement. You can always use Learning Curves, or a plot of one model parameter vs. Training and Validation error to determine whether your model is overfitting. It ...


11

One way to get confidence intervals is to bootstrap your data, say, $B$ times and fit logistic regression models $m_i$ to the dataset $B_i$ for $i = 1, 2, ..., B$. This gives you a distribution for the parameters you are estimating, from which you can find the confidence intervals.


11

If you use logistic regression and the cross-entropy cost function, it's shape is convex and there will be a single minimum. But during optimization, you may find weights that are near to optimal point and not exactly on the optimal point. This means that you can have multiple classifies that reduce the error and maybe set it to zero for the training data ...


11

Your decision boundary is a surface in 3D as your points are in 2D. With Wolfram Language Create the data sets. mqtrue = 5; cqtrue = 30; With[{x = Subdivide[0, 3, 50]}, dat1 = Transpose@{x, mqtrue x + 5 RandomReal[1, Length@x]}; ]; With[{x = Subdivide[7, 10, 50]}, dat2 = Transpose@{x, mqtrue x + cqtrue + 5 RandomReal[1, Length@x]}; ]; View in 2D (...


11

Decision Tree, KNN, & Random Forest (Methods that are suitable for overlapping data) This statement is false. All those methods are good when the decision surface (separating surface) has a highly nonlinear form. They act as a non-parametric local approximation - all parameters are not in fact parameters of the decision function but are meta parameters ...


10

Linear models simply add their features multiplied by corresponding weights. If, for example, you have 1000 sparse features only 3 or 4 of which are active in each instance (and the others are zeros) and 20 dense features that are all non-zeros, then it's pretty likely that dense features will make most of the impact while sparse features will add only a ...


10

Short Answer Yes, logistic regression is a regression algorithm and it does predict a continuous outcome: the probability of an event. That we use it as a binary classifier is due to the interpretation of the outcome. Detail Logistic regression is a type of generalize linear regression model. In an ordinary linear regression model, a continuous outcome, ...


10

There are several issues I see with the implementation. Some are just unnecessarily complicated ways of doing it, but some are genuine errors. Primary takeaways A: Try to start from the math behind the model. The logistic regression is a relatively simple one. Find the two equations you need and stick to them, replicate them letter by letter. B: ...


9

Please take a look at the documentation. The first line shows the default parameters, which include penalty='l2' and C=1.0. You actually cannot disable regularization completely, you can only regularize less... try setting C=1e10 for example.


9

Background I'll start with some background to help you research the solution yourself and then will add some specifics. What you refer to as "biased data" is more commonly known as unbalanced classes in the data science world. Also "customer turnover" is often referred to as churn. Metrics As hoards of Ng'ian devotees will undoubtably point out you need ...


9

The short answer is that sklearn LogisticRegression does not have a built in method to calculate p-values. Here are a few other posts that discuss solutions to this, however. https://stackoverflow.com/questions/27928275/find-p-value-significance-in-scikit-learn-linearregression https://stackoverflow.com/questions/22306341/python-sklearn-how-to-calculate-p-...


8

Unless you have some very specific or exotic requirements, in order to perform logistic (logit and probit) regression analysis in R, you can use standard (built-in and loaded by default) stats package. In particular, you can use glm() function, as shown in the following nice tutorials from UCLA: logit in R tutorial and probit in R tutorial. If you are ...


8

This will compute the sigmoid of a scalar, vector or matrix. function g = sigmoid(z) % SIGMOID Compute sigmoid function % g = SIGMOID(z) computes the sigmoid of z. % Compute the sigmoid of each value of z (z can be a matrix, % vector or scalar). SIGMOID = @(z) 1./(1 + exp(-z)); g = SIGMOID(z); end


7

They are both discriminative models, yes. The logistic regression loss function is conceptually a function of all points. Correctly classified points add very little to the loss function, adding more if they are close to the boundary. The points near the boundary are therefore more important to the loss and therefore deciding how good the boundary is. SVM ...


7

Yes, there is regularization by default. It appears to be L2 regularization with a constant of 1. I played around with this and found out that L2 regularization with a constant of 1 gives me a fit that looks exactly like what sci-kit learn gives me without specifying regularization. from sklearn.linear_model import LogisticRegression model = ...


7

Welcome to the Data Science forum. Yes, data preprocessing is an important aspect of sentiment analysis for better results. What sort of preprocessing to be done largely depends on the quality of your data. You'll have to explore your corpus to understand the types of variables, their functions, permissible values, and so on. Some formats including html and ...


7

I suspect the reason is that the class balance in your test set is different from the class balance in your training set. That will throw everything off. The fundamental assumption made by statistical machine learning methods (including logistic regression) is that the distribution of data in the test set matches the distribution of data in the training ...


7

1) Yes, it makes sense. Trying to create features manually will help the learners (i.e. models) to graspe more information from the raw data because the raw data is not always in a form that is amenable to learning, but you can always construct features from it that are. The feature you are adding are based on one feature. This is common. However, your ...


7

Both logistic regression and SVM are linear models under the hood, and both implement a linear classification rule: $$f_{\mathbf{w},b}(\mathbf{x}) = \mathrm{sign}(\mathbf{w}^T \mathbf{x} + b)$$ Note that I am regarding the "primal", linear form of the SVM here. In both cases the parameters $\mathbf{w}$ and $b$ are estimated by minimizing a certain ...


Only top voted, non community-wiki answers of a minimum length are eligible