23
Logistic regression is regression, first and foremost. It becomes a classifier by adding a decision rule. I will give an example that goes backwards. That is, instead of taking data and fitting a model, I'm going to start with the model in order to show how this is truly a regression problem.
In logistic regression, we are modeling the log odds, or logit, ...
23
The comments about iteration number are spot on. The default SGDClassifier n_iter is 5 meaning you do 5 * num_rows steps in weight space. The sklearn rule of thumb is ~ 1 million steps for typical data. For your example, just set it to 1000 and it might reach tolerance first. Your accuracy is lower with SGDClassifier because it's hitting iteration limit ...
22
Long story short: do what @untitledprogrammer said, try both models and cross-validate to help pick one.
Both decision trees (depending on the implementation, e.g. C4.5) and logistic regression should be able to handle continuous and categorical data just fine. For logistic regression, you'll want to dummy code your categorical variables.
As @...
20
I will try to answer this question through logistic regression, one of the simplest linear classifiers.
The simplest case of logistic regression is if we have a binary classification task ($y \in\{0,1\})$and only one input feature ($x \in R$). In this case the output of logistic regression would be:
$$
\hat y = σ(w \cdot x + b)
$$
where $w$ and $b$ are ...
15
It's hard to say without knowing a little more about your dataset, and how separable your dataset is based on your feature vector, but I would probably suggest using extreme random forest over standard random forests because of your relatively small sample set.
Extreme random forests are pretty similar to standard random forests with the one exception that ...
14
If I understand correctly, you essentially have two forms of features for your models. (1) Text data that you have represented as a sparse bag of words and (2) more traditional dense features. If that is the case then there are 3 common approaches:
Perform dimensionality reduction (such as LSA via TruncatedSVD) on your sparse data to make it dense and ...
12
For low parameters, pretty limited sample size, and a binary classifier logistic regression should be plenty powerful enough. You can use a more advanced algorithm but it's probably overkill.
12
In this case, the two math formulae show you the correct type of multiplication:
$y_i$ and $\text{log}(a_i)$ in the cost function are scalar values. Composing the scalar values into a given sum over each example does not change this, and you never combine one example's values with another in this sum. So each element of $y$ only interacts with its matching ...
11
Since you are doing binary classification, have you tried adjusting the classification threshold? Since your algorithm seems rather insensitive, I would try lowering it and check if there is an improvement.
You can always use Learning Curves, or a plot of one model parameter vs. Training and Validation error to determine whether your model is overfitting. It ...
11
If I understand you correctly, you want to err on the side of overestimating. If so, you need an appropriate, asymmetric cost function. One simple candidate is to tweak the squared loss:
$\mathcal L: (x,\alpha) \to x^2 \left( \mathrm{sgn} x + \alpha \right)^2$
where $-1 < \alpha < 1$ is a parameter you can use to trade off the penalty of ...
10
Short Answer
Yes, logistic regression is a regression algorithm and it does predict a continuous outcome: the probability of an event. That we use it as a binary classifier is due to the interpretation of the outcome.
Detail
Logistic regression is a type of generalize linear regression model.
In an ordinary linear regression model, a continuous outcome, ...
10
Linear models simply add their features multiplied by corresponding weights. If, for example, you have 1000 sparse features only 3 or 4 of which are active in each instance (and the others are zeros) and 20 dense features that are all non-zeros, then it's pretty likely that dense features will make most of the impact while sparse features will add only a ...
10
It could be the way that you encode categorical variables.
If you do One Hot Encoding (dummy) each encoded feature will only have two possible values [0,1]. Binary variables normally have less importance in Decision trees given how it is computed the feature weight.
Let's say per example, that you are trying to predict the condition of a patient at the ...
9
There are several issues I see with the implementation. Some are just unnecessarily complicated ways of doing it, but some are genuine errors.
Primary takeaways
A: Try to start from the math behind the model. The logistic regression is a relatively simple one. Find the two equations you need and stick to them, replicate them letter by letter.
B: ...
9
If you use logistic regression and the cross-entropy cost function, it's shape is convex and there will be a single minimum. But during optimization, you may find weights that are near to optimal point and not exactly on the optimal point. This means that you can have multiple classifies that reduce the error and maybe set it to zero for the training data ...
8
Unless you have some very specific or exotic requirements, in order to perform logistic (logit and probit) regression analysis in R, you can use standard (built-in and loaded by default) stats package. In particular, you can use glm() function, as shown in the following nice tutorials from UCLA: logit in R tutorial and probit in R tutorial.
If you are ...
8
Background
I'll start with some background to help you research the solution yourself and then will add some specifics. What you refer to as "biased data" is more commonly known as unbalanced classes in the data science world. Also "customer turnover" is often referred to as churn.
Metrics
As hoards of Ng'ian devotees will undoubtably point out you need ...
8
One way to get confidence intervals is to bootstrap your data, say, $B$ times and fit logistic regression models $m_i$ to the dataset $B_i$ for $i = 1, 2, ..., B$. This gives you a distribution for the parameters you are estimating, from which you can find the confidence intervals.
7
Please take a look at the documentation. The first line shows the default parameters, which include penalty='l2' and C=1.0.
You actually cannot disable regularization completely, you can only regularize less... try setting C=1e10 for example.
7
Welcome to the Data Science forum.
Yes, data preprocessing is an important aspect of sentiment analysis for better results. What sort of preprocessing to be done largely depends on the quality of your data. You'll have to explore your corpus to understand the types of variables, their functions, permissible values, and so on. Some formats including html and ...
7
1) Yes, it makes sense. Trying to create features manually will help the learners (i.e. models) to graspe more information from the raw data because the raw data is not always in a form that is amenable to learning, but you can always construct features from it that are. The feature you are adding are based on one feature. This is common. However, your ...
7
Both logistic regression and SVM are linear models under the hood, and both implement a linear classification rule:
$$f_{\mathbf{w},b}(\mathbf{x}) = \mathrm{sign}(\mathbf{w}^T \mathbf{x} + b)$$
Note that I am regarding the "primal", linear form of the SVM here.
In both cases the parameters $\mathbf{w}$ and $b$ are estimated by minimizing a certain ...
7
Your decision boundary is a surface in 3D as your points are in 2D.
With Wolfram Language
Create the data sets.
mqtrue = 5;
cqtrue = 30;
With[{x = Subdivide[0, 3, 50]},
dat1 = Transpose@{x, mqtrue x + 5 RandomReal[1, Length@x]};
];
With[{x = Subdivide[7, 10, 50]},
dat2 = Transpose@{x, mqtrue x + cqtrue + 5 RandomReal[1, Length@x]};
];
View in 2D (...
6
A quick glance at the docs for LogisticRegressionWithLBFGS indicates that it uses feature scaling and L2-Regularization by default. I suspect that R's glm is returning a maximum likelihood estimate of the model while Spark's LogisticRegressionWithLBFGS is returning a regularized model estimate. Note how the estimated model weights of the Spark model are all ...
6
One fairly powerful R package for regression with an ordinal categorical response is VGAM, on the CRAN. The vignette contains some examples of ordinal regression, but admittedly I have never tried it on such a large dataset, so I cannot estimate how long it may take. You may find some additional material about VGAM on the author's page. Alternatively you ...
6
I suggest this tutorial on ordered logit: http://www.ats.ucla.edu/stat/r/dae/ologit.htm
It showcases the use of polr in the MASS package, and also explains the assumptions and how to interpret the results.
6
Try using both regression and decision trees. Compare the efficiency of each technique by using a 10 fold cross validation. Stick to the one with higher efficiency. It would be difficult to judge which method would be a better fit just by knowing that your dataset is continuous and, or categorical.
6
I do not understand the next line, the cost. Shouldn't the cost be equal to the xent, i.e. the loss? What is the cost function representing here?
The cost is the error (xent.mean()) + some regularization (0.01 * (w ** 2).sum())
Why is the bias initially set to 0 and not a random number like the weights?
It is possible and common to initialize the ...
6
They are both discriminative models, yes. The logistic regression loss function is conceptually a function of all points. Correctly classified points add very little to the loss function, adding more if they are close to the boundary. The points near the boundary are therefore more important to the loss and therefore deciding how good the boundary is.
SVM ...
6
There is a key difference:
Softmax regression provides class probabilities for mutually exclusive classes.
Logistic regression treats class membership for each class separately. Classes do not need to be mutually exclusive.
The two are equivalent for a scenario with two mutually exclusive classes - e.g. a "positive" and "negative" class - where softmax ...
Only top voted, non community-wiki answers of a minimum length are eligible
