24
votes
What does it mean to "share parameters between features and classes"
I will try to answer this question through logistic regression, one of the simplest linear classifiers.
The simplest case of logistic regression is if we have a binary classification task ($y \in\{0,...
- 8,018
17
votes
Accepted
Python implementation of cost function in logistic regression: why dot multiplication in one expression but element-wise multiplication in another
In this case, the two math formulae show you the correct type of multiplication:
$y_i$ and $\text{log}(a_i)$ in the cost function are scalar values. Composing the scalar values into a given sum over ...
- 29.1k
15
votes
How to get p-value and confident interval in LogisticRegression with sklearn?
The short answer is that sklearn LogisticRegression does not have a built in method to calculate p-values. Here are a few other posts that discuss solutions to this, however.
https://stackoverflow....
- 1,459
15
votes
Accepted
What is the difference between SGD classifier and the Logisitc regression?
Welcome to SE:Data Science.
SGD is a optimization method, while Logistic Regression (LR) is a machine learning algorithm/model. You can think of that a machine learning model defines a loss function, ...
- 2,294
15
votes
Accepted
Why continuous features are more important than categorical features in decision tree models?
It could be the way that you encode categorical variables.
If you do One Hot Encoding (dummy) each encoded feature will only have two possible values [0,1]. Binary variables normally have less ...
- 6,332
14
votes
How to get p-value and confident interval in LogisticRegression with sklearn?
One way to get confidence intervals is to bootstrap your data, say, $B$ times and fit logistic regression models $m_i$ to the dataset $B_i$ for $i = 1, 2, ..., B$. This gives you a distribution for ...
- 613
12
votes
Accepted
How to plot logistic regression decision boundary?
Regarding the code
You should plot the decision boundary after training is finished, not inside the training loop, parameters are constantly changing there; unless you are tracking the change of ...
- 9,392
12
votes
How to plot logistic regression decision boundary?
Your decision boundary is a surface in 3D as your points are in 2D.
With Wolfram Language
Create the data sets.
...
- 705
11
votes
Accepted
How do I implement the sigmoid function in Octave?
This will compute the sigmoid of a scalar, vector or matrix.
...
- 583
11
votes
Accepted
The differences between SVM and Logistic Regression
If you use logistic regression and the cross-entropy cost function, it's shape is convex and there will be a single minimum. But during optimization, you may find ...
- 14.1k
11
votes
My data is highly overlapping, but when I apply logistic regression, it is giving an impressive accuracy of 79%. Why?
Decision Tree, KNN, & Random Forest (Methods that are suitable for overlapping data)
This statement is false. All those methods are good when the decision surface (separating surface) has a ...
- 4,783
8
votes
Accepted
How does binary cross entropy work?
When doing logistic regression you start calculating a bunch of probabilities $p_i$ and your target is maximize the product of those probabilities (as they're considered independent events). The ...
- 251
8
votes
Are linear models better when dealing with too many features? If so, why?
There is some important information missing in your question, i.e. what the standard parameters are and what kind of logistic regression you use.
When you use ...
- 7,596
7
votes
Accepted
Do logistic regression and softmax regression do the same thing?
There is a key difference:
Softmax regression provides class probabilities for mutually exclusive classes.
Logistic regression treats class membership for each class separately. Classes do not need ...
- 29.1k
7
votes
Accepted
Should I use regularization every time?
Normally you use regularization. The exception is if you know the data generating process and can model it exactly. Then you merely estimate the model parameters. In general you will not know the ...
- 10.5k
7
votes
Accepted
Bad classification performance of logistic regression on imbalanced data in testing as compared to training
I suspect the reason is that the class balance in your test set is different from the class balance in your training set. That will throw everything off. The fundamental assumption made by ...
- 3,401
7
votes
Is this a good practice of feature engineering?
1) Yes, it makes sense. Trying to create features manually will help the learners (i.e. models) to graspe more information from the raw data because the raw data is not always in a form that is ...
- 71
7
votes
Accepted
What is the difference between SVM and logistic regression?
Both logistic regression and SVM are linear models under the hood, and both implement a linear classification rule:
$$f_{\mathbf{w},b}(\mathbf{x}) = \mathrm{sign}(\mathbf{w}^T \mathbf{x} + b)$$
Note ...
- 2,121
7
votes
Accepted
Risk prediction vs classification model
I will try to answer your question as shortly as possible.
Yes, if you define probability as a risk, then the probabilities are risk scores. But, there's a catch in these scenarios, you will have to ...
- 1,156
6
votes
How to get p-value and confident interval in LogisticRegression with sklearn?
This is still not implemented and not planned as it seems out of scope of sklearn, as per Github discussion #6773 and #13048.
However, the documentation on linear models now mention that (P-value ...
- 2,378
6
votes
What cost function and penalty are suitable for imbalanced datasets?
So you ask how does class imbalance affect classifier performance under different losses?
You can make a numeric experiment.
I do binary classification by logistic regression. However, the ...
- 1,551
6
votes
Learning rate in logistic regression with sklearn
sklearn.linear_model.LogisticRegression doesn't use SGD, so there's no learning rate.
I think sklearn.linear_model.SGDClassifier...
- 4,356
6
votes
Accepted
AUC and classification report in Logistic regression in python
In order to calculate the AUC, you need to have probabilities. Therefore you should use the following function:
...
- 873
6
votes
How to perform Logistic Regression with a large number of features?
In order to reduce your model down to 7 variables there are a few approaches you could take:
PCA (unsupervised): this creates "new" linear combinations of your data where each proceding component ...
- 543
6
votes
How to perform Logistic Regression with a large number of features?
You're taking the "Rule of 10" too seriously. It's a very rough rule of thumb. It's not intended to be used like you are using it.
It sounds like you are thinking: "I have only 70 positive ...
- 3,401
6
votes
Accepted
Why Root Finding is important in Logistic Regression? (i.e. Newton Raphson)
This blogpost gives a broad answer to your question. In short, Newton's method is not used to find a root of the loss, but a root of the gradient. If you find a root of the gradient, then you are ...
- 6,111
6
votes
Accepted
Is this a good practice of feature engineering?
If you can keep adding new data (based on a main concept such as area i.e. the ZIP code) and the performance of your model improves, then it is of course allowed... assuming you only care about the ...
- 15k
6
votes
Accepted
What are the differences between logistic and linear regression?
As you have mentioned, the output of linear regression is a real value while logistic regression's represents classes(classification). Their main difference is this.
The loss function of linear ...
- 14.1k
6
votes
Accepted
Logistic regression cost function
The cost function of the Logistic Regression derived via Maximum Likelihood Estimation:
If y = 1 (positive): i) cost = 0 if prediction is correct (i.e. h=1), ii) cost $\rightarrow \infty $ if $...
- 4,299
6
votes
When to use Random Forest
Adding some extra general points to the previous answer:
As a decision tree algorithm, Random Forests are less influenced by outliers than
other algorithms. A good discussion about it is here.
They ...
- 161
Only top scored, non community-wiki answers of a minimum length are eligible