1
$\begingroup$

I want to predict the probability that an individual credit_balance is more than value N is true

I need to compare 3 classification methods: Logistic regression, Least discriminant and quadratic discriminant.

For 1 sample prediction I get predict_proba for each of them with different values, how can I decide which model is the best for my prediction?

  1. linear regression predict_proba result is [[0.93227393 0.06772607]]
  2. LDA predict_proba is [[0.94144572 0.05855428]]
  3. QDA precit_proba is [[9.99999999e-01 1.24419207e-09]]

What parameters should we look at to decide which classification is the best to predict the model?

$\endgroup$
  • $\begingroup$ It would be better to simply do ensembling/stacking(simple average of preds) over all your models to reduce the overall bias imparted by each one... $\endgroup$ – Aditya Apr 16 '18 at 6:06
  • $\begingroup$ by linear regression Im assuming you mean logistic regression $\endgroup$ – moksha Apr 16 '18 at 10:25
2
$\begingroup$

You need to take a step back in order to decide which model would suit best for your use case. Before doing that predict_proba is definitely important to calculate the posterior probability of the class labels, but it isn't great for comparing against other model outputs (especially to decide which models suits best for your prediction).

Logistic Regression, QDA and LDA all have different approaches. Logistic regression is based on maximum likelihood estimation while LDA and QDA are based on Bayes theorem. To understand which classifier best suits your model we need to go over the assumptions (assuming you know the mathematical expressions) and then you can judge which best apply to you.


1. Logistic Regression

In Logistic regression, it is possible to directly get the probability of an observation for a class (Y=k) for a particular observation (X=x). There is nothing to assume to run logistic regression for classification. Its generally a safe go to method which is not exigent and is robust.


2. LDA & QDA

LDA and QDA algorithm is based on Bayes theorem and classification of an observation is done in following two steps.

  • Identify the distribution for input X for each of the class (or groups ex Y=k1, k2, k3 etc )
  • Flip the distribution using Bayes theorem to calculate the probability Pr(Y=k|X=x)

Following are the assumption required for LDA and QDA:

  1. LDA Assumption:
    • Common covariance across all response classes σ2 ( for ex σk1 = σk2 = σk3 for k1, k2 , k3 response classes )
    • Distribution of observation in each of the response classes is normal with a class-specific mean (µk) and common covariance σ.
  2. QDA Assumption:
    • Different covariance for each of the response classes. For ex – σk1, σk2, σk3 for response class k1, k2, k3 etc.
    • Distribution of observation in each of the response class is normal with a class-specific mean (µk) and class-specific covariance (σk2).

Notes:

  • LDA (Linear Discriminant Analysis) is used when a linear boundary is required between classifiers.
  • QDA (Quadratic Discriminant Analysis) is used to find a non-linear boundary between classifiers.
  • LDA/QDA, when all its requirements met classifies better than logistic regression (more efficient).
  • Logistic regression is not sensitive to outliers while LDA/QDA is.

To conclude:

  • LDA and QDA work well when class separation and normality assumption holds.
  • Logistic regression has an edge on LDA/QDA for dataset that is not normal.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.