You need to take a step back in order to decide which model would suit best for your use case. Before doing that predict_proba is definitely important to calculate the posterior probability of the class labels, but it isn't great for comparing against other model outputs (especially to decide which models suits best for your prediction).
Logistic Regression, QDA and LDA all have different approaches. Logistic regression is based on maximum likelihood estimation while LDA and QDA are based on Bayes theorem. To understand which classifier best suits your model we need to go over the assumptions (assuming you know the mathematical expressions) and then you can judge which best apply to you.
1. Logistic Regression
In Logistic regression, it is possible to directly get the probability of an observation for a class (Y=k) for a particular observation (X=x). There is nothing to assume to run logistic regression for classification. Its generally a safe go to method which is not exigent and is robust.
2. LDA & QDA
LDA and QDA algorithm is based on Bayes theorem and classification of an observation is done in following two steps.
- Identify the distribution for input X for each of the class (or groups ex Y=k1, k2, k3 etc )
- Flip the distribution using Bayes theorem to calculate the probability Pr(Y=k|X=x)
Following are the assumption required for LDA and QDA:
- LDA Assumption:
- Common covariance across all response classes σ2 ( for ex σk1 = σk2 =
σk3 for k1, k2 , k3 response classes )
- Distribution of observation in each of the response classes is normal
with a class-specific mean (µk) and common covariance σ.
- QDA Assumption:
- Different covariance for each of the response classes. For ex – σk1,
σk2, σk3 for response class k1, k2, k3 etc.
- Distribution of observation in each of the response class is normal
with a class-specific mean (µk) and class-specific covariance (σk2).
- LDA (Linear Discriminant Analysis) is used when a linear boundary is required between classifiers.
- QDA (Quadratic Discriminant Analysis) is used to find a non-linear boundary between classifiers.
- LDA/QDA, when all its requirements met classifies better than logistic regression (more efficient).
- Logistic regression is not sensitive to outliers while LDA/QDA is.
- LDA and QDA work well when class separation and normality assumption holds.
- Logistic regression has an edge on LDA/QDA for dataset that is not normal.