3
$\begingroup$

Let's say I already have a logistic regression model (or other) with N number of explanatory variables and is 70% accurate. Now if there are other variables available, how would I test if the new variables would improve my accuracy without building new model.

$\endgroup$
1
  • $\begingroup$ I don't think there is any way without building a new model. $\endgroup$
    – Ankit Seth
    May 18, 2018 at 12:45

3 Answers 3

5
$\begingroup$

I do not think you can estimate the effect of a variable without adding it to the model. This is because the effect of a variable on the model's discriminatory power depends on

  1. the strength of association between the outcome variable and the new variable
  2. whether the new variable are collinear with some of the old variables

You could in principle estimate both strength of association and collinearity, but it probably is bad practice and would result in overfitting.

Also in general I think it is best to not use accuracy to evaluate a logistic regression (see ref) but rather a proper scoring rule like Brier Score.

Further, when comparing two nested models (i.e. where one model contains a subset of the variables of the other model) I believe that best practice is to compare the AIC or BIC, or perform a likelihood ratio test.

$\endgroup$
2
$\begingroup$

Another thing to consider, if you are unable to refit the full model, but are able to access its fitted values (i.e.as score or probability values), you can try setting up a new logistic regression model with both the fitted values and the new variable as covariates for the same response variable. In this way you will see, whether all of the newly provided information was already captured by the existing model, or will the new covariate add further out of sample accuracy.

$\endgroup$
1
$\begingroup$

There are ways to do this. It can be viewed as an online streaming algorithm. Check this paper: "Online updating method with new variables for big data streams".

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.