1
$\begingroup$

I'm having a hard time to interpret my result of the logistic regression.

I have a few question. Firstly, how can I check if a feature is more important to the others, like that there is a real significane by it.

I have an accuray of 0.58 which is pretty bad. Anyway, why does the RFECV feature selector of sklearn tell me to use feature1 to get the best result but by training my model where I use hte statsmodels library, the model will all 5 variables is slightly better?

Also, why does my two models when I train it with sklearn and statsmodels is different in result? enter image description here

It really confuses me everything. It would be nice if someone could tell me an easy way to interpret my results and do it in one library all.

My codesnippets

import statsmodels.api as sm
X = df_n_4[cols]
y = df_n_4['Survival']

# use train/test split with different random_state values
# we can change the random_state values that changes the accuracy scores
# the scores change a lot, this is why testing scores is a high-variance estimate
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=2)
logit_model = sm.Logit(y_train, X_train).fit()
y_pred = logit_model.predict(X_test)
cf_matrix = confusion_matrix(y_test, y_pred.round())
sns.heatmap(cf_matrix, annot=True)
plt.title('Accuracy:{}'.format(accuracy_score(y_test, y_pred.round())))
plt.ylabel('Actual Szenario');
plt.xlabel('Predicted Szenario');
plt.show()

Part 2

X = df_n_4[cols]
y = df_n_4['Srvival']

# use train/test split with different random_state values
# we can change the random_state values that changes the accuracy scores
# the scores change a lot, this is why testing scores is a high-variance estimate
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=2)
print(len(y_train)," Testdata")

# check classification scores of logistic regression
logreg = LogisticRegression()
logreg.fit(X_train, y_train)
y_pred = logreg.predict(X_test)
print('Train/Test split results:')
print(logreg.__class__.__name__+" accuracy is %2.3f" % accuracy_score(y_test, y_pred))
plt.title('Accuracy Score: {0}, Variablen: feature1'.format(round(accuracy_score(y_test, y_pred),2), size = 15))
cf_matrix = confusion_matrix(y_test, y_pred)
sns.heatmap(cf_matrix, annot=True)
plt.ylabel('Actual Szenario');
plt.xlabel('Predicted Szenario');
plt.show()

I don't have too much statistical background so please take it easy on me! I will provide any further information you need. Thanks so much I'm stuck since a week, as all I read and further do, confuses me even more.

Improve this question
$\endgroup$
10
  • 2
    $\begingroup$ The reason the output from scikit-learn and statsmodel differ is that scikit-learn uses an L2 penalty by default, see also the documentation. $\endgroup$
    – Oxbowerce
    Jan 16 at 20:08
  • 1
    $\begingroup$ With regards to what library you use it depends on whether or not you want to use an L2 penalty, since this seems to only be implemented within scikit-learn. If you do not need any regularization at all you can use both libraries, however statsmodels will be more focused on statistics (see also the output you receive). $\endgroup$
    – Oxbowerce
    Jan 16 at 20:54
  • 1
    $\begingroup$ If with feature importance you mean the odds ratio displayed by scikit-learn, these can simply be gotten by taking e to the power of the coefficient (i.e. a coefficient of 0.000167 lead to an odds ratio of e^0.000167 = 1.000167. Since the statsmodels library also includes the coefficients in its output you can use numpy.exp to convert those to an odds ratio. I'm not sure however if this is a good way to measure feature importance as its based on the regression coefficients which are impacted by the scale of your features. $\endgroup$
    – Oxbowerce
    Jan 16 at 21:59
  • 1
    $\begingroup$ In order to interpret significant features using stats models , you need to look at the p-value. For features where the p-value is less than your chosen level of significance (0.05 or 0.01, etc), generally 0.05, are the features that are significant in the model you fit. In your example, as we see none of the variables have p value less than 0.05 , therefore none of the features are significant. $\endgroup$
    – Ankita Talwar
    Jan 16 at 22:58
  • 1
    $\begingroup$ It does not mean that none of the variables can be used as good predictors for the model, it only means that when you chose all features to fit the logistic model none of them significantly affect the outcome variable. This situation might change when you experiment with different combination of features. You can select features using forward selection, backward elimination and such techniques. There are various feature selection methods. This is when you are trying to fit the model statistically. $\endgroup$
    – Ankita Talwar
    Jan 16 at 23:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.