For neural network feature importance, can I zero-out all features except one in order to gauge that feature's importance? I know shuffling a feature is one approach.

For example, leaving in the 4th feature.

feature_4 = [

_, probabilities = model.predict(feature_4)

The non-linear output of activation functions worries me because activation of the whole is not equal to the sum of individual activations:

from scipy.special import expit #aka sigmoid

>>> expit(2.0)

>>> expit(1.0)+expit(1.0)

And softmax seems much less straightforward in comparison to sigmoid.

  • $\begingroup$ i suppose another reason to shuffle a column would be to preserve the interaction with other features $\endgroup$
    – Kalanos
    Dec 23 '21 at 21:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.