For neural networks, can we tell which parameters are responsible for which features?

For example, in an image classification task, each pixel of an image is a feature. Can I somehow find out which parameters encode the learned information, say from the top-left pixel of my training instances?


Yes, at least you can identify what pixels' are contributing most in the prediction.

Tool like Layerwise Relevance Propagation, used for Explainable AI, serves the similar purpose and evaluate the values(weights) during back propagation and evaluate what pixels are contributing most.

Many opensource implementation are available and on similar track, instead of identifying just relevant pixel you can perform activity for each pixel.

I believe that I'm able to answer your question.

  • $\begingroup$ hi im aware of feature attribution methods but not sure how can i move from there to parameter attribution. can you elaborate a little bit? $\endgroup$
    – SpiderRico
    Apr 8 '20 at 12:00
  • $\begingroup$ Its like back-propagation of the weights till we reached to very first layer. Every neuron in DNN, is a representation of Activation(Linear Equation). Neuron which is contributed in prediction, picked up first. Then during back-propagation identify the weights of previous layer, which contributed in prediction.. Similarly, we traverse in backward direction till we reached to input layer. Finally we have the weights associated with each I/P, for that prediction. Though difficult to explain in few words, tried my best. Refer open documents/videos for more explanation. $\endgroup$ Apr 9 '20 at 10:14
  • $\begingroup$ @SpiderRico, let me know in case you want to discuss more on this? $\endgroup$ Apr 13 '20 at 12:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.