14
$\begingroup$

In Decision Trees, we can understand the output of the tree structure and we can also visualize how the Decision Tree makes decisions. So decision trees have explainability (their output can be explained easily.)

Do we have explainability in Neural Networks like with Decision Trees?

$\endgroup$
3
  • 1
    $\begingroup$ A recent model-agnostic framework is the LIME model. $\endgroup$
    – Emre
    May 22, 2017 at 16:56
  • $\begingroup$ In the field of object recognition/classification using neural networks, heatmaps are popular to visualize/explain a decision such as in heatmapping.org. Tutorials and interactive demonstrations are available. $\endgroup$ May 23, 2017 at 8:10
  • $\begingroup$ In fact according to this new paper arxiv.org/abs/2210.05189 NNs can be represented as a Decision Tree $\endgroup$
    – charL
    Oct 18, 2022 at 4:48

4 Answers 4

9
$\begingroup$

I disagree with the previous answer and with your suggestion for two reasons:

1) Decision trees are based on simple logical decisions which combined together can make more complex decisions. BUT if your input has 1000 dimensions, and the features learned are highly non linear, you get a really big and heavy decision tree which you won't be able to read/understand just by looking at the nodes.

2) Neural networks are similar to that in the sens that the function they learn is understandable only if they are very small. When getting big, you need other tricks to understand them. As @SmallChess suggested, you can read this article called Visualizing and Understanding Convolutional Networks which explains for the particular case of convolutional neural networks, how you can read the weights to understand stuff like "it detected a car in this picture, mainly because of the wheels, not the rest of the components".

These visualizations helped a lot of researchers to actually understand weaknesses in their neural architectures and helped to improve the training algorithms.

$\endgroup$
3
  • $\begingroup$ :-) I found the paper itself harder to understand than the deep convolutional network itself. It's a very mathematical paper. $\endgroup$
    – SmallChess
    May 22, 2017 at 12:40
  • 1
    $\begingroup$ Sorry, I cited the wrong article :-) I just changed it, this one is more graphical, the idea of reversing the convnet is not really hard if you know how convnets work. In the same way, Google deep dream use back propagation to project a particular output in the input space. $\endgroup$
    – Robin
    May 22, 2017 at 12:47
  • $\begingroup$ There is a video where Matt Zeiler expains many of these ideas, called Deconconvolution networks $\endgroup$
    – Alex
    May 8, 2018 at 13:30
7
$\begingroup$

No. Neural network is generally difficult to understand. You trade predictive power for model complexity. While it's possible to visualize the NN weights graphically, they don't tell you exactly how a decision is made. Good luck trying to understanding a deep network.

There is a popular Python package (and it has a paper) that can model a NN locally with a simpler model. You may want to take a look.

$\endgroup$
1
  • 1
    $\begingroup$ haha. I know what it feels like. hugs :D $\endgroup$
    – Dawny33
    May 22, 2017 at 11:25
0
$\begingroup$

https://arxiv.org/abs/1704.02685 provide a NN specific local explanation tool : deep lift. It works by propagating the difference in activation between the instance you want to explain and a reference instance. Getting a reference is a bit tricky, but the tool appears to be interpretable and scalable overall. It can be used on tabular data.

$\endgroup$
0
$\begingroup$

In general, decision trees are easily understandable due to its structure. However, in most application they become so big that you easily lose sight. Additionally, in most cases you would want to use Random Forest as an ensemble method instead of a single Decision Tree and then again there is not one single tree that you can explain.

For neural networks there is a new research approach is coming up called "Explainable AI" which tries to make us understand the reasons for a neural network prediction. One method is the so-called integrated gradient that calculates the importance of each input feature for the prediction.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.