# Questions tagged [explainable-ai]

Use for questions about explainable artificial intelligence (AI), which aims at understanding, interpreting, and explaining the decisions that have been made by complex AI systems

45 questions
Filter by
Sorted by
Tagged with
0answers
9 views

### Are SHAP values additive across examples?

I understand that SHAP values have a property called additivity that means that if you add the SHAP value of each explanatory variable of a particular example to the average prediction of the model on ...
0answers
9 views

### Explanation on how to interpret force_plot result by using SHAP

I have applied SHAP for explaining the outcome of my neural network. For the force_plot I have obtained the following output when trying to look at multiple ...
0answers
12 views

### DeepTaylor and LRP

I'm studying explainable AI. Is it possible to apply DeepTaylor or Layer-wise Relevance Propagation that was made for NNs with ...
0answers
8 views

### Layer-wise Relevance Propagation with the swish activations

I'm studying approaches to explain DNN. I've found a lot of papers on LRP (and Deep Taylor decomposition), but they all explain NNs with the ReLU activations. I'm wondering why nobody applied LRP to ...
0answers
75 views

### How SHAP value explains contribution of features for outliers event?

I'm trying to understand and experiment with how the SHAP value can explain behaviour for each outlier events (rows) and how it can be related to shap.force_plot(). ...
1answer
44 views

### Grad_CAM for time series

I am new to deep learning and trying to build a Grad-cam from time series data. Shape of my input sample is (188,1), its an ECG signal and I have a cnn-1D model for classification. Keras provides ...
1answer
46 views

### How to evaluate the “importance” of a variable in a function

Let's say that we have $$f(x,y,z) = x/k - (y/k) ((z - x/k)/(z - y/k))$$ $$k = constant \in ]0,1[$$ And I need to show in some way that the variable $x$ is more important in some metric that I don't ...
0answers
15 views

### Running into a shape error when implementing GradCAM with Keras

I've been trying to implement GradCAM from keras tutorial (https://keras.io/examples/vision/grad_cam/). I have a custom model with a binary classification with 2 units in the final layer. ...
1answer
16 views

### Shapley contribution when coalition is 0

I am exploring Shapley for channel attribution based on [here] Consider C1, C2, C3, C4 as 4 channels in question. Some of the coalition does not have value, such as ...
2answers
54 views

### Different feature importance results between DNN, Random Forests and Gradient Boosted Decision Trees

I've been modeling metabolite data with 3 different regressor models. I get similar results from running feature importance with Random Forest model and Gradient Boosted Decision Trees (where I used ...
1answer
106 views

### Shapley values without intercept (or without expected_value)

I have a model and I want to derive its interpretability by using feature contributions. In the end, I want to have some contribution per feature such that the sum of contributions equals the ...
0answers
22 views

### How much does each tree of gradient boosting contribute to the global feature importance?

Let's say we are training a GBDT in the Titanic dataset. We have 3 trees in the GBDT. You extract the first tree and calculate the feature importance (no matter if cover, gain...), and Age importance =...
0answers
28 views

### Passing reduced/different feature data to LimeTabularExplainer compared to the original model

I am trying to use LimeTabularExplainer class and explain_instance function to find explainations of my LightGbm (lgb) model. However, the lgb model uses complex feature set which are not ...
1answer
22 views

### Types of maps in Interpretable Machine Learning

I have worked on Interpretable Machine Learning (IML) for over 1 year. However, there are some terminologies that always make me confused. For example, saliency maps/heat maps. Are they same? Are ...
0answers
30 views

### What is the meaning of Information with respect to interpretability approaches in machine learning?

I was going through a pre-print on arXiv named "Quantifying interpretability and trust in machine learning systems". There, I found that a comparison of two interpretability approaches - ...
1answer
18 views

### Explainable AI, how did the computer classified Text?

My question is not about explaining the model or the algorithm like which neurons were triggered and what are the parameters of perceptrons. I will explain further The problem I have medical reports I ...
1answer
188 views

### Can Shapley/Lime values be used for unsupervised learning?

One thing that is really useful when trying to understand what a machine learning model does, is seeing why some instances got predicted. For that Shapley Values and Lime are really usefull. But can ...
0answers
222 views

### Partial Dependence Plot and categorical variables

While reading about machine learning explainability and Partial Dependence Plot (PDP) in this book, the following appeared when dealing with categorical variables: For each of the categories, we get ...
1answer
32 views

### Uncertainty in connection to explainability

When I write "uncertainty" in this post I mean: If I have a classifier into $a_1,..,a_n$ categories and for an observation $x$ I classify $x$ to $a_i$ with probability $p_i$, then the ...
2answers
76 views

### What is a “surrogate model”?

While reading about model explainability and model accountability, the term surrogate model keeps appearing. I had an idea about what it is but it does not seem to make sense anymore: What is a ...
0answers
38 views

### Ways to visualize the outcome of machine learning interpretability techniques (for image classification)

“Machine Learning Interpretability” or “Explainable Artificial Intelligence” has become quite popular in the machine learning community and in recent research. The goal is to make complex (deep ...
0answers
39 views

### Aggregate SHAP importances from different models

A couple of questions on the SHAP approach to the estimation of feature importance. I would like to use the random forest, logistic regression, SVM, and kNN to train four classification models on a ...
1answer
1k views

### How is the “base value” of SHAP values calculated?

I'm trying to understand how the base value is calculated. So I used an example from SHAP's github notebook, Census income classification with LightGBM. Right after I trained the lightgbm model, I ...
0answers
37 views

### Can we use Shap to interpret output changes?

Can we calculate the difference between Shapley values to interpret changes in the output? More precisely, if we get Shapley values for two different inputs, can we compare them to understand how much ...
2answers
19 views

### What does a conservative technique mean in the context of neural networks?

I am reading this article on Layer-wise relevance propagation method and I can't understand this particular paragraph LRP is a conservative technique, meaning the magnitude of any output y is ...
1answer
54 views

### treeExplainer algorithm intuition

I'm reading the paper about the treeExplainer; the pseudo-code of Algorithm 1 is a bit cryptical as most of the variables are not even defined (same with sampling and all details involved). Is there a ...
5answers
2k views

### What is the difference between explainable and interpretable machine learning?

O’Rourke says that explainable ML uses a black box model and explains it afterwards, whereas interpretable ML uses models that are no black boxes. Christoph Molnar says interpretable ML refers to the ...
0answers
123 views

### Tree Path Dependent expected value

I stuck on this topic for couple days and seems I need your help to understand what is expected value in TreeExplainer when feature_perturbation = tree_path_dependent, precisely what is expected value ...
1answer
249 views

### Interpretable xgboost - Calculate cover feature importance

When trying to interpret the results of a gradient boosting (or any decision tree) one can plot the feature importance. There are same parameters in the xgb api such as: weight, gain, cover, ...
0answers
32 views

### Does EDA helps only in case of linear regression?

I know what Explanatory data analysis is and how it helps us investigate and understand the data. What I dont understand is how does this help in case of nonlinear relationships? I mean if I'm using ...
1answer
42 views

### LIME Random Forest explanations:

I'm using LIME to explain my random forest model. Everything is working great. However, I don't quite understand the image that is generated. Taking the example from the Readme: How can it predict ...
0answers
188 views

### Multi-valued categorical features in LIME

I am working with the LIME implementation by Marco Ribeiro (https://github.com/marcotcr/lime). Specifically, I am utilizing the LimeTabularExplainer as I have a mixture of numerical and categorical ...
1answer
425 views

### Explainable AI and unsupervised algorithms

There are several packages that allow explaining ML algorithms (Lime, Shap and so on). However, it is not clear how we can explain unsupervised algorithms for example, if we use PCA for dimensionality ...
2answers
297 views

### Reasons that LIME and SHAP might not agree with intuition

I'm leveraging the Python packages lime and shap to explain single (test-set) predictions that a basic, trained model is making ...
2answers
643 views

### Why continuous features are more important than categorical features in decision tree models?

I have both categorical and continuous features in my prediction model and want to select (and rank) most important features. I have converted all categorical variables into dummy variables using one ...
2answers
146 views

### How to plot number of Trees and OOBs score with Grid Search

I searched to find the answer but I don´t find something with Grid Search. I create a random forest and gradient boosting regressor with grid search. Now I want to make a visualization to see if the ...
0answers
247 views

### SHAP Explanations in case of repeated train/test split

I am building a XGBoost model with Python and trying to explain it using the beautiful shap package. Apart from calculating SHAP values of each feature, I'd like to show graphs such as the two that ...
2answers
380 views

### Explainable anomaly detection

There are plenty of working for explaining prediction in supervised learning (e.g. SHAP values, LIME). What about for anomaly detection in unsupervised learning? Is there any model for which there ...
1answer
83 views

### What is the meaning of an empty SHAP graph in Explainable AI?

Using Python, I created a neural network to perform predictions on a binary class dataset (e.g. will a passenger survive the Titanic?). I am using the SHAP package to explain individual predictions. ...
1answer
59 views

### How to restructure my dataset for interpretability without losing performance?

What I am doing: I am predicting product ratings using boosted trees (XGBoost) with a dataset in this format: What I want to do: I want to use SHAP TreeExplainer to interpret each prediction my model ...
1answer
863 views

### Explanation of how DeepExplainer works to obtain SHAP values in simple terms

I have been using DeepExplainer (DE) to obtain the approximate SHAP values for my MLP model. I am following the SHAP Python library. Now I'd like learn the logic behind DE more. From the relevant ...
1answer
231 views

### Is it valid to compare SHAP values across models?

Let's say I have three models: a random forest with 100 trees a random forest with 1000 trees an xgboost model. I can rank the importance of my features on my dataset for each model using SHAP, and ...
2answers
87 views

### Explainability ML Methods

I am currently working on a Machine-Learning Model. In order to explain how it works, I have looked at Partial Dependence Plots, Feature Importance and all kinds of methods, but one thing still ...
1answer
142 views

### Evaluating machine learning explainers?

I'm working on a project where multiple machine learning explainers (LIME and SHAP, potentially more coming) are applied to pre-trained models (neural networks) to help explain the predictions of ...
1answer
253 views

### How to handle the parameter space of neural networks?

This question is very broad (and might even be closed as "too broad"). It can be considered as a beginners question, because it is largely about getting started in terms of heading into a direction ...