Questions tagged [explainable-ai]

Use for questions about explainable artificial intelligence (AI), which aims at understanding, interpreting, and explaining the decisions that have been made by complex AI systems

Filter by
Sorted by
Tagged with
0
votes
0answers
9 views

Are SHAP values additive across examples?

I understand that SHAP values have a property called additivity that means that if you add the SHAP value of each explanatory variable of a particular example to the average prediction of the model on ...
0
votes
0answers
9 views

Explanation on how to interpret force_plot result by using SHAP

I have applied SHAP for explaining the outcome of my neural network. For the force_plot I have obtained the following output when trying to look at multiple ...
0
votes
0answers
12 views

DeepTaylor and LRP

I'm studying explainable AI. Is it possible to apply DeepTaylor or Layer-wise Relevance Propagation that was made for NNs with ...
0
votes
0answers
8 views

Layer-wise Relevance Propagation with the swish activations

I'm studying approaches to explain DNN. I've found a lot of papers on LRP (and Deep Taylor decomposition), but they all explain NNs with the ReLU activations. I'm wondering why nobody applied LRP to ...
2
votes
0answers
75 views

How SHAP value explains contribution of features for outliers event?

I'm trying to understand and experiment with how the SHAP value can explain behaviour for each outlier events (rows) and how it can be related to shap.force_plot(). ...
0
votes
1answer
44 views

Grad_CAM for time series

I am new to deep learning and trying to build a Grad-cam from time series data. Shape of my input sample is (188,1), its an ECG signal and I have a cnn-1D model for classification. Keras provides ...
2
votes
1answer
46 views

How to evaluate the “importance” of a variable in a function

Let's say that we have $$f(x,y,z) = x/k - (y/k) ((z - x/k)/(z - y/k))$$ $$k = constant \in ]0,1[$$ And I need to show in some way that the variable $x$ is more important in some metric that I don't ...
0
votes
0answers
15 views

Running into a shape error when implementing GradCAM with Keras

I've been trying to implement GradCAM from keras tutorial (https://keras.io/examples/vision/grad_cam/). I have a custom model with a binary classification with 2 units in the final layer. ...
0
votes
1answer
16 views

Shapley contribution when coalition is 0

I am exploring Shapley for channel attribution based on [here][1] Consider C1, C2, C3, C4 as 4 channels in question. Some of the coalition does not have value, such as ...
4
votes
2answers
54 views

Different feature importance results between DNN, Random Forests and Gradient Boosted Decision Trees

I've been modeling metabolite data with 3 different regressor models. I get similar results from running feature importance with Random Forest model and Gradient Boosted Decision Trees (where I used ...
4
votes
1answer
106 views

Shapley values without intercept (or without `expected_value`)

I have a model and I want to derive its interpretability by using feature contributions. In the end, I want to have some contribution per feature such that the sum of contributions equals the ...
0
votes
0answers
22 views

How much does each tree of gradient boosting contribute to the global feature importance?

Let's say we are training a GBDT in the Titanic dataset. We have 3 trees in the GBDT. You extract the first tree and calculate the feature importance (no matter if cover, gain...), and Age importance =...
1
vote
0answers
28 views

Passing reduced/different feature data to LimeTabularExplainer compared to the original model

I am trying to use LimeTabularExplainer class and explain_instance function to find explainations of my LightGbm (lgb) model. However, the lgb model uses complex feature set which are not ...
0
votes
1answer
22 views

Types of maps in Interpretable Machine Learning

I have worked on Interpretable Machine Learning (IML) for over 1 year. However, there are some terminologies that always make me confused. For example, saliency maps/heat maps. Are they same? Are ...
1
vote
0answers
30 views

What is the meaning of Information with respect to interpretability approaches in machine learning?

I was going through a pre-print on arXiv named "Quantifying interpretability and trust in machine learning systems". There, I found that a comparison of two interpretability approaches - ...
1
vote
1answer
18 views

Explainable AI, how did the computer classified Text?

My question is not about explaining the model or the algorithm like which neurons were triggered and what are the parameters of perceptrons. I will explain further The problem I have medical reports I ...
1
vote
1answer
188 views

Can Shapley/Lime values be used for unsupervised learning?

One thing that is really useful when trying to understand what a machine learning model does, is seeing why some instances got predicted. For that Shapley Values and Lime are really usefull. But can ...
1
vote
0answers
222 views

Partial Dependence Plot and categorical variables

While reading about machine learning explainability and Partial Dependence Plot (PDP) in this book, the following appeared when dealing with categorical variables: For each of the categories, we get ...
1
vote
1answer
32 views

Uncertainty in connection to explainability

When I write "uncertainty" in this post I mean: If I have a classifier into $a_1,..,a_n$ categories and for an observation $x$ I classify $x$ to $a_i$ with probability $p_i$, then the ...
3
votes
2answers
76 views

What is a “surrogate model”?

While reading about model explainability and model accountability, the term surrogate model keeps appearing. I had an idea about what it is but it does not seem to make sense anymore: What is a ...
0
votes
0answers
38 views

Ways to visualize the outcome of machine learning interpretability techniques (for image classification)

“Machine Learning Interpretability” or “Explainable Artificial Intelligence” has become quite popular in the machine learning community and in recent research. The goal is to make complex (deep ...
1
vote
0answers
39 views

Aggregate SHAP importances from different models

A couple of questions on the SHAP approach to the estimation of feature importance. I would like to use the random forest, logistic regression, SVM, and kNN to train four classification models on a ...
3
votes
1answer
1k views

How is the “base value” of SHAP values calculated?

I'm trying to understand how the base value is calculated. So I used an example from SHAP's github notebook, Census income classification with LightGBM. Right after I trained the lightgbm model, I ...
1
vote
0answers
37 views

Can we use Shap to interpret output changes?

Can we calculate the difference between Shapley values to interpret changes in the output? More precisely, if we get Shapley values for two different inputs, can we compare them to understand how much ...
0
votes
2answers
19 views

What does a conservative technique mean in the context of neural networks?

I am reading this article on Layer-wise relevance propagation method and I can't understand this particular paragraph LRP is a conservative technique, meaning the magnitude of any output y is ...
2
votes
1answer
54 views

treeExplainer algorithm intuition

I'm reading the paper about the treeExplainer; the pseudo-code of Algorithm 1 is a bit cryptical as most of the variables are not even defined (same with sampling and all details involved). Is there a ...
14
votes
5answers
2k views

What is the difference between explainable and interpretable machine learning?

O’Rourke says that explainable ML uses a black box model and explains it afterwards, whereas interpretable ML uses models that are no black boxes. Christoph Molnar says interpretable ML refers to the ...
1
vote
0answers
123 views

Tree Path Dependent expected value

I stuck on this topic for couple days and seems I need your help to understand what is expected value in TreeExplainer when feature_perturbation = tree_path_dependent, precisely what is expected value ...
2
votes
1answer
249 views

Interpretable xgboost - Calculate cover feature importance

When trying to interpret the results of a gradient boosting (or any decision tree) one can plot the feature importance. There are same parameters in the xgb api such as: weight, gain, cover, ...
1
vote
0answers
32 views

Does EDA helps only in case of linear regression?

I know what Explanatory data analysis is and how it helps us investigate and understand the data. What I dont understand is how does this help in case of nonlinear relationships? I mean if I'm using ...
1
vote
1answer
42 views

LIME Random Forest explanations:

I'm using LIME to explain my random forest model. Everything is working great. However, I don't quite understand the image that is generated. Taking the example from the Readme: How can it predict ...
2
votes
0answers
188 views

Multi-valued categorical features in LIME

I am working with the LIME implementation by Marco Ribeiro (https://github.com/marcotcr/lime). Specifically, I am utilizing the LimeTabularExplainer as I have a mixture of numerical and categorical ...
1
vote
1answer
425 views

Explainable AI and unsupervised algorithms

There are several packages that allow explaining ML algorithms (Lime, Shap and so on). However, it is not clear how we can explain unsupervised algorithms for example, if we use PCA for dimensionality ...
2
votes
2answers
297 views

Reasons that LIME and SHAP might not agree with intuition

I'm leveraging the Python packages lime and shap to explain single (test-set) predictions that a basic, trained model is making ...
9
votes
2answers
643 views

Why continuous features are more important than categorical features in decision tree models?

I have both categorical and continuous features in my prediction model and want to select (and rank) most important features. I have converted all categorical variables into dummy variables using one ...
2
votes
2answers
146 views

How to plot number of Trees and OOBs score with Grid Search

I searched to find the answer but I don´t find something with Grid Search. I create a random forest and gradient boosting regressor with grid search. Now I want to make a visualization to see if the ...
2
votes
0answers
247 views

SHAP Explanations in case of repeated train/test split

I am building a XGBoost model with Python and trying to explain it using the beautiful shap package. Apart from calculating SHAP values of each feature, I'd like to show graphs such as the two that ...
1
vote
2answers
380 views

Explainable anomaly detection

There are plenty of working for explaining prediction in supervised learning (e.g. SHAP values, LIME). What about for anomaly detection in unsupervised learning? Is there any model for which there ...
2
votes
1answer
83 views

What is the meaning of an empty SHAP graph in Explainable AI?

Using Python, I created a neural network to perform predictions on a binary class dataset (e.g. will a passenger survive the Titanic?). I am using the SHAP package to explain individual predictions. ...
1
vote
1answer
59 views

How to restructure my dataset for interpretability without losing performance?

What I am doing: I am predicting product ratings using boosted trees (XGBoost) with a dataset in this format: What I want to do: I want to use SHAP TreeExplainer to interpret each prediction my model ...
3
votes
1answer
863 views

Explanation of how DeepExplainer works to obtain SHAP values in simple terms

I have been using DeepExplainer (DE) to obtain the approximate SHAP values for my MLP model. I am following the SHAP Python library. Now I'd like learn the logic behind DE more. From the relevant ...
3
votes
1answer
231 views

Is it valid to compare SHAP values across models?

Let's say I have three models: a random forest with 100 trees a random forest with 1000 trees an xgboost model. I can rank the importance of my features on my dataset for each model using SHAP, and ...
2
votes
2answers
87 views

Explainability ML Methods

I am currently working on a Machine-Learning Model. In order to explain how it works, I have looked at Partial Dependence Plots, Feature Importance and all kinds of methods, but one thing still ...
7
votes
1answer
142 views

Evaluating machine learning explainers?

I'm working on a project where multiple machine learning explainers (LIME and SHAP, potentially more coming) are applied to pre-trained models (neural networks) to help explain the predictions of ...
2
votes
1answer
253 views

How to handle the parameter space of neural networks?

This question is very broad (and might even be closed as "too broad"). It can be considered as a beginners question, because it is largely about getting started in terms of heading into a direction ...