Questions tagged [explainable-ai]

Use for questions about explainable artificial intelligence (AI), which aims at understanding, interpreting, and explaining the decisions that have been made by complex AI systems

Filter by
Sorted by
Tagged with
2
votes
4answers
1k views

Which AI algorithm is best for chess?

I'm working on my chess bot, and I would like to implement simple artificial intelligence for it. I'm new in it, so I'm unsure how to do it specifically on chess. I heard about Q-learning, Supervised/...
0
votes
0answers
50 views

What is the difference between shap kernel-explainer and deep-explainer

I want to use shap to explain my image classification model. I read that it is better to use shap.DeepExplainer (than ...
2
votes
2answers
39 views

How to stop a text-classification model from depending on only couple of the words from input text instead of entire sentence?

I have a text classification deep-learning model, which takes in a text and outputs a softmax probability. I am using glove embeddings to represent my input text in numerical form for the DL model. ...
1
vote
0answers
22 views

Transparent Matching / Recommendation System [closed]

I am thinking of a matching/recommendation algorithm which matches students to the right teachers for their individual problems. The dataset would look like this: Student Name Age Gender Weak ...
1
vote
1answer
63 views

Understanding hierarchical clustering features importance

I made a hierarchical clustering with scikit : selected_model = AgglomerativeClustering(n_clusters=8) hierarchical_clustering8 = selected_model.fit_predict(answers) ...
0
votes
0answers
28 views

meaning of shapley values using SHAP

I have a binary classification - "BAD" and "GOOD" samples. The features are binary as well, either 0 or 1 (each sample is a boolean vector of size 264. I got about 3000 "BAD&...
0
votes
0answers
8 views

How to do Interpretable AI with multiple image input fields and a single real-valued output image?

I have input data of the shape: (num_ex, num_features, 64, 64) and real-valued output data of shape: (num_ex, 1, 64, 64) I ...
0
votes
0answers
18 views

How can I track the percentage impact of variables in a ML model?

Using a Random Forest Model (for example), I can find out the list of variables that are most important. However, I would like to see the relative impact of the explanatory variables on the target ...
0
votes
1answer
108 views

Interpretable vs. transparent ML algorithms

What is the difference between interpretable algorithm and transparent algorithm? In particular, is there an algorithm that is interpretable but not transparent? Update: In this post we defined ...
0
votes
1answer
56 views

An example of explainable, but not interpretable ML model

This post attempts to explain the difference between explainability and interpretability of ML models. However, the explanation is somewhat unclear. Can somebody provide specific examples of models ...
0
votes
0answers
11 views

How to explain rescaled recommendation scores to end-user?

I constructed a recommendation system based on both Boolean and linear features that present products to potential buyers. The raw scores are not easily digestible for humans (i.e. the highest score ...
1
vote
1answer
70 views

Explainability and Autoencoders

suppose I have an autoencoder as a two-stack LSTM that takes in sequences of $n$ features of some length $m$. Let's say that the dimension of my encoding vector is $k$, so the architecture is of the ...
1
vote
0answers
28 views

Constrastive vs Counterfactual Explanations

Is there any canonical definition for Contrastive vs Counterfactual explanations? In the literature, I keep reading different versions but I wonder if there are good definitions or illustrative ...
0
votes
0answers
19 views

Are SHAP values additive across examples?

I understand that SHAP values have a property called additivity that means that if you add the SHAP value of each explanatory variable of a particular example to the average prediction of the model on ...
0
votes
0answers
22 views

Explanation on how to interpret force_plot result by using SHAP

I have applied SHAP for explaining the outcome of my neural network. For the force_plot I have obtained the following output when trying to look at multiple ...
0
votes
0answers
18 views

DeepTaylor and LRP

I'm studying explainable AI. Is it possible to apply DeepTaylor or Layer-wise Relevance Propagation that was made for NNs with ...
0
votes
0answers
11 views

Layer-wise Relevance Propagation with the swish activations

I'm studying approaches to explain DNN. I've found a lot of papers on LRP (and Deep Taylor decomposition), but they all explain NNs with the ReLU activations. I'm wondering why nobody applied LRP to ...
2
votes
0answers
254 views

How SHAP value explains contribution of features for outliers event?

I'm trying to understand and experiment with how the SHAP value can explain behaviour for each outlier events (rows) and how it can be related to shap.force_plot(). ...
0
votes
0answers
313 views

Grad_CAM for time series

I am new to deep learning and trying to build a Grad-cam from time series data. Shape of my input sample is (188,1), its an ECG signal and I have a cnn-1D model for classification. Keras provides ...
2
votes
1answer
65 views

How to evaluate the "importance" of a variable in a function

Let's say that we have $$f(x,y,z) = x/k - (y/k) ((z - x/k)/(z - y/k))$$ $$k = constant \in ]0,1[$$ And I need to show in some way that the variable $x$ is more important in some metric that I don't ...
0
votes
1answer
25 views

Shapley contribution when coalition is 0

I am exploring Shapley for channel attribution based on [here][1] Consider C1, C2, C3, C4 as 4 channels in question. Some of the coalition does not have value, such as ...
4
votes
2answers
100 views

Different feature importance results between DNN, Random Forests and Gradient Boosted Decision Trees

I've been modeling metabolite data with 3 different regressor models. I get similar results from running feature importance with Random Forest model and Gradient Boosted Decision Trees (where I used ...
5
votes
1answer
150 views

Shapley values without intercept (or without `expected_value`)

I have a model and I want to derive its interpretability by using feature contributions. In the end, I want to have some contribution per feature such that the sum of contributions equals the ...
1
vote
0answers
34 views

Passing reduced/different feature data to LimeTabularExplainer compared to the original model

I am trying to use LimeTabularExplainer class and explain_instance function to find explainations of my LightGbm (lgb) model. However, the lgb model uses complex feature set which are not ...
0
votes
1answer
24 views

Types of maps in Interpretable Machine Learning

I have worked on Interpretable Machine Learning (IML) for over 1 year. However, there are some terminologies that always make me confused. For example, saliency maps/heat maps. Are they same? Are ...
1
vote
0answers
32 views

What is the meaning of Information with respect to interpretability approaches in machine learning?

I was going through a pre-print on arXiv named "Quantifying interpretability and trust in machine learning systems". There, I found that a comparison of two interpretability approaches - ...
1
vote
1answer
23 views

Explainable AI, how did the computer classified Text?

My question is not about explaining the model or the algorithm like which neurons were triggered and what are the parameters of perceptrons. I will explain further The problem I have medical reports I ...
1
vote
1answer
498 views

Can Shapley/Lime values be used for unsupervised learning?

One thing that is really useful when trying to understand what a machine learning model does, is seeing why some instances got predicted. For that Shapley Values and Lime are really usefull. But can ...
1
vote
0answers
274 views

Partial Dependence Plot and categorical variables

While reading about machine learning explainability and Partial Dependence Plot (PDP) in this book, the following appeared when dealing with categorical variables: For each of the categories, we get ...
1
vote
1answer
43 views

Uncertainty in connection to explainability

When I write "uncertainty" in this post I mean: If I have a classifier into $a_1,..,a_n$ categories and for an observation $x$ I classify $x$ to $a_i$ with probability $p_i$, then the ...
3
votes
2answers
123 views

What is a "surrogate model"?

While reading about model explainability and model accountability, the term surrogate model keeps appearing. I had an idea about what it is but it does not seem to make sense anymore: What is a ...
0
votes
0answers
38 views

Ways to visualize the outcome of machine learning interpretability techniques (for image classification)

“Machine Learning Interpretability” or “Explainable Artificial Intelligence” has become quite popular in the machine learning community and in recent research. The goal is to make complex (deep ...
1
vote
0answers
119 views

Aggregate SHAP importances from different models

A couple of questions on the SHAP approach to the estimation of feature importance. I would like to use the random forest, logistic regression, SVM, and kNN to train four classification models on a ...
3
votes
1answer
3k views

How is the "base value" of SHAP values calculated?

I'm trying to understand how the base value is calculated. So I used an example from SHAP's github notebook, Census income classification with LightGBM. Right after I trained the lightgbm model, I ...
1
vote
0answers
37 views

Can we use Shap to interpret output changes?

Can we calculate the difference between Shapley values to interpret changes in the output? More precisely, if we get Shapley values for two different inputs, can we compare them to understand how much ...
1
vote
2answers
44 views

What does a conservative technique mean in the context of neural networks?

I am reading this article on Layer-wise relevance propagation method and I can't understand this particular paragraph LRP is a conservative technique, meaning the magnitude of any output y is ...
2
votes
1answer
62 views

treeExplainer algorithm intuition

I'm reading the paper about the treeExplainer; the pseudo-code of Algorithm 1 is a bit cryptical as most of the variables are not even defined (same with sampling and all details involved). Is there a ...
16
votes
5answers
3k views

What is the difference between explainable and interpretable machine learning?

O’Rourke says that explainable ML uses a black box model and explains it afterwards, whereas interpretable ML uses models that are no black boxes. Christoph Molnar says interpretable ML refers to the ...
1
vote
0answers
208 views

Tree Path Dependent expected value

I stuck on this topic for couple days and seems I need your help to understand what is expected value in TreeExplainer when feature_perturbation = tree_path_dependent, precisely what is expected value ...
2
votes
1answer
402 views

Interpretable xgboost - Calculate cover feature importance

When trying to interpret the results of a gradient boosting (or any decision tree) one can plot the feature importance. There are same parameters in the xgb api such as: weight, gain, cover, ...
1
vote
0answers
34 views

Does EDA helps only in case of linear regression?

I know what Explanatory data analysis is and how it helps us investigate and understand the data. What I dont understand is how does this help in case of nonlinear relationships? I mean if I'm using ...
1
vote
1answer
61 views

LIME Random Forest explanations:

I'm using LIME to explain my random forest model. Everything is working great. However, I don't quite understand the image that is generated. Taking the example from the Readme: How can it predict ...
2
votes
0answers
247 views

Multi-valued categorical features in LIME

I am working with the LIME implementation by Marco Ribeiro (https://github.com/marcotcr/lime). Specifically, I am utilizing the LimeTabularExplainer as I have a mixture of numerical and categorical ...
1
vote
1answer
757 views

Explainable AI and unsupervised algorithms

There are several packages that allow explaining ML algorithms (Lime, Shap and so on). However, it is not clear how we can explain unsupervised algorithms for example, if we use PCA for dimensionality ...
2
votes
2answers
440 views

Reasons that LIME and SHAP might not agree with intuition

I'm leveraging the Python packages lime and shap to explain single (test-set) predictions that a basic, trained model is making ...
9
votes
2answers
885 views

Why continuous features are more important than categorical features in decision tree models?

I have both categorical and continuous features in my prediction model and want to select (and rank) most important features. I have converted all categorical variables into dummy variables using one ...
2
votes
2answers
174 views

How to plot number of Trees and OOBs score with Grid Search

I searched to find the answer but I don´t find something with Grid Search. I create a random forest and gradient boosting regressor with grid search. Now I want to make a visualization to see if the ...
3
votes
2answers
465 views

SHAP Explanations in case of repeated train/test split

I am building a XGBoost model with Python and trying to explain it using the beautiful shap package. Apart from calculating SHAP values of each feature, I'd like to show graphs such as the two that ...
1
vote
2answers
491 views

Explainable anomaly detection

There are plenty of working for explaining prediction in supervised learning (e.g. SHAP values, LIME). What about for anomaly detection in unsupervised learning? Is there any model for which there ...
2
votes
1answer
154 views

What is the meaning of an empty SHAP graph in Explainable AI?

Using Python, I created a neural network to perform predictions on a binary class dataset (e.g. will a passenger survive the Titanic?). I am using the SHAP package to explain individual predictions. ...