Questions tagged [explainable-ai]

Use for questions about explainable artificial intelligence (AI), which aims at understanding, interpreting, and explaining the decisions that have been made by complex AI systems

Filter by
Sorted by
Tagged with
0 votes
1 answer
16 views

SHAP values are explaining the wrong output value

I was checking the local accuracy property of the SHAP values. It states that for a data point $(X,y)$, the SHAP values $(s_1,s_2,s_3,...)$ of features $(x_1,x_2,x_3,...)$ sum up to the difference of ...
Abhay Gupta's user avatar
0 votes
0 answers
9 views

İnternal and hold out test for shap

There is an internal test set and a hold out test set. I explain the model with Shap library. Should I use the internal test set or the external test set? What should I do if there is a difference?
Nemo's user avatar
  • 1
0 votes
0 answers
11 views

how to get similar traning data for a test sample?

I want to get the most similar samples for test sample on which the model choose it's - output. SHAP isn't useful because it show the contribution of each feature. I want to get the most similar ...
user3668129's user avatar
0 votes
1 answer
95 views

Determining SHAP values for combined features

Say I have a dataset with three features: X1, X2 and X1 + X2. How would I determine the SHAP values for just singular features? I'll denote I denote the SHAP value of say X1 as SHAP(X1). Would it be ...
user156361's user avatar
0 votes
0 answers
25 views

Alternative to partial dependence plots

In my understanding, to make the partial dependence plot, you select a feature of interest x and leave other features X unchanged, then for each example in the validation dataset you sample values in ...
mindoverflow's user avatar
0 votes
0 answers
64 views

Shapley value calculation in the case of absent features

The basic assumption in Shapely value calculation (which is also known as situational importance) is the following: $$ v_i(x) = f(x_1, ..., x_n) - E[f(x_1, ...,X_i,..., x_n)] $$ that is, the ...
Sanyo Mn's user avatar
  • 123
0 votes
0 answers
14 views

XAI model to evaluate MLP

I need to build a XAI model and I don't know where to begin. I have seen different algorithms but can't think of how to create a model with them.Could anyone help me?
alex martinez's user avatar
0 votes
0 answers
16 views

eXplainable Artificial Intelligence (XAI). Need help building a XAI model to explain the results of an IDS classifier

I need some help building a XAI model with Keras to explain the results of an MLP working as an IDS. I have resarched about XAI but the only thing I find is small portions of code that just use ...
alex martinez's user avatar
0 votes
0 answers
25 views

TreeSHAP algorithm: how do UNWIND and EXTEND work?

In the paper Consistent Individualized Feature Attribution for Tree Ensembles, the authors describe the well-known TreeSHAP algorithm. It aims to calculate the SHAP values of feature $i$ for all $i$ ...
Noomkwah's user avatar
  • 101
0 votes
0 answers
35 views

Why shouldn't the attention matrices $W^Q$, $W^K$, $W^V$ be the same?

My question is why the equally shaped attention head matrices $W^Q$, $W^K$, $W^V$ should not be the same $W = W^Q =W^K= W^V$. In my understanding of transformer-based language models one attention ...
Hans-Peter Stricker's user avatar
0 votes
1 answer
210 views

How do I interpret GRAD-CAM's feature attribution to time series zero-padding in a CNN classifier?

Problem setting: MTS Classification with CNN architecture I have a multivariate time series (MTS) dataset that contains 30 features. The goal is to solve a classification problem on this MTS dataset. ...
Victor Neverland's user avatar
0 votes
1 answer
28 views

Best practise XAI: understand features which build up cluster and explain underlaying structure

I want to cluster my data and show which features were used to define the clusters to show the structure in my data. To explain the use case: Imaging I have data from many products and I want show the ...
soph's user avatar
  • 113
1 vote
1 answer
33 views

Is there such thing as dataset imrovement?

I know that we can use explained machine learning to find why a model chose a certain classification. I wonder if there is a way I can find which features are going to improve my current model. I will ...
asmgx's user avatar
  • 549
0 votes
2 answers
119 views

Which intrinsically explainable model has the highest performance?

Explainable AI can be achieved through intrinsically explainable models, like logistic and linear regression, or post-hoc explanations, like SHAP. I want to use an intrinsically explainable model on ...
Connor's user avatar
  • 631
1 vote
1 answer
167 views

Can we add baseline to SHAP?

I have a doubt. I am currently using an integrated gradient for the DNN model for explainability. In that, we can specify the baseline as a parameter to the function. I am using all zeros for this. I ...
Pritam Sinha's user avatar
2 votes
2 answers
581 views

What is the difference (interpretation) between the partial R^2 and the SHAP value for a linear regression model?

To calculate the coefficient of partial determination R2 for a given variable: We calculate the R2 with and without that variable and substract them. This implies fitting a different model with and ...
skan's user avatar
  • 185
1 vote
1 answer
363 views

Captum vs GNNExplainer for explainability in Graph Neural Networks

I'm new to Graph Neural Networks and interested in exploring frameworks that allow the identification of nodes/edges that underlie prediction. I came across : (1) a model architecture (GNNExplainer) ...
batlike's user avatar
  • 111
4 votes
1 answer
465 views

Is multicollinarity a problem when interpreting SHAP values from an XGBoost model?

I'm using an XGBoost model for multi-class classification and is looking at feature importance by using SHAP values. I'm curious if multicollinarity is a problem for the interpretation of the SHAP ...
hideonbush's user avatar
1 vote
0 answers
11 views

Why calculating how much removed sentences with most contributing words to the result helps to show that a model is "*faithful*"?

I don't understand how the calculation score taking out the sentences where the words contribute the most of to the result helps to show to what extent a model is "faithful" to a reasoning ...
Revolucion for Monica's user avatar
1 vote
0 answers
38 views

How to interpret integrated gradients in an NLP toxic text classification use-case?

I am trying to understand how integrated gradients work in the NLP case. Let $F: \mathbb{R}^{n} \rightarrow[0,1]$ a function representing a neural network, $x \in \mathbb{R}^{n}$ an input and $x' \in ...
Revolucion for Monica's user avatar
0 votes
1 answer
208 views

What is the SHAP values for a liner model? How do we derive that?

What is the SHAP values for a linear model? it is given as below in the documentation Assuming features are independent leads to interventional SHAP values which for a linear model are coef[i] * (x[i]...
NAS_2339's user avatar
  • 243
2 votes
1 answer
1k views

Is there a way to output feature importance based on the outputted class?

I'm running a random forest classifier in Python (two classes). I am using the feature_importances_ method of the ...
Erik M's user avatar
  • 83
2 votes
4 answers
4k views

Which AI algorithm is best for chess?

I'm working on my chess bot, and I would like to implement simple artificial intelligence for it. I'm new in it, so I'm unsure how to do it specifically on chess. I heard about Q-learning, Supervised/...
Jenia's user avatar
  • 129
2 votes
2 answers
109 views

How to stop a text-classification model from depending on only couple of the words from input text instead of entire sentence?

I have a text classification deep-learning model, which takes in a text and outputs a softmax probability. I am using glove embeddings to represent my input text in numerical form for the DL model. ...
Naveen Reddy Marthala's user avatar
1 vote
0 answers
39 views

Transparent Matching / Recommendation System [closed]

I am thinking of a matching/recommendation algorithm which matches students to the right teachers for their individual problems. The dataset would look like this: Student Name Age Gender Weak ...
alexryder's user avatar
1 vote
1 answer
1k views

Understanding hierarchical clustering features importance

I made a hierarchical clustering with scikit : selected_model = AgglomerativeClustering(n_clusters=8) hierarchical_clustering8 = selected_model.fit_predict(answers) ...
Alex Dana's user avatar
  • 121
0 votes
0 answers
96 views

meaning of shapley values using SHAP

I have a binary classification - "BAD" and "GOOD" samples. The features are binary as well, either 0 or 1 (each sample is a boolean vector of size 264. I got about 3000 "BAD&...
Guy Barash's user avatar
0 votes
1 answer
854 views

Interpretable vs. transparent ML algorithms

What is the difference between interpretable algorithm and transparent algorithm? In particular, is there an algorithm that is interpretable but not transparent? Update: In this post we defined ...
Qwerty's user avatar
  • 3
0 votes
1 answer
565 views

An example of explainable, but not interpretable ML model

This post attempts to explain the difference between explainability and interpretability of ML models. However, the explanation is somewhat unclear. Can somebody provide specific examples of models ...
Qwerty's user avatar
  • 3
1 vote
1 answer
909 views

Explainability and Autoencoders

suppose I have an autoencoder as a two-stack LSTM that takes in sequences of $n$ features of some length $m$. Let's say that the dimension of my encoding vector is $k$, so the architecture is of the ...
Mariah's user avatar
  • 338
1 vote
0 answers
92 views

Constrastive vs Counterfactual Explanations

Is there any canonical definition for Contrastive vs Counterfactual explanations? In the literature, I keep reading different versions but I wonder if there are good definitions or illustrative ...
Carlos Mougan's user avatar
2 votes
0 answers
775 views

How SHAP value explains contribution of features for outliers event?

I'm trying to understand and experiment with how the SHAP value can explain behaviour for each outlier events (rows) and how it can be related to shap.force_plot(). ...
Mario's user avatar
  • 400
3 votes
1 answer
251 views

How to evaluate the "importance" of a variable in a function

Let's say that we have $$f(x,y,z) = x/k - (y/k) ((z - x/k)/(z - y/k))$$ $$k = constant \in ]0,1[$$ And I need to show in some way that the variable $x$ is more important in some metric that I don't ...
Allan Araujo's user avatar
2 votes
1 answer
90 views

Shapley contribution when coalition is 0

I am exploring Shapley for channel attribution based on [here][1] Consider C1, C2, C3, C4 as 4 channels in question. Some of the coalition does not have value, such as ...
Kenny's user avatar
  • 121
5 votes
2 answers
405 views

Different feature importance results between DNN, Random Forests and Gradient Boosted Decision Trees

I've been modeling metabolite data with 3 different regressor models. I get similar results from running feature importance with Random Forest model and Gradient Boosted Decision Trees (where I used ...
Orion's user avatar
  • 51
5 votes
1 answer
414 views

Shapley values without intercept (or without `expected_value`)

I have a model and I want to derive its interpretability by using feature contributions. In the end, I want to have some contribution per feature such that the sum of contributions equals the ...
David Masip's user avatar
  • 6,061
1 vote
0 answers
52 views

Passing reduced/different feature data to LimeTabularExplainer compared to the original model

I am trying to use LimeTabularExplainer class and explain_instance function to find explainations of my LightGbm (lgb) model. However, the lgb model uses complex feature set which are not ...
the M's user avatar
  • 11
0 votes
1 answer
39 views

Types of maps in Interpretable Machine Learning

I have worked on Interpretable Machine Learning (IML) for over 1 year. However, there are some terminologies that always make me confused. For example, saliency maps/heat maps. Are they same? Are ...
Giang Nguyen's user avatar
1 vote
0 answers
34 views

What is the meaning of Information with respect to interpretability approaches in machine learning?

I was going through a pre-print on arXiv named "Quantifying interpretability and trust in machine learning systems". There, I found that a comparison of two interpretability approaches - ...
Aabhas Vij's user avatar
1 vote
1 answer
31 views

Explainable AI, how did the computer classified Text?

My question is not about explaining the model or the algorithm like which neurons were triggered and what are the parameters of perceptrons. I will explain further The problem I have medical reports I ...
asmgx's user avatar
  • 549
1 vote
1 answer
2k views

Can Shapley/Lime values be used for unsupervised learning?

One thing that is really useful when trying to understand what a machine learning model does, is seeing why some instances got predicted. For that Shapley Values and Lime are really usefull. But can ...
Carlos Mougan's user avatar
1 vote
0 answers
403 views

Partial Dependence Plot and categorical variables

While reading about machine learning explainability and Partial Dependence Plot (PDP) in this book, the following appeared when dealing with categorical variables: For each of the categories, we get ...
Carlos Mougan's user avatar
1 vote
1 answer
61 views

Uncertainty in connection to explainability

When I write "uncertainty" in this post I mean: If I have a classifier into $a_1,..,a_n$ categories and for an observation $x$ I classify $x$ to $a_i$ with probability $p_i$, then the ...
Mariah's user avatar
  • 338
3 votes
2 answers
368 views

What is a "surrogate model"?

While reading about model explainability and model accountability, the term surrogate model keeps appearing. I had an idea about what it is but it does not seem to make sense anymore: What is a ...
Carlos Mougan's user avatar
0 votes
0 answers
49 views

Ways to visualize the outcome of machine learning interpretability techniques (for image classification)

“Machine Learning Interpretability” or “Explainable Artificial Intelligence” has become quite popular in the machine learning community and in recent research. The goal is to make complex (deep ...
member_10's user avatar
2 votes
0 answers
452 views

Aggregate SHAP importances from different models

A couple of questions on the SHAP approach to the estimation of feature importance. I would like to use the random forest, logistic regression, SVM, and kNN to train four classification models on a ...
CaffeineMan's user avatar
5 votes
1 answer
9k views

How is the "base value" of SHAP values calculated?

I'm trying to understand how the base value is calculated. So I used an example from SHAP's github notebook, Census income classification with LightGBM. Right after I trained the lightgbm model, I ...
David293836's user avatar
1 vote
0 answers
61 views

Can we use Shap to interpret output changes?

Can we calculate the difference between Shapley values to interpret changes in the output? More precisely, if we get Shapley values for two different inputs, can we compare them to understand how much ...
giogix's user avatar
  • 163
1 vote
2 answers
276 views

What does a conservative technique mean in the context of neural networks?

I am reading this article on Layer-wise relevance propagation method and I can't understand this particular paragraph LRP is a conservative technique, meaning the magnitude of any output y is ...
Black Jack 21's user avatar
2 votes
1 answer
75 views

treeExplainer algorithm intuition

I'm reading the paper about the treeExplainer; the pseudo-code of Algorithm 1 is a bit cryptical as most of the variables are not even defined (same with sampling and all details involved). Is there a ...
zzzbob's user avatar
  • 45