Questions tagged [explainable-ai]
Use for questions about explainable artificial intelligence (AI), which aims at understanding, interpreting, and explaining the decisions that have been made by complex AI systems
67
questions
0
votes
1
answer
27
views
Determining SHAP values for combined features
Say I have a dataset with three features: X1, X2 and X1 + X2. How would I determine the SHAP values for just singular features?
I'll denote I denote the SHAP value of say X1 as SHAP(X1). Would it be ...
0
votes
0
answers
15
views
Alternative to partial dependence plots
In my understanding, to make the partial dependence plot, you select a feature of interest x and leave other features X unchanged, then for each example in the validation dataset you sample values in ...
0
votes
0
answers
41
views
Shapley value calculation in the case of absent features
The basic assumption in Shapely value calculation (which is also known as situational importance) is the following:
$$
v_i(x) = f(x_1, ..., x_n) - E[f(x_1, ...,X_i,..., x_n)]
$$
that is, the ...
- 123
0
votes
0
answers
11
views
XAI model to evaluate MLP
I need to build a XAI model and I don't know where to begin. I have seen different algorithms but can't think of how to create a model with them.Could anyone help me?
0
votes
0
answers
13
views
eXplainable Artificial Intelligence (XAI). Need help building a XAI model to explain the results of an IDS classifier
I need some help building a XAI model with Keras to explain the results of an MLP working as an IDS. I have resarched about XAI but the only thing I find is small portions of code that just use ...
0
votes
0
answers
18
views
TreeSHAP algorithm: how do UNWIND and EXTEND work?
In the paper Consistent Individualized Feature Attribution for Tree Ensembles, the authors describe the well-known TreeSHAP algorithm.
It aims to calculate the SHAP values of feature $i$ for all $i$ ...
- 101
0
votes
0
answers
27
views
Why shouldn't the attention matrices $W^Q$, $W^K$, $W^V$ be the same?
My question is why the equally shaped attention head matrices $W^Q$, $W^K$, $W^V$ should not be the same $W = W^Q =W^K= W^V$. In my understanding of transformer-based language models one attention ...
0
votes
1
answer
127
views
How do I interpret GRAD-CAM's feature attribution to time series zero-padding in a CNN classifier?
Problem setting: MTS Classification with CNN architecture
I have a multivariate time series (MTS) dataset that contains 30 features. The goal is to solve a classification problem on this MTS dataset. ...
0
votes
1
answer
27
views
Best practise XAI: understand features which build up cluster and explain underlaying structure
I want to cluster my data and show which features were used to define the clusters to show the structure in my data.
To explain the use case: Imaging I have data from many products and I want show the ...
- 113
1
vote
1
answer
29
views
Is there such thing as dataset imrovement?
I know that we can use explained machine learning to find why a model chose a certain classification.
I wonder if there is a way I can find which features are going to improve my current model.
I will ...
- 539
0
votes
2
answers
83
views
Which intrinsically explainable model has the highest performance?
Explainable AI can be achieved through intrinsically explainable models, like logistic and linear regression, or post-hoc explanations, like SHAP.
I want to use an intrinsically explainable model on ...
- 617
0
votes
0
answers
14
views
Using subgroup discovery for a very small subset of a huge dataset?
I want to use subgroup discovery to find some interesting patterns in a small subset of my dataset. The dataset contains about 100.000 records, the subset I want to look at only 100 records. Does it ...
- 1
1
vote
1
answer
133
views
Can we add baseline to SHAP?
I have a doubt. I am currently using an integrated gradient for the DNN model for explainability. In that, we can specify the baseline as a parameter to the function. I am using all zeros for this. I ...
- 193
2
votes
2
answers
411
views
What is the difference (interpretation) between the partial R^2 and the SHAP value for a linear regression model?
To calculate the coefficient of partial determination R2 for a given variable:
We calculate the R2 with and without that variable and substract them. This implies fitting a different model with and ...
- 185
1
vote
1
answer
252
views
Captum vs GNNExplainer for explainability in Graph Neural Networks
I'm new to Graph Neural Networks and interested in exploring frameworks that allow the identification of nodes/edges that underlie prediction.
I came across :
(1) a model architecture (GNNExplainer) ...
- 111
4
votes
1
answer
420
views
Is multicollinarity a problem when interpreting SHAP values from an XGBoost model?
I'm using an XGBoost model for multi-class classification and is looking at feature importance by using SHAP values. I'm curious if multicollinarity is a problem for the interpretation of the SHAP ...
- 41
1
vote
0
answers
11
views
Why calculating how much removed sentences with most contributing words to the result helps to show that a model is "*faithful*"?
I don't understand how the calculation score taking out the sentences where the words contribute the most of to the result helps to show to what extent a model is "faithful" to a reasoning ...
1
vote
0
answers
33
views
How to interpret integrated gradients in an NLP toxic text classification use-case?
I am trying to understand how integrated gradients work in the NLP case.
Let $F: \mathbb{R}^{n} \rightarrow[0,1]$ a function representing a neural network, $x \in \mathbb{R}^{n}$ an input and $x' \in ...
0
votes
1
answer
192
views
What is the SHAP values for a liner model? How do we derive that?
What is the SHAP values for a linear model?
it is given as below in the documentation
Assuming features are independent leads to interventional SHAP values which for a linear model are coef[i] * (x[i]...
- 233
2
votes
1
answer
914
views
Is there a way to output feature importance based on the outputted class?
I'm running a random forest classifier in Python (two classes). I am using the feature_importances_ method of the ...
- 83
2
votes
4
answers
4k
views
Which AI algorithm is best for chess?
I'm working on my chess bot, and I would like to implement simple artificial intelligence for it. I'm new in it, so I'm unsure how to do it specifically on chess. I heard about Q-learning, Supervised/...
- 129
2
votes
2
answers
90
views
How to stop a text-classification model from depending on only couple of the words from input text instead of entire sentence?
I have a text classification deep-learning model, which takes in a text and outputs a softmax probability. I am using glove embeddings to represent my input text in numerical form for the DL model.
...
1
vote
0
answers
36
views
Transparent Matching / Recommendation System [closed]
I am thinking of a matching/recommendation algorithm which matches students to the right teachers for their individual problems.
The dataset would look like this:
Student
Name
Age
Gender
Weak ...
- 31
1
vote
1
answer
898
views
Understanding hierarchical clustering features importance
I made a hierarchical clustering with scikit :
selected_model = AgglomerativeClustering(n_clusters=8)
hierarchical_clustering8 = selected_model.fit_predict(answers)
...
- 121
0
votes
0
answers
91
views
meaning of shapley values using SHAP
I have a binary classification - "BAD" and "GOOD" samples.
The features are binary as well, either 0 or 1 (each sample is a boolean vector of size 264. I got about 3000 "BAD&...
- 151
0
votes
1
answer
792
views
Interpretable vs. transparent ML algorithms
What is the difference between interpretable algorithm and transparent algorithm? In particular, is there an algorithm that is interpretable but not transparent?
Update: In this post we defined ...
- 3
0
votes
1
answer
450
views
An example of explainable, but not interpretable ML model
This post attempts to explain the difference between explainability and interpretability of ML models. However, the explanation is somewhat unclear. Can somebody provide specific examples of models ...
- 3
1
vote
1
answer
758
views
Explainability and Autoencoders
suppose I have an autoencoder as a two-stack LSTM that takes in sequences of $n$ features of some length $m$.
Let's say that the dimension of my encoding vector is $k$, so the architecture is of the ...
- 338
1
vote
0
answers
81
views
Constrastive vs Counterfactual Explanations
Is there any canonical definition for Contrastive vs Counterfactual explanations?
In the literature, I keep reading different versions but I wonder if there are good definitions or illustrative ...
- 6,092
2
votes
0
answers
734
views
How SHAP value explains contribution of features for outliers event?
I'm trying to understand and experiment with how the SHAP value can explain behaviour for each outlier events (rows) and how it can be related to shap.force_plot(). ...
- 364
3
votes
1
answer
201
views
How to evaluate the "importance" of a variable in a function
Let's say that we have
$$f(x,y,z) = x/k - (y/k) ((z - x/k)/(z - y/k))$$
$$k = constant \in ]0,1[$$
And I need to show in some way that the variable $x$ is more important in some metric that I don't ...
- 51
2
votes
1
answer
79
views
Shapley contribution when coalition is 0
I am exploring Shapley for channel attribution based on [here][1]
Consider C1, C2, C3, C4 as 4 channels in question.
Some of the coalition does not have value, such as
...
- 121
5
votes
2
answers
340
views
Different feature importance results between DNN, Random Forests and Gradient Boosted Decision Trees
I've been modeling metabolite data with 3 different regressor models. I get similar results from running feature importance with Random Forest model and Gradient Boosted Decision Trees (where I used ...
- 51
5
votes
1
answer
373
views
Shapley values without intercept (or without `expected_value`)
I have a model and I want to derive its interpretability by using feature contributions. In the end, I want to have some contribution per feature such that the sum of contributions equals the ...
- 6,031
1
vote
0
answers
51
views
Passing reduced/different feature data to LimeTabularExplainer compared to the original model
I am trying to use LimeTabularExplainer class and explain_instance function to find explainations of my LightGbm (lgb) model. However, the lgb model uses complex feature set which are not ...
- 11
0
votes
1
answer
38
views
Types of maps in Interpretable Machine Learning
I have worked on Interpretable Machine Learning (IML) for over 1 year. However, there are some terminologies that always make me confused.
For example, saliency maps/heat maps. Are they same?
Are ...
- 235
1
vote
0
answers
34
views
What is the meaning of Information with respect to interpretability approaches in machine learning?
I was going through a pre-print on arXiv named "Quantifying interpretability and trust in machine learning systems". There, I found that a comparison of two interpretability approaches - ...
- 11
1
vote
1
answer
31
views
Explainable AI, how did the computer classified Text?
My question is not about explaining the model or the algorithm
like which neurons were triggered and what are the parameters of perceptrons.
I will explain further
The problem
I have medical reports I ...
- 539
1
vote
1
answer
1k
views
Can Shapley/Lime values be used for unsupervised learning?
One thing that is really useful when trying to understand what a machine learning model does, is seeing why some instances got predicted. For that Shapley Values and Lime are really usefull. But can ...
- 6,092
1
vote
0
answers
383
views
Partial Dependence Plot and categorical variables
While reading about machine learning explainability and Partial Dependence Plot (PDP) in this book, the following appeared when dealing with categorical variables:
For each of the categories, we get ...
- 6,092
1
vote
1
answer
59
views
Uncertainty in connection to explainability
When I write "uncertainty" in this post I mean:
If I have a classifier into $a_1,..,a_n$ categories and for an observation $x$ I classify $x$ to $a_i$ with probability $p_i$, then the ...
- 338
3
votes
2
answers
257
views
What is a "surrogate model"?
While reading about model explainability and model accountability, the term surrogate model keeps appearing. I had an idea about what it is but it does not seem to make sense anymore:
What is a ...
- 6,092
0
votes
0
answers
44
views
Ways to visualize the outcome of machine learning interpretability techniques (for image classification)
“Machine Learning Interpretability” or “Explainable Artificial Intelligence” has become quite popular in the machine learning community and in recent research. The goal is to make complex (deep ...
2
votes
0
answers
405
views
Aggregate SHAP importances from different models
A couple of questions on the SHAP approach to the estimation of feature importance.
I would like to use the random forest, logistic regression, SVM, and kNN to train four classification models on a ...
- 21
5
votes
1
answer
8k
views
How is the "base value" of SHAP values calculated?
I'm trying to understand how the base value is calculated. So I used an example from SHAP's github notebook, Census income classification with LightGBM.
Right after I trained the lightgbm model, I ...
- 197
1
vote
0
answers
56
views
Can we use Shap to interpret output changes?
Can we calculate the difference between Shapley values to interpret changes in the output? More precisely, if we get Shapley values for two different inputs, can we compare them to understand how much ...
- 163
1
vote
2
answers
260
views
What does a conservative technique mean in the context of neural networks?
I am reading this article on Layer-wise relevance propagation method and I can't understand this particular paragraph
LRP is a conservative technique, meaning the magnitude of any output y is ...
- 173
2
votes
1
answer
72
views
treeExplainer algorithm intuition
I'm reading the paper about the treeExplainer; the pseudo-code of Algorithm 1 is a bit cryptical as most of the variables are not even defined (same with sampling and all details involved).
Is there a ...
- 45
20
votes
5
answers
5k
views
What is the difference between explainable and interpretable machine learning?
O’Rourke says that explainable ML uses a black box model and explains it afterwards, whereas interpretable ML uses models that are no black boxes.
Christoph Molnar says interpretable ML refers to the ...
- 585
1
vote
0
answers
416
views
Tree Path Dependent expected value
I stuck on this topic for couple days and seems I need your help to understand what is expected value in TreeExplainer when feature_perturbation = tree_path_dependent, precisely what is expected value ...
- 11