Questions tagged [explainable-ai]
Use for questions about explainable artificial intelligence (AI), which aims at understanding, interpreting, and explaining the decisions that have been made by complex AI systems
72 questions
0
votes
0
answers
22
views
Can I compare mean absolute shap values for different models?
I am comparing 3 different classifiers ANN, XG Boost and Random Forest in making predictions. I also used SHAP for feature importance. I am only interested in the top 10 features based on SHA. The 3 ...
0
votes
0
answers
23
views
Explainable multi instance regression?
I have data where each top-level instance has a few independent variables and a numeric target variable. In addition, there are 1 to k multi-feature vectors associated with each top-level instance. ...
- 101
0
votes
0
answers
68
views
Can I still use LSTM on my small datasets?
As for the datasets, I manually split it into three sets, consists of a Train Set (420 samples), Validation Set (140 samples), and Test Set (140 samples). When I run the datasets on the LSTM, the test ...
-1
votes
1
answer
127
views
2
votes
2
answers
60
views
How best to explain Forecasts when model cannot be accessed?
My company purchases demand forecasts from an external vendor (after providing them with our historical data). My manager wants to explain the forecasts that we are receiving and has requested for ...
- 53
4
votes
1
answer
334
views
Why do Shapley value solutions remain consistent when the value function of the empty set changes in the ML context?
Hey there data science stack exchange - question about SHAP.
In the original Shapley value formulation from Lloyd, one assumption is that the value function of the empty set equals zero, $v(\emptyset) ...
- 143
0
votes
0
answers
39
views
Which method is better to understand key drivers/feature importance in prediction?
After applying two different classifiers (EBM Classifier and Random Forest Classifier) and getting similar scores, I used InterpretML functionality to identify the most relevant features in each model....
0
votes
1
answer
92
views
SHAP values are explaining the wrong output value
I was checking the local accuracy property of the SHAP values. It states that for a data point $(X,y)$, the SHAP values $(s_1,s_2,s_3,...)$ of features $(x_1,x_2,x_3,...)$ sum up to the difference of ...
- 11
0
votes
0
answers
26
views
İnternal and hold out test for shap
There is an internal test set and a hold out test set. I explain the model with Shap library. Should I use the internal test set or the external test set? What should I do if there is a difference?
- 1
0
votes
0
answers
28
views
how to get similar traning data for a test sample?
I want to get the most similar samples for test sample on which the model choose it's - output.
SHAP isn't useful because it show the contribution of each feature.
I want to get the most similar ...
- 527
0
votes
1
answer
240
views
Determining SHAP values for combined features
Say I have a dataset with three features: X1, X2 and X1 + X2. How would I determine the SHAP values for just singular features?
I'll denote I denote the SHAP value of say X1 as SHAP(X1). Would it be ...
0
votes
0
answers
118
views
Shapley value calculation in the case of absent features
The basic assumption in Shapely value calculation (which is also known as situational importance) is the following:
$$
v_i(x) = f(x_1, ..., x_n) - E[f(x_1, ...,X_i,..., x_n)]
$$
that is, the ...
- 123
0
votes
0
answers
52
views
Why shouldn't the attention matrices $W^Q$, $W^K$, $W^V$ be the same?
My question is why the equally shaped attention head matrices $W^Q$, $W^K$, $W^V$ should not be the same $W = W^Q =W^K= W^V$. In my understanding of transformer-based language models one attention ...
1
vote
0
answers
307
views
How do I interpret GRAD-CAM's feature attribution to time series zero-padding in a CNN classifier?
Problem setting: MTS Classification with CNN architecture
I have a multivariate time series (MTS) dataset that contains 30 features. The goal is to solve a classification problem on this MTS dataset. ...
0
votes
1
answer
39
views
Best practise XAI: understand features which build up cluster and explain underlaying structure
I want to cluster my data and show which features were used to define the clusters to show the structure in my data.
To explain the use case: Imaging I have data from many products and I want show the ...
- 113
1
vote
1
answer
33
views
Is there such thing as dataset imrovement?
I know that we can use explained machine learning to find why a model chose a certain classification.
I wonder if there is a way I can find which features are going to improve my current model.
I will ...
- 549
0
votes
2
answers
307
views
Which intrinsically explainable model has the highest performance?
Explainable AI can be achieved through intrinsically explainable models, like logistic and linear regression, or post-hoc explanations, like SHAP.
I want to use an intrinsically explainable model on ...
- 671
1
vote
1
answer
231
views
Can we add baseline to SHAP?
I have a doubt. I am currently using an integrated gradient for the DNN model for explainability. In that, we can specify the baseline as a parameter to the function. I am using all zeros for this. I ...
- 193
2
votes
2
answers
894
views
What is the difference (interpretation) between the partial R^2 and the SHAP value for a linear regression model?
To calculate the coefficient of partial determination R2 for a given variable:
We calculate the R2 with and without that variable and substract them. This implies fitting a different model with and ...
- 185
1
vote
1
answer
508
views
Captum vs GNNExplainer for explainability in Graph Neural Networks
I'm new to Graph Neural Networks and interested in exploring frameworks that allow the identification of nodes/edges that underlie prediction.
I came across :
(1) a model architecture (GNNExplainer) ...
- 111
4
votes
1
answer
534
views
Is multicollinarity a problem when interpreting SHAP values from an XGBoost model?
I'm using an XGBoost model for multi-class classification and is looking at feature importance by using SHAP values. I'm curious if multicollinarity is a problem for the interpretation of the SHAP ...
- 41
1
vote
0
answers
13
views
Why calculating how much removed sentences with most contributing words to the result helps to show that a model is "*faithful*"?
I don't understand how the calculation score taking out the sentences where the words contribute the most of to the result helps to show to what extent a model is "faithful" to a reasoning ...
1
vote
0
answers
55
views
How to interpret integrated gradients in an NLP toxic text classification use-case?
I am trying to understand how integrated gradients work in the NLP case.
Let $F: \mathbb{R}^{n} \rightarrow[0,1]$ a function representing a neural network, $x \in \mathbb{R}^{n}$ an input and $x' \in ...
0
votes
1
answer
234
views
What is the SHAP values for a liner model? How do we derive that?
What is the SHAP values for a linear model?
it is given as below in the documentation
Assuming features are independent leads to interventional SHAP values which for a linear model are coef[i] * (x[i]...
- 263
2
votes
1
answer
2k
views
Is there a way to output feature importance based on the outputted class?
I'm running a random forest classifier in Python (two classes). I am using the feature_importances_ method of the ...
- 83
2
votes
4
answers
5k
views
Which AI algorithm is best for chess?
I'm working on my chess bot, and I would like to implement simple artificial intelligence for it. I'm new in it, so I'm unsure how to do it specifically on chess. I heard about Q-learning, Supervised/...
- 129
4
votes
2
answers
163
views
Points to remember when embarking on an organization-wide turn to AI solutions
In our organization, we are currently in the phase of building up team, skills to automate and implement AI based solutions. So, we are very early in this AI journey.
Right now, we are also working on ...
- 2,655
2
votes
2
answers
144
views
How to stop a text-classification model from depending on only couple of the words from input text instead of entire sentence?
I have a text classification deep-learning model, which takes in a text and outputs a softmax probability. I am using glove embeddings to represent my input text in numerical form for the DL model.
...
1
vote
0
answers
45
views
Transparent Matching / Recommendation System [closed]
I am thinking of a matching/recommendation algorithm which matches students to the right teachers for their individual problems.
The dataset would look like this:
Student
Name
Age
Gender
Weak ...
- 31
1
vote
1
answer
1k
views
Understanding hierarchical clustering features importance
I made a hierarchical clustering with scikit :
selected_model = AgglomerativeClustering(n_clusters=8)
hierarchical_clustering8 = selected_model.fit_predict(answers)
...
- 121
0
votes
0
answers
104
views
meaning of shapley values using SHAP
I have a binary classification - "BAD" and "GOOD" samples.
The features are binary as well, either 0 or 1 (each sample is a boolean vector of size 264. I got about 3000 "BAD&...
- 151
0
votes
1
answer
944
views
Interpretable vs. transparent ML algorithms
What is the difference between interpretable algorithm and transparent algorithm? In particular, is there an algorithm that is interpretable but not transparent?
Update: In this post we defined ...
- 3
0
votes
1
answer
702
views
An example of explainable, but not interpretable ML model
This post attempts to explain the difference between explainability and interpretability of ML models. However, the explanation is somewhat unclear. Can somebody provide specific examples of models ...
- 3
1
vote
1
answer
1k
views
Explainability and Autoencoders
suppose I have an autoencoder as a two-stack LSTM that takes in sequences of $n$ features of some length $m$.
Let's say that the dimension of my encoding vector is $k$, so the architecture is of the ...
- 338
1
vote
0
answers
105
views
Constrastive vs Counterfactual Explanations
Is there any canonical definition for Contrastive vs Counterfactual explanations?
In the literature, I keep reading different versions but I wonder if there are good definitions or illustrative ...
- 6,370
2
votes
0
answers
820
views
How SHAP value explains contribution of features for outliers event?
I'm trying to understand and experiment with how the SHAP value can explain behaviour for each outlier events (rows) and how it can be related to shap.force_plot(). ...
- 432
3
votes
1
answer
312
views
How to evaluate the "importance" of a variable in a function
Let's say that we have
$$f(x,y,z) = x/k - (y/k) ((z - x/k)/(z - y/k))$$
$$k = constant \in ]0,1[$$
And I need to show in some way that the variable $x$ is more important in some metric that I don't ...
- 51
2
votes
1
answer
102
views
Shapley contribution when coalition is 0
I am exploring Shapley for channel attribution based on [here][1]
Consider C1, C2, C3, C4 as 4 channels in question.
Some of the coalition does not have value, such as
...
- 121
5
votes
2
answers
674
views
Different feature importance results between DNN, Random Forests and Gradient Boosted Decision Trees
I've been modeling metabolite data with 3 different regressor models. I get similar results from running feature importance with Random Forest model and Gradient Boosted Decision Trees (where I used ...
- 51
6
votes
1
answer
501
views
Shapley values without intercept (or without `expected_value`)
I have a model and I want to derive its interpretability by using feature contributions. In the end, I want to have some contribution per feature such that the sum of contributions equals the ...
- 6,106
1
vote
0
answers
56
views
Passing reduced/different feature data to LimeTabularExplainer compared to the original model
I am trying to use LimeTabularExplainer class and explain_instance function to find explainations of my LightGbm (lgb) model. However, the lgb model uses complex feature set which are not ...
- 11
0
votes
1
answer
40
views
Types of maps in Interpretable Machine Learning
I have worked on Interpretable Machine Learning (IML) for over 1 year. However, there are some terminologies that always make me confused.
For example, saliency maps/heat maps. Are they same?
Are ...
- 235
1
vote
0
answers
34
views
What is the meaning of Information with respect to interpretability approaches in machine learning?
I was going through a pre-print on arXiv named "Quantifying interpretability and trust in machine learning systems". There, I found that a comparison of two interpretability approaches - ...
- 11
1
vote
1
answer
32
views
Explainable AI, how did the computer classified Text?
My question is not about explaining the model or the algorithm
like which neurons were triggered and what are the parameters of perceptrons.
I will explain further
The problem
I have medical reports I ...
- 549
1
vote
2
answers
2k
views
Can Shapley/Lime values be used for unsupervised learning?
One thing that is really useful when trying to understand what a machine learning model does, is seeing why some instances got predicted. For that Shapley Values and Lime are really usefull. But can ...
- 6,370
1
vote
0
answers
442
views
Partial Dependence Plot and categorical variables
While reading about machine learning explainability and Partial Dependence Plot (PDP) in this book, the following appeared when dealing with categorical variables:
For each of the categories, we get ...
- 6,370
1
vote
1
answer
64
views
Uncertainty in connection to explainability
When I write "uncertainty" in this post I mean:
If I have a classifier into $a_1,..,a_n$ categories and for an observation $x$ I classify $x$ to $a_i$ with probability $p_i$, then the ...
- 338
3
votes
2
answers
523
views
What is a "surrogate model"?
While reading about model explainability and model accountability, the term surrogate model keeps appearing. I had an idea about what it is but it does not seem to make sense anymore:
What is a ...
- 6,370
2
votes
0
answers
508
views
Aggregate SHAP importances from different models
A couple of questions on the SHAP approach to the estimation of feature importance.
I would like to use the random forest, logistic regression, SVM, and kNN to train four classification models on a ...
- 21
6
votes
1
answer
10k
views
How is the "base value" of SHAP values calculated?
I'm trying to understand how the base value is calculated. So I used an example from SHAP's github notebook, Census income classification with LightGBM.
Right after I trained the lightgbm model, I ...
- 207