Questions tagged [explainable-ai]
Use for questions about explainable artificial intelligence (AI), which aims at understanding, interpreting, and explaining the decisions that have been made by complex AI systems
72 questions
20
votes
5
answers
6k
views
What is the difference between explainable and interpretable machine learning?
O’Rourke says that explainable ML uses a black box model and explains it afterwards, whereas interpretable ML uses models that are no black boxes.
Christoph Molnar says interpretable ML refers to the ...
- 605
9
votes
2
answers
2k
views
Why continuous features are more important than categorical features in decision tree models?
I have both categorical and continuous features in my prediction model and want to select (and rank) most important features.
I have converted all categorical variables into dummy variables using one ...
- 103
7
votes
1
answer
226
views
Evaluating machine learning explainers?
I'm working on a project where multiple machine learning explainers (LIME and SHAP, potentially more coming) are applied to pre-trained models (neural networks) to help explain the predictions of ...
- 171
6
votes
1
answer
501
views
Shapley values without intercept (or without `expected_value`)
I have a model and I want to derive its interpretability by using feature contributions. In the end, I want to have some contribution per feature such that the sum of contributions equals the ...
- 6,106
6
votes
1
answer
10k
views
How is the "base value" of SHAP values calculated?
I'm trying to understand how the base value is calculated. So I used an example from SHAP's github notebook, Census income classification with LightGBM.
Right after I trained the lightgbm model, I ...
- 207
6
votes
1
answer
3k
views
Is it valid to compare SHAP values across models?
Let's say I have three models:
a random forest with 100 trees
a random forest with 1000 trees
an xgboost model.
I can rank the importance of my features on my dataset for each model using SHAP, and ...
- 88
5
votes
2
answers
674
views
Different feature importance results between DNN, Random Forests and Gradient Boosted Decision Trees
I've been modeling metabolite data with 3 different regressor models. I get similar results from running feature importance with Random Forest model and Gradient Boosted Decision Trees (where I used ...
- 51
5
votes
1
answer
2k
views
Explanation of how DeepExplainer works to obtain SHAP values in simple terms
I have been using DeepExplainer (DE) to obtain the approximate SHAP values for my MLP model. I am following the SHAP Python library.
Now I'd like learn the logic behind DE more. From the relevant ...
- 153
4
votes
2
answers
163
views
Points to remember when embarking on an organization-wide turn to AI solutions
In our organization, we are currently in the phase of building up team, skills to automate and implement AI based solutions. So, we are very early in this AI journey.
Right now, we are also working on ...
- 2,655
4
votes
1
answer
334
views
Why do Shapley value solutions remain consistent when the value function of the empty set changes in the ML context?
Hey there data science stack exchange - question about SHAP.
In the original Shapley value formulation from Lloyd, one assumption is that the value function of the empty set equals zero, $v(\emptyset) ...
- 143
4
votes
1
answer
534
views
Is multicollinarity a problem when interpreting SHAP values from an XGBoost model?
I'm using an XGBoost model for multi-class classification and is looking at feature importance by using SHAP values. I'm curious if multicollinarity is a problem for the interpretation of the SHAP ...
- 41
3
votes
2
answers
523
views
What is a "surrogate model"?
While reading about model explainability and model accountability, the term surrogate model keeps appearing. I had an idea about what it is but it does not seem to make sense anymore:
What is a ...
- 6,370
3
votes
3
answers
2k
views
SHAP Explanations in case of repeated train/test split
I am building a XGBoost model with Python and trying to explain it using the beautiful shap package. Apart from calculating SHAP values of each feature, I'd like to show graphs such as the two that ...
- 133
3
votes
1
answer
312
views
How to evaluate the "importance" of a variable in a function
Let's say that we have
$$f(x,y,z) = x/k - (y/k) ((z - x/k)/(z - y/k))$$
$$k = constant \in ]0,1[$$
And I need to show in some way that the variable $x$ is more important in some metric that I don't ...
- 51
3
votes
1
answer
476
views
What is the meaning of an empty SHAP graph in Explainable AI? [closed]
Using Python, I created a neural network to perform predictions on a binary class dataset (e.g. will a passenger survive the Titanic?). I am using the SHAP package to explain individual predictions. ...
- 31
2
votes
4
answers
5k
views
Which AI algorithm is best for chess?
I'm working on my chess bot, and I would like to implement simple artificial intelligence for it. I'm new in it, so I'm unsure how to do it specifically on chess. I heard about Q-learning, Supervised/...
- 129
2
votes
1
answer
1k
views
Interpretable xgboost - Calculate cover feature importance
When trying to interpret the results of a gradient boosting (or any decision tree) one can plot the feature importance.
There are same parameters in the xgb api such as: weight, gain, cover, ...
- 6,370
2
votes
2
answers
259
views
How to plot number of Trees and OOBs score with Grid Search
I searched to find the answer but I don´t find something with Grid Search.
I create a random forest and gradient boosting regressor with grid search. Now I want to make a visualization to see if the ...
- 347
2
votes
2
answers
894
views
What is the difference (interpretation) between the partial R^2 and the SHAP value for a linear regression model?
To calculate the coefficient of partial determination R2 for a given variable:
We calculate the R2 with and without that variable and substract them. This implies fitting a different model with and ...
- 185
2
votes
1
answer
2k
views
Is there a way to output feature importance based on the outputted class?
I'm running a random forest classifier in Python (two classes). I am using the feature_importances_ method of the ...
- 83
2
votes
2
answers
144
views
How to stop a text-classification model from depending on only couple of the words from input text instead of entire sentence?
I have a text classification deep-learning model, which takes in a text and outputs a softmax probability. I am using glove embeddings to represent my input text in numerical form for the DL model.
...
2
votes
4
answers
927
views
Explainable anomaly detection
There are plenty of working for explaining prediction in supervised learning (e.g. SHAP values, LIME).
What about for anomaly detection in unsupervised learning? Is there any model for which there ...
- 121
2
votes
2
answers
60
views
How best to explain Forecasts when model cannot be accessed?
My company purchases demand forecasts from an external vendor (after providing them with our historical data). My manager wants to explain the forecasts that we are receiving and has requested for ...
- 53
2
votes
1
answer
102
views
Shapley contribution when coalition is 0
I am exploring Shapley for channel attribution based on [here][1]
Consider C1, C2, C3, C4 as 4 channels in question.
Some of the coalition does not have value, such as
...
- 121
2
votes
1
answer
77
views
treeExplainer algorithm intuition
I'm reading the paper about the treeExplainer; the pseudo-code of Algorithm 1 is a bit cryptical as most of the variables are not even defined (same with sampling and all details involved).
Is there a ...
- 45
2
votes
2
answers
566
views
Reasons that LIME and SHAP might not agree with intuition
I'm leveraging the Python packages lime and shap to explain single (test-set) predictions that a basic, trained model is making ...
- 121
2
votes
2
answers
120
views
Explainability ML Methods
I am currently working on a Machine-Learning Model. In order to explain how it works, I have looked at Partial Dependence Plots, Feature Importance and all kinds of methods, but one thing still ...
- 131
2
votes
1
answer
569
views
How to handle the parameter space of neural networks?
This question is very broad (and might even be closed as "too broad"). It can be considered as a beginners question, because it is largely about getting started in terms of heading into a direction ...
- 400
2
votes
0
answers
820
views
How SHAP value explains contribution of features for outliers event?
I'm trying to understand and experiment with how the SHAP value can explain behaviour for each outlier events (rows) and how it can be related to shap.force_plot(). ...
- 432
2
votes
0
answers
508
views
Aggregate SHAP importances from different models
A couple of questions on the SHAP approach to the estimation of feature importance.
I would like to use the random forest, logistic regression, SVM, and kNN to train four classification models on a ...
- 21
2
votes
0
answers
338
views
Multi-valued categorical features in LIME
I am working with the LIME implementation by Marco Ribeiro (https://github.com/marcotcr/lime). Specifically, I am utilizing the LimeTabularExplainer as I have a mixture of numerical and categorical ...
- 21
1
vote
2
answers
2k
views
Explainable AI and unsupervised algorithms
There are several packages that allow explaining ML algorithms (Lime, Shap and so on). However, it is not clear how we can explain unsupervised algorithms for example, if we use PCA for dimensionality ...
1
vote
2
answers
297
views
What does a conservative technique mean in the context of neural networks?
I am reading this article on Layer-wise relevance propagation method and I can't understand this particular paragraph
LRP is a conservative technique, meaning the magnitude of any output y is ...
- 173
1
vote
2
answers
2k
views
Can Shapley/Lime values be used for unsupervised learning?
One thing that is really useful when trying to understand what a machine learning model does, is seeing why some instances got predicted. For that Shapley Values and Lime are really usefull. But can ...
- 6,370
1
vote
1
answer
264
views
LIME Random Forest explanations:
I'm using LIME to explain my random forest model. Everything is working great. However, I don't quite understand the image that is generated. Taking the example from the Readme:
How can it predict ...
- 135
1
vote
1
answer
508
views
Captum vs GNNExplainer for explainability in Graph Neural Networks
I'm new to Graph Neural Networks and interested in exploring frameworks that allow the identification of nodes/edges that underlie prediction.
I came across :
(1) a model architecture (GNNExplainer) ...
- 111
1
vote
1
answer
1k
views
Understanding hierarchical clustering features importance
I made a hierarchical clustering with scikit :
selected_model = AgglomerativeClustering(n_clusters=8)
hierarchical_clustering8 = selected_model.fit_predict(answers)
...
- 121
1
vote
1
answer
1k
views
Explainability and Autoencoders
suppose I have an autoencoder as a two-stack LSTM that takes in sequences of $n$ features of some length $m$.
Let's say that the dimension of my encoding vector is $k$, so the architecture is of the ...
- 338
1
vote
1
answer
32
views
Explainable AI, how did the computer classified Text?
My question is not about explaining the model or the algorithm
like which neurons were triggered and what are the parameters of perceptrons.
I will explain further
The problem
I have medical reports I ...
- 549
1
vote
1
answer
64
views
Uncertainty in connection to explainability
When I write "uncertainty" in this post I mean:
If I have a classifier into $a_1,..,a_n$ categories and for an observation $x$ I classify $x$ to $a_i$ with probability $p_i$, then the ...
- 338
1
vote
0
answers
307
views
How do I interpret GRAD-CAM's feature attribution to time series zero-padding in a CNN classifier?
Problem setting: MTS Classification with CNN architecture
I have a multivariate time series (MTS) dataset that contains 30 features. The goal is to solve a classification problem on this MTS dataset. ...
1
vote
1
answer
33
views
Is there such thing as dataset imrovement?
I know that we can use explained machine learning to find why a model chose a certain classification.
I wonder if there is a way I can find which features are going to improve my current model.
I will ...
- 549
1
vote
1
answer
231
views
Can we add baseline to SHAP?
I have a doubt. I am currently using an integrated gradient for the DNN model for explainability. In that, we can specify the baseline as a parameter to the function. I am using all zeros for this. I ...
- 193
1
vote
0
answers
13
views
Why calculating how much removed sentences with most contributing words to the result helps to show that a model is "*faithful*"?
I don't understand how the calculation score taking out the sentences where the words contribute the most of to the result helps to show to what extent a model is "faithful" to a reasoning ...
1
vote
0
answers
55
views
How to interpret integrated gradients in an NLP toxic text classification use-case?
I am trying to understand how integrated gradients work in the NLP case.
Let $F: \mathbb{R}^{n} \rightarrow[0,1]$ a function representing a neural network, $x \in \mathbb{R}^{n}$ an input and $x' \in ...
1
vote
0
answers
45
views
Transparent Matching / Recommendation System [closed]
I am thinking of a matching/recommendation algorithm which matches students to the right teachers for their individual problems.
The dataset would look like this:
Student
Name
Age
Gender
Weak ...
- 31
1
vote
0
answers
105
views
Constrastive vs Counterfactual Explanations
Is there any canonical definition for Contrastive vs Counterfactual explanations?
In the literature, I keep reading different versions but I wonder if there are good definitions or illustrative ...
- 6,370
1
vote
0
answers
56
views
Passing reduced/different feature data to LimeTabularExplainer compared to the original model
I am trying to use LimeTabularExplainer class and explain_instance function to find explainations of my LightGbm (lgb) model. However, the lgb model uses complex feature set which are not ...
- 11
1
vote
0
answers
34
views
What is the meaning of Information with respect to interpretability approaches in machine learning?
I was going through a pre-print on arXiv named "Quantifying interpretability and trust in machine learning systems". There, I found that a comparison of two interpretability approaches - ...
- 11
1
vote
0
answers
442
views
Partial Dependence Plot and categorical variables
While reading about machine learning explainability and Partial Dependence Plot (PDP) in this book, the following appeared when dealing with categorical variables:
For each of the categories, we get ...
- 6,370