Questions tagged [explainable-ai]

Use for questions about explainable artificial intelligence (AI), which aims at understanding, interpreting, and explaining the decisions that have been made by complex AI systems

Filter by
Sorted by
Tagged with
0 votes
0 answers
14 views

Explainable AI in manufacturing

Is explainable AI important in manufacturing processes where prediction accuracy is high favourable? How could explainable AI be implemented on a manufacturing dataset if all its predictors were ...
user avatar
0 votes
0 answers
9 views

Derivation of path dependent feature attributions in Tree SHAP

I was reading through the TreeSHAP paper by Lundberg & Lee. They proposed that every path can be considered an individual model and considering additivity property of SHAP - we can sum up the ...
user avatar
3 votes
1 answer
150 views

Is multicollinarity a problem when interpreting SHAP values from an XGBoost model?

I'm using an XGBoost model for multi-class classification and is looking at feature importance by using SHAP values. I'm curious if multicollinarity is a problem for the interpretation of the SHAP ...
user avatar
0 votes
0 answers
18 views

SHAP values interpretation for clasification

I'm trying to understand how SHAP values are calculated for Classification. As far as I understand for each feature the SHAP values are calculated by: $$ \phi_i = \sum_{S \subseteq F \setminus {i}} \...
user avatar
  • 1
1 vote
0 answers
8 views

Why calculating how much removed sentences with most contributing words to the result helps to show that a model is "*faithful*"?

I don't understand how the calculation score taking out the sentences where the words contribute the most of to the result helps to show to what extent a model is "faithful" to a reasoning ...
user avatar
0 votes
0 answers
29 views

Exact Shap calculations for logistic regression?

Given the relatively simple form of the model of standard logistic regression. I was wondering if there is an exact calculation of shap values for logistic regressions. To be clear I am looking for a ...
user avatar
  • 2,175
0 votes
0 answers
37 views

Anomaly detection and root cause analysis

ARIMA is widely used for anomaly detection on time-series data e.g. stock price prediction. ARIMA assumes that future value of a variable (stock price in our case) is dependent on its previous values. ...
user avatar
0 votes
0 answers
46 views

SHAP KernelExplainer AttributeError numpy.ndarray

I've developed a text classifier of the form of python function that can input a np.array of strings (each string is one observation). ...
user avatar
  • 101
0 votes
0 answers
10 views

What is "Gradient × Hidden States" explainability method? Is there any documentation about it?

I am doing a literature review on post-hoc explainability methods based on gradient. I stumbled upon one I didn't heard of to extract highlights from a trained model in this post-hoc fashion: We ...
user avatar
1 vote
0 answers
7 views

How to interpret integrated gradients in an NLP toxic text classification use-case?

I am trying to understand how integrated gradients work in the NLP case. Let $F: \mathbb{R}^{n} \rightarrow[0,1]$ a function representing a neural network, $x \in \mathbb{R}^{n}$ an input and $x' \in ...
user avatar
0 votes
0 answers
16 views

Using ML Interpretability Techniques for Data Analysis Instead of Strictly Model Analysis

Hope you lot are doing alright. I have been looking into Explainable AI and model interpretability lately, and I had an idea but am wondering whether it would constitute a valid use case. There is a ...
user avatar
0 votes
0 answers
190 views

Why does SHAP's TreeExplainer with "interventional" method not match exact SHAP?

I am trying to understand the concepts/definitions behind the SHAP method of explaining model predictions. In particular I've read the original SHAP paper and the TreeExplainer paper. The original ...
user avatar
  • 1
0 votes
1 answer
68 views

What is the SHAP values for a liner model? How do we derive that?

What is the SHAP values for a linear model? it is given as below in the documentation Assuming features are independent leads to interventional SHAP values which for a linear model are coef[i] * (x[i]...
user avatar
  • 231
1 vote
1 answer
55 views

Is there a way to output feature importance based on the outputted class?

I'm running a random forest classifier in Python (two classes). I am using the feature_importances_ method of the ...
user avatar
  • 73
0 votes
0 answers
16 views

What have my models learnt?

I am doing a time series classification task. I used LSTM, Bi-LSTM. Bi-LSTM works a little bit better than single layer LSTM. And concatenating two Bi-LSTM outputs with another input gives me a better ...
user avatar
2 votes
4 answers
2k views

Which AI algorithm is best for chess?

I'm working on my chess bot, and I would like to implement simple artificial intelligence for it. I'm new in it, so I'm unsure how to do it specifically on chess. I heard about Q-learning, Supervised/...
user avatar
  • 129
0 votes
0 answers
222 views

What is the difference between shap kernel-explainer and deep-explainer

I want to use shap to explain my image classification model. I read that it is better to use shap.DeepExplainer (than ...
user avatar
2 votes
2 answers
42 views

How to stop a text-classification model from depending on only couple of the words from input text instead of entire sentence?

I have a text classification deep-learning model, which takes in a text and outputs a softmax probability. I am using glove embeddings to represent my input text in numerical form for the DL model. ...
user avatar
1 vote
0 answers
27 views

Transparent Matching / Recommendation System [closed]

I am thinking of a matching/recommendation algorithm which matches students to the right teachers for their individual problems. The dataset would look like this: Student Name Age Gender Weak ...
user avatar
1 vote
1 answer
248 views

Understanding hierarchical clustering features importance

I made a hierarchical clustering with scikit : selected_model = AgglomerativeClustering(n_clusters=8) hierarchical_clustering8 = selected_model.fit_predict(answers) ...
user avatar
  • 121
0 votes
0 answers
40 views

meaning of shapley values using SHAP

I have a binary classification - "BAD" and "GOOD" samples. The features are binary as well, either 0 or 1 (each sample is a boolean vector of size 264. I got about 3000 "BAD&...
user avatar
0 votes
0 answers
8 views

How to do Interpretable AI with multiple image input fields and a single real-valued output image?

I have input data of the shape: (num_ex, num_features, 64, 64) and real-valued output data of shape: (num_ex, 1, 64, 64) I ...
user avatar
  • 101
0 votes
0 answers
20 views

How can I track the percentage impact of variables in a ML model?

Using a Random Forest Model (for example), I can find out the list of variables that are most important. However, I would like to see the relative impact of the explanatory variables on the target ...
user avatar
0 votes
1 answer
343 views

Interpretable vs. transparent ML algorithms

What is the difference between interpretable algorithm and transparent algorithm? In particular, is there an algorithm that is interpretable but not transparent? Update: In this post we defined ...
user avatar
  • 3
0 votes
1 answer
151 views

An example of explainable, but not interpretable ML model

This post attempts to explain the difference between explainability and interpretability of ML models. However, the explanation is somewhat unclear. Can somebody provide specific examples of models ...
user avatar
  • 3
1 vote
1 answer
182 views

Explainability and Autoencoders

suppose I have an autoencoder as a two-stack LSTM that takes in sequences of $n$ features of some length $m$. Let's say that the dimension of my encoding vector is $k$, so the architecture is of the ...
user avatar
  • 328
1 vote
0 answers
47 views

Constrastive vs Counterfactual Explanations

Is there any canonical definition for Contrastive vs Counterfactual explanations? In the literature, I keep reading different versions but I wonder if there are good definitions or illustrative ...
user avatar
2 votes
0 answers
410 views

How SHAP value explains contribution of features for outliers event?

I'm trying to understand and experiment with how the SHAP value can explain behaviour for each outlier events (rows) and how it can be related to shap.force_plot(). ...
user avatar
  • 389
2 votes
1 answer
98 views

How to evaluate the "importance" of a variable in a function

Let's say that we have $$f(x,y,z) = x/k - (y/k) ((z - x/k)/(z - y/k))$$ $$k = constant \in ]0,1[$$ And I need to show in some way that the variable $x$ is more important in some metric that I don't ...
user avatar
1 vote
1 answer
41 views

Shapley contribution when coalition is 0

I am exploring Shapley for channel attribution based on [here][1] Consider C1, C2, C3, C4 as 4 channels in question. Some of the coalition does not have value, such as ...
user avatar
  • 111
4 votes
2 answers
146 views

Different feature importance results between DNN, Random Forests and Gradient Boosted Decision Trees

I've been modeling metabolite data with 3 different regressor models. I get similar results from running feature importance with Random Forest model and Gradient Boosted Decision Trees (where I used ...
user avatar
  • 41
5 votes
1 answer
203 views

Shapley values without intercept (or without `expected_value`)

I have a model and I want to derive its interpretability by using feature contributions. In the end, I want to have some contribution per feature such that the sum of contributions equals the ...
user avatar
  • 5,674
1 vote
0 answers
40 views

Passing reduced/different feature data to LimeTabularExplainer compared to the original model

I am trying to use LimeTabularExplainer class and explain_instance function to find explainations of my LightGbm (lgb) model. However, the lgb model uses complex feature set which are not ...
user avatar
  • 11
0 votes
1 answer
26 views

Types of maps in Interpretable Machine Learning

I have worked on Interpretable Machine Learning (IML) for over 1 year. However, there are some terminologies that always make me confused. For example, saliency maps/heat maps. Are they same? Are ...
user avatar
1 vote
0 answers
32 views

What is the meaning of Information with respect to interpretability approaches in machine learning?

I was going through a pre-print on arXiv named "Quantifying interpretability and trust in machine learning systems". There, I found that a comparison of two interpretability approaches - ...
user avatar
1 vote
1 answer
24 views

Explainable AI, how did the computer classified Text?

My question is not about explaining the model or the algorithm like which neurons were triggered and what are the parameters of perceptrons. I will explain further The problem I have medical reports I ...
user avatar
  • 497
1 vote
1 answer
756 views

Can Shapley/Lime values be used for unsupervised learning?

One thing that is really useful when trying to understand what a machine learning model does, is seeing why some instances got predicted. For that Shapley Values and Lime are really usefull. But can ...
user avatar
1 vote
0 answers
309 views

Partial Dependence Plot and categorical variables

While reading about machine learning explainability and Partial Dependence Plot (PDP) in this book, the following appeared when dealing with categorical variables: For each of the categories, we get ...
user avatar
1 vote
1 answer
47 views

Uncertainty in connection to explainability

When I write "uncertainty" in this post I mean: If I have a classifier into $a_1,..,a_n$ categories and for an observation $x$ I classify $x$ to $a_i$ with probability $p_i$, then the ...
user avatar
  • 328
3 votes
2 answers
141 views

What is a "surrogate model"?

While reading about model explainability and model accountability, the term surrogate model keeps appearing. I had an idea about what it is but it does not seem to make sense anymore: What is a ...
user avatar
0 votes
0 answers
38 views

Ways to visualize the outcome of machine learning interpretability techniques (for image classification)

“Machine Learning Interpretability” or “Explainable Artificial Intelligence” has become quite popular in the machine learning community and in recent research. The goal is to make complex (deep ...
user avatar
1 vote
0 answers
199 views

Aggregate SHAP importances from different models

A couple of questions on the SHAP approach to the estimation of feature importance. I would like to use the random forest, logistic regression, SVM, and kNN to train four classification models on a ...
user avatar
4 votes
1 answer
4k views

How is the "base value" of SHAP values calculated?

I'm trying to understand how the base value is calculated. So I used an example from SHAP's github notebook, Census income classification with LightGBM. Right after I trained the lightgbm model, I ...
user avatar
1 vote
0 answers
37 views

Can we use Shap to interpret output changes?

Can we calculate the difference between Shapley values to interpret changes in the output? More precisely, if we get Shapley values for two different inputs, can we compare them to understand how much ...
user avatar
  • 153
1 vote
2 answers
96 views

What does a conservative technique mean in the context of neural networks?

I am reading this article on Layer-wise relevance propagation method and I can't understand this particular paragraph LRP is a conservative technique, meaning the magnitude of any output y is ...
user avatar
2 votes
1 answer
67 views

treeExplainer algorithm intuition

I'm reading the paper about the treeExplainer; the pseudo-code of Algorithm 1 is a bit cryptical as most of the variables are not even defined (same with sampling and all details involved). Is there a ...
user avatar
  • 45
18 votes
5 answers
4k views

What is the difference between explainable and interpretable machine learning?

O’Rourke says that explainable ML uses a black box model and explains it afterwards, whereas interpretable ML uses models that are no black boxes. Christoph Molnar says interpretable ML refers to the ...
user avatar
1 vote
0 answers
303 views

Tree Path Dependent expected value

I stuck on this topic for couple days and seems I need your help to understand what is expected value in TreeExplainer when feature_perturbation = tree_path_dependent, precisely what is expected value ...
user avatar
2 votes
1 answer
547 views

Interpretable xgboost - Calculate cover feature importance

When trying to interpret the results of a gradient boosting (or any decision tree) one can plot the feature importance. There are same parameters in the xgb api such as: weight, gain, cover, ...
user avatar
1 vote
0 answers
45 views

Does EDA helps only in case of linear regression?

I know what Explanatory data analysis is and how it helps us investigate and understand the data. What I dont understand is how does this help in case of nonlinear relationships? I mean if I'm using ...
user avatar
  • 223