15

It could be the way that you encode categorical variables. If you do One Hot Encoding (dummy) each encoded feature will only have two possible values [0,1]. Binary variables normally have less importance in Decision trees given how it is computed the feature weight. Let's say per example, that you are trying to predict the condition of a patient at the ...


10

I found this article by Cynthia Rudin which goes a bit more into the difference between the two terms that is in line with your source from O'Rourke. At the core it is about the time and mechanism of the explanation: A priori (Interpretable) vs. A posterio (Explainable) I found this quote to be very helpful and inline with my own thoughts (emphasis mine): ...


4

A more detailed explanation of cover can be found in the code cover: the sum of second order gradient of training data classified to the leaf, if it is square loss, this simply corresponds to the number of instances in that branch. Deeper in the tree a node is, lower this metric will be You can find this here: cover definition in the code This basically ...


3

In decision tree models features are ranked based on the number of splits they're involved in. Continuous features are split more often than categorical ones.


3

This is very similar to fitting a linear regression and not including an intercept, and I think they will face similar issues. To be very concrete, consider an example with $f(x)=1,\ E=1, \ \phi_1=1, \ \phi_2=-1$. Then your scaling factor is undefined, trying to divide by zero. Well OK, but you won't often get such exact numbers. Let's tweak them to $$f(x)=...


3

PCA isn't a great example, as it's inherently explainable, in the same way that linear regression is inherently explainable. In supervised learning, at a high level, the Shapley value for each feature tells you the extent to which including that feature in an ML model allows you to make better predictions about the target variable than if you did not have ...


2

Shapley values were designed in the context of game theory (source), to share value created by a coalition of player in a game. It has multiple properties, including linearity. The linearity ensure that if you were to average your models, the resulting Shapley value would be the average of Shapley values for individual models. Shapley values are comparable ...


2

To plot feature importance using gridsearch use: x= X_train_v1.columns,y= rf_grid_search_v1.best_estimator_.feature_importances_


2

The LIME framework can probably be used to do this as well. Outlier detection sets a specific label to outliers (say 1), and another one to inliers (say 0). From then on, you can train interpretable models (decision trees for instance) to predict the labels set by your unsupervised model. I don't know much about SHAP values, but I guess, with this approach,...


2

A "surrogate" is just a stand-in or proxy. In data science the word "surrogate" is used in more than one way (Bayesian hyperparameter optimization comes to mind). For interpretability, it seems to be used mostly to mean a more-interpretable model (maybe linear/logistic regression) that is trained to approximate the main, usually black-...


2

A surrogate model is an approximation model for a given function. The original function is generally a black box function that we can sample from and based on the samples we can optimize our surrogate model to approximate the behaviour of the original function. A surrogate model can be a neural network, an ensemble method, a gaussian process which can be ...


2

As you say, it's the value of a feature-less model, which generally is the average of the outcome variable in the training set (often in log-odds, if classification). With force_plot, you actually pass your desired base value as the first parameter; in that notebook's case it is explainer.expected_value[1], the average of the second class. https://github....


2

In this layered graph structure, the relevance conservation property can be formulated as follows: Let j and k be indices for neurons of two successive layers. Let Rk be the relevance of neuron k for the prediction f (x). We define R j←k as the share of Rk that is redistributed to neuron j in the lower layer. The conservation property for this neuron imposes ...


2

I don't think so, not directly. SHAP is trying to explain each feature's effect on the prediction, but you have no label here. It might be better to ask therefore, what are you trying to explain? In the case of an isolation forest, you can find the short path through the trees to any anomaly. That path tells you why the trees separated it, based on what ...


2

Few factors that can cause such differences - A common issue with the default RF FE approach. It's not reliable in situations where potential predictor variables vary in their scale of measurement or their number of categories Read Correlated Features When you have correlated features, RF Tree-based FI approach will divide the importance depending upon ...


2

The way to start would be to understand intuitively what makes some variable important. For example, the output value of the function might depend to a large extend on the value on that variable (so correlation would be quite high between output and that variable). Another example would be to make that variable zero and measure the correlation between the ...


1

Why would you expect them to be the same? In one hand Random Forest and Gradient boosting are two types of different ensembles. Even if their estimator is a decision tree and they both seem to measure in scikit learn impurity-based feature importance. The result will be different. Not much is my guess, but different. For Deep Neural Networks, you are ...


1

There are libraries like ELI5 or LIME, which can provide explanations for text classification, here is a link to an example: https://eli5.readthedocs.io/en/latest/tutorials/black-box-text-classifiers.html#textexplainer import eli5 from eli5.lime import TextExplainer te = TextExplainer(random_state=42) te.fit(doc, pipe.predict_proba) te.show_prediction(...


1

There is a bit of confusion, I'm afraid: The definition you propose for uncertainty doesn't really represent the concept of uncertainty: if $p_i$ is the probability that $x$ belongs to category $a_i$, then $1-p_i$ is just the probability that $x$ doesn't belong to category $a_i$. Yes, if the classifier assigns a very high $p_i$, say 0.99, it is supposed to ...


1

The answer is within your question itself. Conservation means something which has been conserved. Here through the backpropagation process the output is conserved. And the output is the sum of the all the intermediate neurons/pixels that are contributing in determining which feature in a particular input vector contribute the most to a neural network's ...


1

The paper you refer to actually states the following intuition: Algorithm 1 estimates $E[f(X)|do(X_S = x_S )]$ by recursively following the decision path for $x$ if the split feature is in $S$, and taking the weighted average of both branches if the split feature is not in $S$. It seems to be a slight modification of the original description in arxiv ...


1

Explainable Machine Learning is the domain of AI. It consists of interpretable models. One could say the difference is that one is a tool and the other is a field of study. In brief, interpretable machine learning is a tool used to solve problems present in the domain of explainable machine learning. To define your answer: One shall use an interpretable ...


1

As for as explanation is concerned, we need explainability/interpretability at every level- data explanation- tsne, simple plotting. model explainability- by creating surrogate models global explainability- feature importance for all training data local explainability- explanation of every prediction. all types of explainability on iris dataset, will be ...


1

I think the answer mostly lies in the fact that these are just approximations and they're not super exact because of the small data set and nature of decision trees. The prediction was really 1.0 (I'm guessing all trees' leaves agreed entirely on the prediction). If gill_size != broad, it would still be 1.0, you could say (1.13 is meaningless). But maybe ...


1

As pointed by Sean Owen, this is probably linked to correlation in the input variables. Concerning LIME : you can read here : https://christophm.github.io/interpretable-ml-book/lime.html (paragraph 5.7) why a correlation between inputs could pose problem, more specifically due to an implementation problem. (gaussian sampling ignore features correlation). ...


1

From https://en.wikipedia.org/wiki/Shapley_value, it is possible to understand that direct computation of Shapley values is difficult with their general formula : $$ \varphi_i(v) = \frac{1}{\text{number of players}} \sum_{\text{coalitions excluding }i} \frac{\text{marginal contribution of }i\text{ to coalition}}{\text{number of coalitions excluding } i \...


1

If you want to see what is the best parameters choosen for your model you can use rf_grid_search_v1.best_estimator_


1

I think this depends in part on why you want the shap values. It can be helpful to separate (a) explaining the model, and (b) explaining the data; model explainability tools like shap only address (a), which hopefully serves as a proxy for (b). If all you want is model explanations [(a)], then I think your approach 1 is fine; that retrained model is the one ...


1

I'd propose doing a similar thing that you do with your accuracy and Kappa metrics — calculate SHAP values for all 500 splits, and take the average of these $n\_samples \times n\_features \times 500$ matrixes in the third dimension to get $n\_samples \times n\_features$ matrixes, which you can use to create your desired plots.


1

You can look at the different evaluations as evaluation of the different stages of the model. by evaluating the model's performance on the training dataset, you could assess how and what the model has learned from the data structure. This evaluation is mainly relevant for your research stage and the reporting of your results. by evaluating the model's ...


Only top voted, non community-wiki answers of a minimum length are eligible