I'm reading the following really interesting paper

https://arxiv.org/pdf/1602.04938.pdf on local interpretable model explanations

on page 3 however particularly section 3.3 Sampling for Local Exploration they mention obtaining perturbed samples $z' \in \{0,1\}^{d'}$, it then says

"we recover the sample in the original representation $z \in \mathbb{R}^{d}$ and obtain $f(z)$ "

with no indication how this is done, surely the map is not injective? If not how would you know you recovered the correct sample? To this end, i wondering how something like this might be done in practice, moving from one feature space $\mathbb{R}^{d}$ to another $\{0,1\}^{d'}$. I'd really appreciate any help.


1 Answer 1


Welcome to the community @iaaml! I hope I understood the concept right by briefly going through your reference. This is my impression:

in 3.1, they say

For example, a possible interpretable representation for text classification is a binary vector indicating the presence or absence of a word.

So, I suppose the point is something like Sparse Representation (you may look for sparse representation to see the methods and examples). For instance, an image vector which is predicted as cat can be explained by a 3d explainer, namely, nose, glasses and mouth. Nose and mouth are evidence for cat while glasses is a human face feature (very naive example). So having a vocabulary for all explainers, you can come up with a sparse representation of each decision in which, significant criteria are represented with 0 or 1 to help validating prediction. It happens by examining the features which contribute the most to that class (that's why in Figure.1 they could understand that "No Fatigue" observation is against the prediction).

To obtain such representation, you can build the vocabulary (a set of features which are significant and all together cover whole or most of the space). Then you map your data on these space in which presence of each element of vocabulary is 0 or 1.


I have three samples and their corresponding predictions:

a) Spanish is the main language in Buenos Aires         :Argentina
b) Apple released its new software                      :IT
c) Apple is the man agriculture product in Buenos Aires :Argentina

Using BoW for constructing the feature space, Apple becomes an important feature as it is a famous IT company, but in the last sentence it affects in a wrong way. Alongside, you can also have a map of which feature contributes the most to which class (let's say through Mutual Information or any other feature ranking method, specified on different classes) and construct the matrices below:

   BuenosAires  Apple
a     1           0
b     0           1
c     1           1

when you have what should happen according to the feature ranking for each class:

   BuenosAires  Apple
a     1           0
b     0           1
c     1           0

comparing these two gives you the probably-wrong Apple in last sentence (like "No Fatigue" in Figure.1). The first matrix is the mapping that you do, and the second is the mapping that feature ranking gives you.

Hope I understood it right!

  • $\begingroup$ Hi Kasra, I think i understand the mapping from lets say sentences to the bag of words example this is the mapping $x \in \mathbb{R}^{d}$ to $x' \in \{0,1\}^{d'}$. What i'm confused with is that the space that gets perturbed is $\{0,1\}^{d'}$ in a sense allowing us to create synthetic data, but then to evaluate the a model in this new space given some $z'$ from the perturbation how do we find the original $z$? So i guess in this case it would be like saying given a bag of words representation how would i know what the original sentence looked like? Correct me if you actually did explain this. $\endgroup$
    – iaaml
    Commented Oct 26, 2018 at 13:55
  • $\begingroup$ I just looked into the sparse matrix representation, it looks like although there are infinitely many solutions they find a way to get back to the original space so would that be the solution? $\endgroup$
    – iaaml
    Commented Oct 26, 2018 at 14:03
  • $\begingroup$ Yes but here I have the same confusion. The fact is that if they choose a fraction of features, then they can not recover the point in the original space as they cut the information. I think they choose a fraction of features and set others to zero. In this case you can have a representation which approximates a desired neighborhood in the original space. Or they purturb a fraction of features WHILE KEEPING THE REST. So concatenating new samples with rest of original features takes them back to the original space. $\endgroup$ Commented Oct 26, 2018 at 14:19
  • $\begingroup$ I admit it was not an easy question. Just tried it. Seems we need to sleep on it more. $\endgroup$ Commented Oct 26, 2018 at 14:19
  • 1
    $\begingroup$ Thanks anyway! Look forward to finding a potential solution $\endgroup$
    – iaaml
    Commented Oct 26, 2018 at 15:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.