How to evaluate the "importance" of a variable in a function

Question

Let's say that we have $$f(x,y,z) = x/k - (y/k) ((z - x/k)/(z - y/k))$$ $$k = constant \in ]0,1[$$

And I need to show in some way that the variable $x$ is more important in some metric that I don't know which one could be good. I thought about analyze the partial derivatives of that functions, but I don't think that is a good way, because one will only see some restricted path through the surface.

Another approach would be do monte carlo simulation in some domain of interest (it exists, actually is a real world problem) and see that the variance of the function value increases more varying x fixing the another variables rather than the same simulation with $x$ fixed.

In some extent one can thinking in this function as a model, and I need to calculate the variable importance, it could be a way as well.

But I'm really puzzled how to approach this problem.

Any metric or some approach is welcomed!

Thank you very much!

Answer 1

The way to start would be to understand intuitively what makes some variable important.

For example, the output value of the function might depend to a large extend on the value on that variable (so correlation would be quite high between output and that variable).

Another example would be to make that variable zero and measure the correlation between the original function and the new function (eg it would exhibit low correlation).

Still another way would be to compute the rate of change (derivative) of the function with respect to the rates of change (derivatives) of each variable and see how much a small perturbation of that variable affects the output of the function.

Similar other approaches are possible as well. Since you have the formula of the function wrt the variables one of the above approaches is fine.

There is no agreed upon universal metric for variable/feature importance but there are various approaches one can take depending on problem at hand.

A generic approach is proposed below (from Assessing Variable Importance for Predictive Models of Arbitrary Type)

[A] “good” variable importance measure should exhibit these characteristics:

1. it should be applicable to any model type;
2. it should be sensible to combine or compare results across different model types.

One strategy that meets both of these criteria is that suggested by Friedman (2001), applying a random permutation to each covariate and thus effectively removing it from the model, and then examining the impact of this change on the predictive performance of the resulting model. The basic idea is that if we remove an important covariate from consideration, the best achievable model performance should suffer significantly, while if we remove an unimportant covariate, we should see little or no change in performance.

[..]Ideally, we would like to implement the strategy just described by applying many random permutations and combining the results, but this ideal approach is computationally expensive. In his application to gradient boosting machines, Friedman (2001) could “borrow strength” by combining the random covariate permutations with the random subsampling inherent in the model fitting procedure. In characterizing arbitrary models, however, we cannot exploit their internal structure, but we can borrow strength by combining results across the different models we are fitting. Thus, the approach taken here consists of the following steps:

1. Run a DataRobot modeling project based on the original data and retrieve the results;
2. For each covariate whose influence we are interested in assessing:
    a. Generate a modified dataset, replacing the covariate with its random permutation, leaving all other covariates and the target variable unmodified;
    b. Set up and run a new modeling project based on this modified dataset;
    c. Retrieve the project information and compute the performance differences between the models in this project and the same models in the original project.
3. Characterize each covariate, in one of the following three ways:
    a. Select an individual model of particular interest (e.g., the model that performs best in the original project) and compute its performance degradation;
    b. Compute the average performance degradation over all project models;
    c. Compute a performance-weighted average of the degradation, like that defined in Section 5.

Further reading:

1. Variable importance analysis: A comprehensive review

2. Assessing Variable Importance for Predictive Models of Arbitrary Type

3. Variable-importance Measures

4. Methods to quantify variable importance: implications for the analysis of noisy ecological data