Let’s say I have a set of input variables (A, B, C and D) and I predict a target (y) using a machine learning model (XGBRegressor in my case) with a reasonable performance (5% relative error on test set).

from sklearn.datasets import make_regression
import pandas as pd
from xgboost import XGBRegressor

X, y = make_regression(n_samples=500, n_features=4, n_informative=2, noise=0.3)
X = pd.DataFrame(X, columns=['A', 'B', 'C', 'D'])

model = XGBRegressor()
model.fit(X, y)

Now, I want to do some kind of sensitivity analysis on this model by answering two questions:

  1. What is the impact of a 5% independent increase in variables A, B and C (not D) on the target variable?

  2. From variables A, B, C and D; which combination of values of A, B and C (without touching D) increases the target y value by 10, minimizing the sum of A, B and C.

I have already answered question one (see this gist). However, how can question 2 be coded? I imagine that this implies an optimization problem.

| improve this question | | | | |

Without further knowledge of the domain, there is no simple answer.

  • Are the predictor variables discrete or continuous? If they are discrete, you could search all combinations that sum to 0, then all combinations that sum to 1, etc. until you find a combination that increases y by 10. If they are continuous, then:
  • Is the target variable monotonically increasing with each of the predictor variables? In other words, given A, B, C and D, when increasing A, B or C, will y always either increase or stay the same? If the predictors are continuous and the target variable is monotonically increasing, you can do a search similar to the one above by increasing each variable by a small step until y has increased by more than 10, this will give you an idea, within the range of that small constant, of where the answer to your question lies. You can then reduce the size of the step to find a more precise answer within that range.
  • If the target variable is continuous and not monotonically increasing with the predictor variables, this gets much more complicated. Suppose you've found two points (A1, B1, C1, D) and (A2, B2, C2, D) that do not increase y by 10: nothing guarantees that there is not a point between the two, (A3, B3, C3, D), that does. Your best bet then would probably be to explore the model. Depending on the number of trees in your ensemble, this may or may not be possible (computationally). I think you would have to find all possible combinations of leaves where D is unchanged and y is increased by more than 10, then find the minimum amongst these.
| improve this answer | | | | |
  • $\begingroup$ My variables and targets are all continuous. I was able to code an optimization, but only when the target variable increases monotonically (linear regression). As you indicate, gradient optimization seems impossible with a decision tree model. I will try to see how global heuristic optimization works (as in pygmo). My variables correspond to laboratory physical tests performed on athletes and my target corresponds to a field physical test. I thought I could answer the two questions I propose to guide coaches' decisions ("You must improve test A by 20% to reach your goal"). $\endgroup$ – Romain Martinez Sep 27 '18 at 12:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.