I have real technological process, that explained with complex model (xgboost). I.e. current mass of a product (y) depends on current temperature (x1), pressure (x2) and so on. I would like to solve optimization task: which minimal values of the features can be selected, that mass of a product can reach the maximum? It looks like simple optimization task: ||y-y0||^2 where y - equation of the model process and y0 - maximum or some of the closest to maximum values. But it is impossible to get weighted coefficients of the xgboost, so I can`t use skopt and even if I can get the coefficients, the real equation will be very difficult. Only decision that I have right now is sort out all possible values for all possible features, make predictions for this features and choose optimal, if y will reach maximum or close to it. Could you give an advice, how can I solve this problem?

  • $\begingroup$ Have you tried Shap and Eli5 ? People apply PCA to reduce the features too in some comps $\endgroup$
    – Aditya
    Commented Jun 18, 2018 at 2:15
  • $\begingroup$ Shap and Eli5 will help to explain weighted coefficients of xgboost, but I need to solve optimization task: in which minimum x1...xn y reached it`s maximum $\endgroup$ Commented Jun 18, 2018 at 7:49

1 Answer 1


There are several algorithms which can help you in a smart way.

Usually, those algorithms are used to tune the hyperparameters of a model, so this is what you will find in the tutorials/examples. In your case, you have to find a good set of features instead of a good set of hyperparameters, but the principle is the same.

My suggestions:

1) SMAC. This is based on Bayesian optimization. It's an iterative process where a proxy function is built and maximized:

  • the function to be optimized (your XGBoost model) is evaluated in a point (in the feature's hyperspace) where the optimizer believes it can find the maxima (or, in the very first iteration, in a point given by the user);
  • the result is added to the set of all the evaluation points, and this set is used to build the proxy function;
  • the proxy function is maximized, and the coordinates of that maximum are believed to be the same where the original function will have a maximum too.

Those three steps are repeated as much as you want. So, repeat from the first step;

It works both for continuous and for categorical features, and you can also impose some constraints between features.

Here an example for your case, in Python (code not tested):

from smac.configspace import ConfigurationSpace
from ConfigSpace.hyperparameters import UniformFloatHyperparameter, UniformIntegerHyperparameter
from smac.scenario.scenario import Scenario
from smac.facade.smac_facade import SMAC

#a continuous feature that you know has to lie in the [25 ~ 40] range
cont_feat = UniformFloatHyperparameter("a_cont_feature", 25., 40., default_value=35.)

#another continuous feature, [0.05 ~ 4] range
cont_feat2 = UniformFloatHyperparameter("another_cont_feature", 0.05, 4, default_value=1)

#a binary feature
bin_feat = UniformIntegerHyperparameter("a_bin_feature", 0, 1, default_value=1)

#the configuration space where to search for the maxima
cs = ConfigurationSpace()

cs.add_hyperparameters([cont_feat, cont_feat2, bin_feat])

# Scenario object
scenario = Scenario({"run_obj": "quality",   # we optimize quality
                     "runcount-limit": 1000,  # maximum function evaluations
                     "cs": cs,               # the configuration space
                     "cutoff_time": None

#here we include the XGBoost model
def f_to_opt(cfg):

    #here be careful! Your features need to be in the correct order for a correct evaluation of the XGB model
    features = {k : cfg[k] for k in cfg if cfg[k]}
    prediction = model.predict(features)

    return prediction

smac = SMAC(scenario=scenario, rng=np.random.RandomState(42),
opt_feat_set = smac.optimize()

#the set of features which maximize the output
print (opt_feat_set)

2) dlib optimisation. This converge much faster than the previous. As disclaimer, I have to say that this is an algorithm which in principle works only with functions that fulfill a certain criteria, and XGBoost models as functions do not. But in the reality it turns out that this procedure works also for less stringent functions, at least in the cases I tried it. So maybe you want also give a try.

An example code:

import dlib

#here we include the XGBoost model. Note that we cannot use categorical/integer/binary features
def f_to_opt(cont_feat, cont_feat2):
    return model.predict([cont_feat, cont_feat2])

x,y = dlib.find_max_global(holder_table, 
                           [25, 0.05],  # Lower bound constraints on cont_feat and cont_feat2 respectively
                           [40, 4],    # Upper bound constraints on cont_feat and cont_feat2 respectively
                           1000)         # The number of times find_max_global() will call  f_to_opt
  • 1
    $\begingroup$ Excellent! First method is what i was looking for. Thanks a lot! $\endgroup$ Commented Jun 24, 2018 at 13:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.