# Calculating optimal path(s) to increase probability of a classification

I have trained several classifiers using Python's scikit-learn which are fairly accurate when applied on a test set at identifying different classes with a standardized set of input features. These different classifiers provide a certain probability for the classification.

The input features are controllable physical parameters that I am measuring (e.g. temperature, volume) which intricately influence an output which can essentially be either 1 or 0 (and others in mutli-class cases). I can already do basic identification, but what I am curious about is: given an initial feature vector starting in class 0, are there known methods to find the optimal ways to change my input features so as to increase my probability of going into class 1? The input feature space has a high number of dimensions, and there may be certain constraints on the inputs (e.g. temperature cannot exceed a certain value if volume is kept at a particular value).

• Does the response depend on the state; what the configuration was recently? Read about active learning and reinforcement learning. – Emre Jun 29 '18 at 22:39
• The response would be based on figuring out how to optimally change the input themselves to increase the probability of going from an output of 0 (the current state) to 1 (the desired state). – Mathews24 Jun 30 '18 at 3:38
• @Emre To clarify, I am not necessarily interested in further training the classifier. I am interested in using its already learned algorithm to guide me in determining how I should change the inputs to achieve a certain output (i.e. higher probability of an output of 1). I have ideas on how to create this scheme, but am wondering if implementations already exist. – Mathews24 Jun 30 '18 at 13:53
• Depends what kind of classifier, but if you are using DNN this is basically the idea behind adversarial attacks. – Bruno Klein Jul 2 '18 at 0:14
• @BrunoKlein I am namely using random forests, although would be interested in learning about analogous examples for multilayer perceptron or even logistic regression. I'm certainly looking into adversarial attacks as it's relevant, but have yet to find concrete existing codes/packages which are able to provide optimal inputs based on a desired output and already trained classifiers. – Mathews24 Jul 2 '18 at 4:31

This is a multivariate optimisation problem. You have a function f(X) that returns an output that you want to maximise - Maximising the probability of belonging to class. That function f(X) is just your model.

You also mention "constraints", and again this is standard constrained optimisation territory.

The problem itself may be hard to solve, but the framework seems to be that.

Have a look at scipy optimise here

• Yes, it is exactly as you say a multivariate optimization problem for which there are known methods. What I am seeking are implementations that can currently tackle this problem and particular methods to do so with existing classifiers (e.g. random forest, logistic regression). If none exist, that is fine, but this appears to be a quite general problem. – Mathews24 Jul 9 '18 at 0:49
• Hm. I'm not aware of there being out-of-the-box implementations. But any non linear optimizer where you can pass in a function should fit nicely into your problem I believe. Then the question is: how simple to use and easy to pass any function, is the optimizer? I think most are quite good, and the main differences are the ones that handle multivariate inputs or not. – f.g. Jul 10 '18 at 6:50
• Exactly. For logistic regression this seems doable. But for random forests, I'm quite uncertain although will explore. – Mathews24 Jul 10 '18 at 13:29
• I agree that Random Forests will be harder - the output will be some non convex function of all the inputs. Non deterministic non linear optimiser will be the thing to explore - to the extent that it won't get stuck in local optima. – f.g. Jul 15 '18 at 7:18

Following your model choice, you could pick n observation labeled "0" and create from each m syinthetic new cases to feed to your best model in order to get predicitons.

Basically you brute force-simulate new data changing the features starting from the input vector you already have.

Say you have this feat. vector, you know is labeled zero:

x1 x2 x3 x4  x5 x6  y
20 5  1  0.5 6  10  0


From this you simulate m new feature vector, using some prior knowledge about x1-x6.

For example you start changing some inputs with higher values, and other with lower values (again it's best if you have some prior knowledge).

After predicting the probability for each m new vector, you'll start seeing some pattern perhaps.

To me, this is something worth trying and more interesting than simple importance measures obtained from RFs. Because they tend not to get the "big" picture, in other words the relationship conjoined between features.

This are just some thoughts, hope it helps.

• Yes, this kind of brute-force approach appears to be the way to go. I've searched quite exhaustively without finding much substance, but it would be helpful to have concrete references that may implement similar techniques for different classifiers. – Mathews24 Jul 9 '18 at 0:52
• that's because it's really hard to generalize it. For starters for each X variable, you could simulate random numbers between the min and max value found for each variable in your data. – RLave Jul 9 '18 at 9:28
• say x1 is one variable, with R, you could to: runif(n, min = min(x1), max = max(x2)). n sets how many numbers to simulate. You repeat this for all your X (with the same n) and you'll have new data to feed to the model in order to get the probabilities – RLave Jul 9 '18 at 9:31
• then start with simple plots, on the x-axis the values from ONE variable (es: x1), and on the y-axis the probabilites you get from the model, when x1 varies, but all other X are costant. With this you can see how the probs change with x1, repeat for all X and you'll have a better idea about the relationship. – RLave Jul 9 '18 at 9:33

This can be answered by performing Sensitivity Analysis on your model (white or black box). This is a well-known technique in the field of Operations Research and it also applies to Machine Learning models. These are some helpful references:

https://ieeexplore.ieee.org/document/7600185/

https://rd.springer.com/chapter/10.1007/978-3-319-43742-2_17

http://www2.cs.uregina.ca/~jtyao/Papers/NAFIPS-E-178.pdf

• Those links were very helpful in getting a qualitative grasp of the concept and some of the numerous methods involved. What I'm seeking are more concrete example of codes attempting these very tasks on different types of classifiers. For example, the second paper has a code appendix, but it's not particularly enlightening nor well-documented. Simple examples combining different classifiers (e.g. via scikit-learn) with these optimization methods would be helpful. – Mathews24 Jul 9 '18 at 1:44
• This is not mentioned in your question, i.e. that you are looking for code examples. – pcko1 Jul 9 '18 at 6:50
• My apologies for the lack of clarity. It is the reason I mentioned/tagged Python's scikit-learn and was searching for known methods to use on these classifiers. This was additionally elaborated in the comments of the original question where I request existing implementations/codes. – Mathews24 Jul 9 '18 at 16:00

You have said in a comment that you use random forests and scikit-learn. The first step that you might take is to identify the features that have the greatest weight in deciding the outcome.

As you can see here this is your classifier's property feature_importances_ which ranks the inputs by their importance. From my experience their importance will usually follow the power-law and you could set a threshold that will reduce the number of dimensions you are looking at significantly.

From here it depends on what approach you would rather take - you can train a classifier that takes these features and outputs the changes you need to switch the resulting class (use regularization to minimize the changes) or you can use an optimization black-box to solve the problem on a per-case basis.

Specifically you could use the scipy.optimize.minimize method to minimize the -(classifier confidence) with constraints.

For example if you want to find $\bar{x}$ s.t. $sample * \bar{x}$ maximizes the classifier's confidence, you could do something on the lines of:

def min_func(x):
return -cls.fit(sample*x)

x0 = [1.3, 0.7, 0.8, 1.9, 1.2]
minimizing_xs = minimize(min_func, x0, method='SLSQP', constraints=cons)


This optimization approach will naturally lend itself to a similar learning approach where you try to learn a generalized vector $\bar{x}$.

• Yes, feature importance has been conducted to already reduce the number of dimensions. What kind of optimization black box are you referring to? – Mathews24 Jul 9 '18 at 1:45
• @Mathews24 - I have added an example to the answer. – ginge Jul 10 '18 at 8:07