# How to optimize input parameters given target and scoring parameters

I'm new to machine learning/optimization, so I apologize in advance if this has been answered before. I don't know which search terms to use.

I have a large dataset where I have a number of input parameters $$I_1$$, ... $$I_n$$ ($$n$$ up to 10), a target parameter $$T$$ and a scoring parameter $$S$$. There are roughly a million rows of real-life data, where each row has the input parameters, the target parameter and the scoring parameter. The data is operational data from a piece of machinery where $$T$$ is output power, $$S$$ is fuel consumption and $$I_1$$, ... $$I_n$$ are tuning parameters for the machinery. The relationships of the tuning parameters to $$S$$ and $$T$$ are unknown but likely nonlinear. So I basically want to understand how to tune the machinery to produce the desired output power as fuel-efficiently as possible given the parameters I can tune.

What I need is some way of getting to a function where I input $$T$$ and get out the optimal combination of $$I_1$$, ... $$I_n$$ that minimizes $$S$$ for that value of $$T$$. I work in Python, and I assume that it's some combination of sklearn and scipy, but I haven't been able to figure out the steps to take for this type of problem. Thanks in advance.

If your data have simple linear relationship (or you can transform it such that this is the case) you can do regression on $$T$$ and $$S$$, and after that you should have the linear relationship between input parameters and target parameters. After that you can use scipy minimize or any other library to perform constrained optimization. If I follow from my example my idea would be to perform a predefined trade-off between $$T$$ and $$S$$ for example $$\max 0.8 T + 0. 2S$$. This should be easy to code with scipy.

• Thank you. I did forget to tell that the relationships are unknown but very likely nonlinear. Nov 11 '19 at 10:46
• You can apply transformation(log transform, polynomial features, etc) first and see if they perform quite well. For your case it is better to keep a simple model since we are more interested in the inverse function(prediciting what features that give rise to a target) rather than the forward problem(predicting the target). And inverse problem is not as easy to solve when features gets complicated Nov 11 '19 at 11:55
• Another thing you might for example use random forest and neural network for your case and you get a nice prediction. If you were to present how to model the relationship, i dont think you can explain that nicely. You might have the solution but it bears little to no interpretability in my opinion. Nov 11 '19 at 11:57

I can think of several approaches.

### Simple data approach

This first simple approach does not require machine learning techniques at all. Pick a value of $$T$$. From your (quite large) dataset, extract observations for which the $$T$$ value is close enough. Among those observations, the optimal case could be the one minimizing $$S$$, or, if the $$T$$ range is too broad, the one minimizing some combination of $$S$$ and $$T$$ (the performance/cost trade-off must be defined by someone who knows the business).

This approach can work well if the dataset correctly covers the input space with sufficient density. With 1 million observations and 10 dimensions, this could not be enough if the problem is highly erratic.

### Learning approach

Instead of using actual data to optimize $$S$$, you could train models to predict both $$S$$ and $$T$$, and evaluate them on the overall input space. This will give additional "fake" data, in both dense and sparse areas, which can be used to do the same as above.

This approach can be risky, because the model is an estimator, so produces results that are expected to be good. In a large input space, it is likely that a global predictor will fail to be accurate everywhere. To limit the risk, it is possible to combine actual data with results predicted by the models. Using models which include information on their prediction error (such as Gaussian processes) could also help.

### Smarter but more complex: local models

Here's another idea that I find smarter if the problem is rather complex, but requires more work. We start again with the full dataset and a selected value for $$T$$, and filter the dataset to keep values close to $$T$$ (but not too close this time, because we need training data).

It is likely that we select more than one area from the input space (meaning several operating modes of the machine, for instance). So the next step is to split the data into subsets corresponding to these operating modes. This can be done with unsupervised techniques; if the data is highly non linear, I would recommend using kernel-based techniques (spectral clustering for instance). The unsupervised step can include, or not, the $$S$$ value along with standard inputs $$I_k$$.

We now have several subsets, each corresponding to a machine operating mode. Based of the $$S$$ value distribution in each, some may be put aside. On each of the remaining subsets, it is now possible to train models to predict both $$T$$ and $$S$$, as in the second approach; except that this time, we can use simpler models and/or expect them to be more efficient, because we are working on a very small volume of the input space, in which the phenomena should be simpler.

The models built this way can then be evaluated to minimize the value of $$S$$, or more likely, once again, a combination of $$S$$ and $$T$$ that makes sense in terms of business activity.

• Excellent answer, thanks! I don't have enough reputation to upvote, but I definitely appreciate the insight. Dec 3 '19 at 9:39