I am working on a production optimization problem; a very similar idea to what is described by Vegard Flovik How to use machine learning for production optimization. The following image, taken from the referred post, summarizes it very well:

enter image description here

First step is obvious, and I do have a model in the form of machine learning or neural networks model. How would I go about the second step? How can I use the trained model as the function evaluator for further multi-dimensional nonlinear optimization (e.g. maximization) either via Scipy, Bayesian Optimization etc.?

I cannot seem to find a practical example. Having a closed-form analytical function as the objective of an optimization problem is well-established. The article Optimization with SciPy and application ideas to machine learning by Tirthajyoti Sarkar gives a few examples using Scipy, & introduces packages that do optimizations with bound constrains and more. Yet examples are quite simple (a closed-form mathematical function) and he only glosses over the extension of such idea to use NN as the objective function, I'm quoting:

You are free to choose an analytical function, a deep learning network (perhaps as a regression model), or even a complicated simulation model, and throw them all together into the pit of optimization.

Any leads/hints/links are appreciated!


In order to have a concrete example, let's imagine we have a dummy data set with a set of feature and a imaginary ProductionYield that is a nonlinear combination of input variables:

import numpy as np
import pandas as pd

df = pd.DataFrame(columns=['Pressure','Temprerature','Speed','ProductionYield'])

df['Pressure'] = np.random.randint(low= 2, high=10, size=2000)
df['Temprerature'] = np.random.randint(10, 30, size=2000)
df['Speed'] = np.random.weibull(2, size=2000)
df['ProductionYield'] = (df['Pressure'])**2 + df['Temprerature'] * df['Speed'] + 10
df['ProductionYield']= df['ProductionYield'].clip(0, 100)

   Pressure  Temprerature     Speed  ProductionYield
0         7            20  1.810557        95.211139
1         2            29  0.674221        33.552409
2         8            17  0.537533        83.138065
3         3            24  1.945914        65.701938
4         6            23  0.514679        57.837610

1.Predictive Algorithm (a simple Neural Network):

## Train/Test Split
from sklearn.model_selection import train_test_split

x_train, x_test, y_train, y_test = train_test_split(df[['Pressure','Temprerature','Speed']].values, df['ProductionYield'].values, test_size=0.33, random_state=42)

## Build NN Model
import tensorflow as tf
from tensorflow.keras import layers

def build_model():
    # create model
    model = tf.keras.Sequential()
    model.add(layers.Dense(64, input_dim=3, kernel_initializer='normal', activation='relu'))
    model.add(layers.Dense(128, kernel_initializer='normal', activation='relu'))
    model.add(layers.Dense(1, kernel_initializer='normal'))
    # Compile model
    model.compile(loss='mean_squared_error', optimizer='adam')
    return model

model = build_model()
model.fit(x_train, y_train,
          verbose=0, epochs=1000)

2.Otimization [Core of the Problem]:

Problem lies herein, when a ML/NN is trained, I do not get to see (export as I wish) the mathematical form of the function (here in this example NN) and its variables (which should be my feature variables) to do the optimization as we do with closed-form explicit mathematical functions.

[UPDATE 15.01.2021]:

Following Valentin's great answer, I've put pieces together in a practical example showcasing how one can use a ML/NN model as an input function for further optimization (herein via scipy.optimize) using the dummy data set shown in the Appendix. Please see this notebook for more details.

  • 2
    $\begingroup$ You should be able to just use your model's prediction method as the objective function. What optimization framework do you want to use? $\endgroup$ – Ben Reiniger Jan 13 at 16:51
  • $\begingroup$ Thanks for the comment. Well, that doesn't work. I will add an update shortly to the question to show what I have tried. I wouldn't care what optimization framework to use, whatever works at this point is suitable. $\endgroup$ – TwinPenguins Jan 14 at 14:17
  • 1
    $\begingroup$ I realized I was too narrowly focused, and that my previous comment applies exactly when the optimization method is a blackbox, derivative-free one. So +1 to Valentin. As for the error, your objective function is the prediction of the model, so you need to be passing model.predict or similar, possibly wrapping for format of the input and output. And Valentin seems to have addressed that in an edit; I expect debugging may take a few iterations, but the original question seems answered. $\endgroup$ – Ben Reiniger Jan 14 at 15:19

This post seems similar to yours and may help. It seems that what you are looking for is a derivative-free optimization method. The Wikipedia page for the concept lists such methods.

Intuitively, these techniques will sample the function (the network in your case) with various inputs (pressure, temperature, speed) and will figure out which inputs optimize it. Where they differ is in their sampling strategy, as it may be impractical or expensive to sample.

You can use scipy.optimize.minimize to do that. Pass your network as func and use an initial guess, which can be the last values of your variables. Scipy expects a function with the following signature: fun(x, *args) -> float where x is a one-dimensional numpy array. This might mean you will need to wrap your network in something like this:

def wrapper(x, *args) -> float:
    network_input = _numpy_to_valid_network_input(x)

    network_output = network.predict(network_input, *args)

    scipy_output = _network_output_to_float(network_output)

    return scipy_output

And then, you can pass wrapper as your func. Negating the output of scipy_output will turn the minimization problem into a maximization problem. If your input variables are bounded, i.e. [0, 100], you can do two things:

  • Use algorithms that allow you to define these bounds explicitly (i.e. L-BFGS-B using the bounds parameter)
  • Implicitly bound your inputs by using a sigmoid function for instance. By doing something like I did below, you can create a function bounded_value that always returns values within the range of your choice, even if the optimization algorithm might try any float.
import numpy as np

def sigmoid(x):
  return 1 / (1 + np.exp(-x))

def bounded_value(x, min_value=0, max_value=100):
   return min_value + (sigmoid(x) * (max_value - min_value))

If you want to bound your output, that's even easier. If your goal is to minimize, return a very large value if your network outputs any value outside of your range. Obviously, ensuring that your network always returns sensible values could be achieved by tweaking the loss used during training (which is difficult with AutoKeras).

If you want to implement your own method, I would recommend coordinate descent as you don't have many input dimensions (3) and it is quite simple to implement from scratch. Obviously, a brute-force approach where you randomly sample the space and choose the inputs that yielded the best function value is even easier to implement.

  • 1
    $\begingroup$ Thanks for the answer. You touched upon an important concept, derivative-free optimization. Still beyond intuition of the problem, I am facing technical errors at the first step using either scipy.optimize.minimize or any others for that matter when passing NN as func. I will update my question to narrow down the problem. $\endgroup$ – TwinPenguins Jan 14 at 14:20
  • 1
    $\begingroup$ I think I know what the problem might be, I edited my answer $\endgroup$ – Valentin Calomme Jan 14 at 14:29
  • 1
    $\begingroup$ Thanks a lot. You suggestion works, see updated Notebook. I do have a few concerns and follow-up questions if you do not mind. One, is that it seems 'Nelder-Mead' is the derivative-free algo, but does not take constraints. Herein variables should be bound (forgot to mention in the original question). With 'Nelder-Mead' variables goes out-of-bound of what it is in the data (kind of make sense sure). Any idea what other algo to use, maybe COBYLA is good alternative? But how do I pass the constrains since I have no clue about how model takes variable (would it take my original variable labels?) $\endgroup$ – TwinPenguins Jan 14 at 15:45
  • 1
    $\begingroup$ And one more thing, in order to maximize for the target, I have negated the scipy_output in the wrapper, it seems to be fine as scipy optimizer seems to give a maximum value. although, still like input variables, output goes beyond what it may be sensible! In my actual case, output is bound too, let's say [0, 100]! But this way of maximizing make sense to you? $\endgroup$ – TwinPenguins Jan 14 at 15:50
  • 1
    $\begingroup$ Good questions. I updated my answer to answer them. $\endgroup$ – Valentin Calomme Jan 14 at 16:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.