I am working on a production optimization problem; a very similar idea to what is described by Vegard Flovik How to use machine learning for production optimization. The following image, taken from the referred post, summarizes it very well:
First step is obvious, and I do have a model in the form of machine learning or neural networks model. How would I go about the second step? How can I use the trained model as the function evaluator for further multi-dimensional nonlinear optimization (e.g. maximization) either via Scipy, Bayesian Optimization etc.?
I cannot seem to find a practical example. Having a closed-form analytical function as the objective of an optimization problem is well-established. The article Optimization with SciPy and application ideas to machine learning by Tirthajyoti Sarkar gives a few examples using Scipy, & introduces packages that do optimizations with bound constrains and more. Yet examples are quite simple (a closed-form mathematical function) and he only glosses over the extension of such idea to use NN as the objective function, I'm quoting:
You are free to choose an analytical function, a deep learning network (perhaps as a regression model), or even a complicated simulation model, and throw them all together into the pit of optimization.
Any leads/hints/links are appreciated!
In order to have a concrete example, let's imagine we have a dummy data set with a set of feature and a imaginary ProductionYield that is a nonlinear combination of input variables:
import numpy as np import pandas as pd df = pd.DataFrame(columns=['Pressure','Temprerature','Speed','ProductionYield']) df['Pressure'] = np.random.randint(low= 2, high=10, size=2000) df['Temprerature'] = np.random.randint(10, 30, size=2000) df['Speed'] = np.random.weibull(2, size=2000) df['ProductionYield'] = (df['Pressure'])**2 + df['Temprerature'] * df['Speed'] + 10 df['ProductionYield']= df['ProductionYield'].clip(0, 100) Pressure Temprerature Speed ProductionYield 0 7 20 1.810557 95.211139 1 2 29 0.674221 33.552409 2 8 17 0.537533 83.138065 3 3 24 1.945914 65.701938 4 6 23 0.514679 57.837610
1.Predictive Algorithm (a simple Neural Network):
## Train/Test Split from sklearn.model_selection import train_test_split x_train, x_test, y_train, y_test = train_test_split(df[['Pressure','Temprerature','Speed']].values, df['ProductionYield'].values, test_size=0.33, random_state=42) ## Build NN Model import tensorflow as tf from tensorflow.keras import layers def build_model(): # create model model = tf.keras.Sequential() model.add(layers.Dense(64, input_dim=3, kernel_initializer='normal', activation='relu')) model.add(layers.Dense(128, kernel_initializer='normal', activation='relu')) model.add(layers.Dense(1, kernel_initializer='normal')) # Compile model model.compile(loss='mean_squared_error', optimizer='adam') return model model = build_model() model.fit(x_train, y_train, validation_split=0.2, verbose=0, epochs=1000)
2.Otimization [Core of the Problem]:
Problem lies herein, when a ML/NN is trained, I do not get to see (export as I wish) the mathematical form of the function (here in this example NN) and its variables (which should be my feature variables) to do the optimization as we do with closed-form explicit mathematical functions.
Following Valentin's great answer, I've put pieces together in a practical example showcasing how one can use a ML/NN model as an input function for further optimization (herein via scipy.optimize) using the dummy data set shown in the Appendix. Please see this notebook for more details.