I am developing a specialized tool to perform dynamic simulations of a specific family of physical systems. This tool has two major parts: establishing the kinetics, and deciding how far into the future to step with the current state and kinetics. I am currently using an industry standard for determining the time step related parameters but I was wondering if using ML I could find a better one for the specific set of problems that I am interested in.
The parameters I seek are a positive integer $k$ and the real-valued time step $\Delta t$ that minimize an error function $\varepsilon$ (or is not too far from its minimum). These values will depend on the current state of the system, denoted by 6 real values $s_i$, plus a set of parameters that define the physical system: an integer $d$, and several real values $p_j$.
I can run simulations given values of $s_j$, $d$, $p_j$, $k$ and $ \Delta t$ and compute the value of the $\varepsilon$. I actually have the data for a multidimensional grid for specific sets of values of each simulation parameter. I am also able to specify which values of $k$ and $\Delta t$ are considered "good" given $s_i$, $d$ and $p_j$ for labeling.
Being new to ML and considering the ample opportunity for knowledge gaps from being self-taught, I haven't been able to figure out a way to use the data to produce a NN or forest that given $s_i$, $d$ and $p_j$ it gives me a pair of values $(k,\Delta t)$ that would yield a good-enough $\varepsilon$.
What recommendations on method or approach would you give me to build a ML model that achieves this?
input: $s_i$, $d$, $p_j$ desired output: $k$, $\Delta t$ such that $\varepsilon$ is low enough
What I have tried: Using pandas and sklearn in python I filtered my data into one-row-per-distinct-input where the "surviving" pair $(k,\Delta t)$ was the best for the corresponding set of input values. I used this data with random forest using out-of-the-box values for the metaparameters, I fed a sample of the filtered data for training and compared its error measure when using a sample of the filtered data for test. Results didn't impress.
Also, using similar samples as described above, but on normalized data, I trained NNs with different number of layers and layer sizes (I made up the numbers of each). Input layer was the input parameters, output layer had two nodes, one for $k$ and one for $\Delta t$. Training used the corresponding values normalized. Results didn't impress.
Finally I also tried using a different output layer consisting of as many nodes as $k$ has possible values, and for training I converted the values into a "one hot" style output. For instance if $k$ can take the values $4$, $5$ or $6$ and for a specific set of input values the $k$ that produced the best result was $6$ then I would set the corresponding output to $(0,0,1)$. I only used the filtered data for this too.
I don't know if only using the filtered data (where each row is a unique set of input parameters and the corresponding best pair of $k$ and $\Delta t$) was good for training or bad for training.
I also don't know if my approaches with forests and NNs are the appropriate way to discover the underlying structure, if there's one, that would reveal a better way to determine the values of $k$ and $\Delta t$.
I further think that some of the $p_j$ could have an effect while other values of might not be as important. But I suspect that not those $p_j$ that matter but a mix of those might be an even better quantity to help determine $k$ and $\Delta t$.