Using standard ML models for modeling a derivative when the data set only contains function values

Question

I want to model a process that is a function of time, $X(t)$. I have a data set which corresponds to a coarse sampling of the function values $X$ at different time points, $t_1, t_2, ...$. I have a analytic-derived model for the derivative $\frac{dX}{dt}$ = $F(X(t),G(t))$ which has a couple tunable parameters in the (known) functions $F$ and $G$. I've been able to tune these models with the following process:

1. Chose a set of parameters for the functions $F$ and $G$.
2. Do a forward-Euler method from the earliest time in my data set to the ending point in my data set, generating the $X(t)$ which corresponds to the above parameters in $F$ and $G$.
3. Use this $X(t)$, along with my measured data points to generate a model error.
4. Use the above modeled error to go back to (1), iterating and doing an optimization over the parameters in the expression for $\frac{dX}{dt}$ to minimize the error.

Note that in the above I have to deal with the function itself and not the derivative because my measured data is on a coarser time grid than the derivative needs to be computed. For example, my derivative changes meaningfully on a scale of $\Delta t$, but my data set might only have points every $5\Delta t$ to $10 \Delta t$. I cannot take the function values themselves from the data set and create a derivative myself because of this discrepancy in the time scales.

Now this process works pretty well, and I'm able to achieve a pretty good error. However, I'm wondering if there's a better way to model the derivative $\frac{dX}{dt}$ than this analytic model. For example, would a neural network or a regression tree be able to do a better job?

My question is, how would I do this in a computationally efficient way? Let's take a neural network as an example. I'm used to having inputs and their associated outputs when training a NN with standard packages. But the problem is, if I want to model the derivative, I don't have the derivative as an output to train on. I only know the function values and have only been able to compute a error to minimize on through the forward Euler integration. Of course I could treat the NN parameters as I did with the parameters above, and do a brute force minimization on them, but I obviously lose a lot of the efficiency of backprop in doing so (or, if I was doing this with a decision tree, would lose the logic of how regression trees are normally optimized).

Is there a better way for me to do this? Are there methods that exist to handle this problem? If anyone could point me in a fruitful direction it would be much appreciated.

Answer 1

This is actually really easy to implement in any deep learning framework with automatic differentiation capability.

You need two neural networks. One to approximate the $X$, another to approximate the $dX/dt$.

• The first step is easy. You have samples of input and target for $X$. Just follow the normal process to train you network to learn $X$
• For the second step is that you don't have the target values (i.e. gradient) directly. The trick is that you already got an approximator for $X$ from the first step, so use that to get your target for the second network.

I am most familiar with pytorch, so I will use it as an example.

import torch
from torch.nn import Sequential, Linear, ReLU
from torch.autograd import Variable
from torch.optim import Adam
import numpy as np

x = np.linspace(0, 10, 101).reshape(-1, 1)
y = np.sin(x) # Using sin as an example

x = Variable(torch.Tensor(x), requires_grad=True) 
y = Variable(torch.Tensor(y), requires_grad=False)

f = Sequential(....) # Build f approximator
f_optimizer = Adam(f.parameters())

for epoch in range(1000):
    loss= f(x) - y
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

# Build grad approximator
f_grad = Sequential(....) 

# need a new optimizer to train f_grad
f_optimizer = Adam(f_grad.parameters()) 

for epoch in range(1000):
    # Directly backprop from the output to the gradient of the f
    model(x).backward() 
    loss= f_grad(x) - x.grad # x.grad is the grad of f w.r.t x
    f_optimizer.zero_grad()
    loss.backward()
    f_optimizer.step()  

f_grad # this is the gradient function you are looking for

(I don't have a machine with pytorch install at the moment, so don't expect the code to run with some debugging)

Answer 2

Honestly, this does not sound like a machine learning problem. I can think of two approaches:

• If you have an analytic model for $dX/dt$, you can integrate it over $t$ to obtain a model for $X$. Then find the parameters that minimize squared distance to the data (plus some regularization perhaps). A grid search might be sufficient.
• Use standard methods of interpolating a function, such as spline interpolation. The advantage of this is that you get a smooth closed-form solution.

Please also keep in mind that the derivative for a finite set of points $X(t_1), X(t_2), ..., X(t_n)$ does not exist. There are infinitely many functions that pass through all points (and almost all of them are not differentiable). In other words: You have to make additional assumptions on the function that you are looking for. With the methods above, you make these assumptions explicitly, which I think is better than burying them deep inside a machine learning algorithm.