Neural net layer that preserves spatial information

Question

I'm attempting to use a neural network as a kind of interpolator for a high-dimensional function.

We're doing this to circumvent the need for a physical model that calculates this function exactly, but is expensive to run. We have a dataset of past runs of this physical model that we can now use as the basis for interpolation.

The input is a set of 28 parameter values.

The output we're trying to estimate consists of the shape of 432 periodic functions (i.e. value of 432 different functions along the circumference of a circle). We know that the output is uniquely determined by the 28 input values; there's no noise.

For example, here is one of these periodic functions. For this example, it's only sampled with 12 points (though in production we will be sampling it more finely, perhaps 36 points around a circle).

My current neural network Keras implementation is:

• Input layer: 28 units (one per input parameter)
• One hidden layer: fully connected, 1000 units
• Output layer: fully connected, 432 periodic functions × 12 samples = 5184 units

As illustrated above, this works okay (even before optimizing hyperparameters).

But I'm concerned that with these fully-connected layers, I'm treating the 5184 outputs as unrelated numbers, completely disjointed from one another. In reality, there's some spatial relationship between the 12 samples describing each of the 432 underlying functions in the output. But I'm just throwing this spatial information away.

Is there another kind of layer which could take advantage of this spatial relationship, and thus lend itself better to this kind of regression?

If I were going the other way round, i.e. from spatial information to distinct values, I would consider convolution layers (perhaps with a periodic boundary). But what about this reverse case?

Answer 1

Rather than trying to predict each value explicitly, you could pick a parametric function that you think might be expressive enough to describe your target function (for example, a polynomial of order 6), and get your network to learn the parameters of that polynomial that give you a minimum loss. This is basically polynomial regression. Since your function is periodic you might use a more cycle-based basis set, like linear combinations of sine waves.

For example:

1) Input data to the network 2) output a set of values that you treat as parameters to your function 3) for each target point, compare it to the predicted value from your function 4) calculate a loss (like mean squared error) based on the difference between the function predictions and the targets.

Using a function will (1) restrict the space of possible outputs, and (2) make nearby outputs be self-similar.