How good are LSTMs in generalizing when learning curves?

Question

I'm interested in the following scenario: I want to learn a mapping that maps a function to another function, i.e. I want to approximate a functional operator. If one is unfimiliar with operators one can look up functional analysis but basically it is a mapping like this $A:f\mapsto g$ where $f$ and $g$ are for example continuous functions.

Preferably I want to learn $A$ by using a neural network. Consequently I came across neural operators which seem well suited. Now for data generation I asked myself how I can generate randomly functions $f$ such that my learned neural network is prepared for upcoming new input functions, i.e. I want to know how well it extrapolates, in a sense of predicting on unseen functions. The following paper https://arxiv.org/pdf/2212.06347 deals exactly with this question.

But now I wonder: I am also able to learn an operator with a LSTM, especially when the functions $f$ and $g$ are time-series. How well do LSTMs extrapolate in the above sense? Is it enough to generate only extreme cases of functions for the LSTM to be able to extrapolate?

I know that I may not have an universal approximation theorem regarding operators and LSTMs but I always thought that one is also able to learn operators with LSTMs but mabye there is my misconception.

Answer 1

LSTMs in an encoder-decoder arrangement would ingest $f$, render an encoding, and then decode that into a new sequence $g$. LSTMs are prone to overfitting, especially for small datasets, so I don't think training it with a few extreme samples will be effective. If the LSTM overfits and regularisation doesn't help, you could try simpler GRU cells.

It might help to know roughly what size the dataset would be, and the sequence lengths for $f$ and $g$ (including whether or not they're the same for all samples).

I think you'd need to train a model in order to assess whether its mappings are adequate for your needs given the available data and target performance.