Can we use prior information to improve the results of a neural network? If so, how do we incorporate it into the learning process?

Question

Let $f_\alpha(x)$ be the function $$ f_\alpha(x) = x^2 + \alpha\sin(x), $$ on the interval $[-5,5]$. Suppose $\alpha = 2$, and our goal is to learn the function $f_2(x)$ using some form of neural network. $f_2$ looks like this:

We are given a set of noisy measurements of $f_2$ from which we would like to learn $f_2$. Specifically, we are given a set of random samples $\{y_i,x_i\}_{i=1}^n$ with $$ y_i = f_2(x_i) + N(0,\sigma), $$ where $N(0,\sigma)$ is normally distributed noise with standard deviation $\sigma$:

Now we can use the samples $\{y_i,x_i\}_{i=1}^n$ to estimate the unknown function $f_2$. However, before doing that, note that the general behavior of $f_2$ is captured well by the function $f_0(x) = x^2$:

Now suppose that instead of just being given the set of noisy samples $\{y_i,x_i\}_{i=1}^n$ of $f_2$ we are given both the noisy samples and the function $f_0$ which we are told is a decent approximation of $f_2$. In other words, we have some prior knowledge on what the function $f_2$ generally looks like.

Can we somehow incorporate this prior knowledge of $f_2$ into our neural network learning process so that we can get a better estimate of $f_2$ than estimating it based on just the noisy samples on their own?

If so, what are our options for incorporating this knowledge into a neural network? Does the type of neural network (CNN, RNN, etc...) affect the way we incorporate the prior information?

P.S. I am coming from statistics/mathematics and while I understand the general principles of neural networks I have only just started using them.

P.P.S. Here is the Matlab code for the images

rng(123);

N = 100;
x = linspace(-5,5,N);
alpha = 2;
sigma = 4;

f_0 = x.^2;
f_alpha = x.^2 + alpha*sin(5*x);
f_sigma = f_alpha + sigma*randn(N,1).';

figure
hold on, grid on
plot(x,f_0,'k--')
plot(x,f_alpha,'b')
plot(x,f_sigma,'r.')

Answer 1

What you may be referring to is transfer learning.

The process of transfer learning mainly boils down to using layers of big pre-trained models (super set) in your smaller model (subset) during training.

Additional configuration may include freezing some of the pretrained layers (no updates during training) and unfreezing others (update weights/biases during backprop).

It is very common in recurrent networks to use pre-trained word embedding layers (word2vec, GloVe).

Answer 2

I haven't started with NN yet but is there an Ensemble model you can put together? I know sklearn has an Ensemble model AdaBoost and GradientBoost. Is that possible with NN?