2
$\begingroup$

Let $f_\alpha(x)$ be the function $$ f_\alpha(x) = x^2 + \alpha\sin(x), $$ on the interval $[-5,5]$. Suppose $\alpha = 2$, and our goal is to learn the function $f_2(x)$ using some form of neural network. $f_2$ looks like this:

enter image description here

We are given a set of noisy measurements of $f_2$ from which we would like to learn $f_2$. Specifically, we are given a set of random samples $\{y_i,x_i\}_{i=1}^n$ with $$ y_i = f_2(x_i) + N(0,\sigma), $$ where $N(0,\sigma)$ is normally distributed noise with standard deviation $\sigma$:

enter image description here

Now we can use the samples $\{y_i,x_i\}_{i=1}^n$ to estimate the unknown function $f_2$. However, before doing that, note that the general behavior of $f_2$ is captured well by the function $f_0(x) = x^2$:

enter image description here

Now suppose that instead of just being given the set of noisy samples $\{y_i,x_i\}_{i=1}^n$ of $f_2$ we are given both the noisy samples and the function $f_0$ which we are told is a decent approximation of $f_2$. In other words, we have some prior knowledge on what the function $f_2$ generally looks like.

Can we somehow incorporate this prior knowledge of $f_2$ into our neural network learning process so that we can get a better estimate of $f_2$ than estimating it based on just the noisy samples on their own?

If so, what are our options for incorporating this knowledge into a neural network? Does the type of neural network (CNN, RNN, etc...) affect the way we incorporate the prior information?

P.S. I am coming from statistics/mathematics and while I understand the general principles of neural networks I have only just started using them.

P.P.S. Here is the Matlab code for the images

rng(123);

N = 100;
x = linspace(-5,5,N);
alpha = 2;
sigma = 4;

f_0 = x.^2;
f_alpha = x.^2 + alpha*sin(5*x);
f_sigma = f_alpha + sigma*randn(N,1).';

figure
hold on, grid on
plot(x,f_0,'k--')
plot(x,f_alpha,'b')
plot(x,f_sigma,'r.')
$\endgroup$
1
$\begingroup$

What you may be referring to is transfer learning.

The process of transfer learning mainly boils down to using layers of big pre-trained models (super set) in your smaller model (subset) during training.

Additional configuration may include freezing some of the pretrained layers (no updates during training) and unfreezing others (update weights/biases during backprop).

It is very common in recurrent networks to use pre-trained word embedding layers (word2vec, GloVe).

$\endgroup$
0
$\begingroup$

I haven't started with NN yet but is there an Ensemble model you can put together? I know sklearn has an Ensemble model AdaBoost and GradientBoost. Is that possible with NN?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.