[Also see the cross post at stats SE: here
Posted there in hope of new answers, as it has a larger community]

I have a large dataset $D_1$.
I have a Feed-forward deep Neural Network $N$, with hidden layers, that I trained on $D_1$, using MSE Loss and standard back-propagation and obtained the parameters.

Now, I know the network architecture and the learned parameters.

Next, I obtained a new batch of data, $D_2$ - it is a large dataset similar to $D_1$.

I want to again learn the parameters of $N$, as though I had trained it from scratch on $D_1 \cup D_2$, but only by training on $D_2$ and using the earlier known parameters on $D_1$.

Is this some standard problem? How can I do this?

I can obtain basic info about $D_1$ such as size of the dataset; but training is expensive.


1 Answer 1


This sounds like potential concept drift as part of transfer learning, where you want to take some or all parts of a previous model on old data (in this case the parameters of $N$) and update those parts with new data.

After saving your model $N$, you can update the parameters in $N$ by fitting $N$ on $D_2$ and using a lower learning rate in the optimizer. The lower learning rate than in the your original model helps not significantly update the weights and prevents overfitting. You'll probably have to play around to get the right learning rate.


Not the answer you're looking for? Browse other questions tagged or ask your own question.