Suppose we have trained a regression model $M$ on a fixed set of $n$ features, $F_1,F_2,…,F_n$ on a particular dataset $G$. Now assume that after model training, additional features ($F_{n+1},…$) become available for a subset $H\subset G$.

What would be the best way incorporate these features to improve predictions on the subset $H$?

I can think of a few possible solutions:

  • Train a new model $N$ on the dataset $G$ where the new features are null in $G \setminus H$. This could be useful when $|H| \ll |G|$.
  • Train a new model $N$ on the dataset $H$, disregarding the old model and the (useful) training data $G \setminus H$.
  • Train a submodel $M'$, using $M$ as a starting point, on the new features ($F_{n+1},…$). This has the advantage of not throwing away the useful training done beforehand.

To me the last option seems like the best solution. Unfortunately, I cannot find any literature doing this sort of thing. Does that depend on the type of model? Or is it better to train a new model?


That is commonly called incremental or online machine learning.

Whether to train a new model or augment an existing model is an empirical question. It is often a function of model size (augmenting uses less computational resource) and the value of new data (augmenting increases the weight of new data, whereas retraining weights all data equally).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.