How to incorporate new features in an existing machine learning model?

Question

Suppose we have trained a regression model $M$ on a fixed set of $n$ features, $F_1,F_2,…,F_n$ on a particular dataset $G$. Now assume that after model training, additional features ($F_{n+1},…$) become available for a subset $H\subset G$.

What would be the best way incorporate these features to improve predictions on the subset $H$?

I can think of a few possible solutions:

• Train a new model $N$ on the dataset $G$ where the new features are null in $G \setminus H$. This could be useful when $|H| \ll |G|$.
• Train a new model $N$ on the dataset $H$, disregarding the old model and the (useful) training data $G \setminus H$.
• Train a submodel $M'$, using $M$ as a starting point, on the new features ($F_{n+1},…$). This has the advantage of not throwing away the useful training done beforehand.

To me the last option seems like the best solution. Unfortunately, I cannot find any literature doing this sort of thing. Does that depend on the type of model? Or is it better to train a new model?

Answer 1

That is commonly called incremental or online machine learning.

Whether to train a new model or augment an existing model is an empirical question. It is often a function of model size (augmenting uses less computational resource) and the value of new data (augmenting increases the weight of new data, whereas retraining weights all data equally).

Answer 2

There are a few ways to incorporate new features into an existing machine learning model:

1. Train a new model: Training a new model from scratch on the entire dataset (including the new features) is one option. This can be a good choice if the new features are very different from the old features and you want to start from a clean slate.

2. Fine-tune the existing model: You can fine-tune the existing model by adding the new features to the input data and retraining the model on the expanded dataset. This can be a good option if the new features are related to the old features and you want to preserve the knowledge that the existing model has already learned.

3. Train a submodel: You can train a submodel on the new features and then combine it with the existing model. This can be a good option if the new features are related to the old features and you want to preserve the knowledge that the existing model has already learned, but you also want to allow the submodel to specialize on the new features.

Which approach is best will depend on the specifics of your problem and the characteristics of the new features. In general, it is usually a good idea to try a few different approaches and see which one works best for your problem.