Suppose you have a model that has been trained on $N$ data over $E$ epochs. This means that the model has seen each of the $N$ examples, $E$ times.
Now say you got $M$ more training data. Normally you would want to train the new ones for $E$ epochs as well.
However if $N$ and $M$ don't come from the same underlying distribution (or don't represent it adequately), this would result in the model "forgetting" the first $N$ examples and "paying more attention" to the latter $M$ ones.
You could try training your model for $<E$ epochs, so that it learns the latter but doesn't forget the former, but that is purely empirical and very hard to achieve in practice.
You can a few things to avoid this:
- Retrain your whole model using both the $N+M$ examples (which you would shuffle). This would require a new complete training of the model on regular occasions and would be increasingly difficult to train (due to the ever increasing size of the training data). This is a very inefficient solution and wouldn't work for any on-line training application
- Make use of a model that supports on-line training. Some algorithms support incremental (on-line) training, without you needing to retrain the whole thing. A scikit-learn comparison is available.
- Customize an algorithm so that is has the desired effect. For example, you could train a linear SVM incrementally, with a large regularization penalty and an SGD classifier. This is discussed in more detail for scikit-learn here.