11
$\begingroup$

What I mean is the following: Instead of processing all the training data at once and calculating a model, we process one data point at a time and update the model directly afterwards.

I have seen the terms "on-line (or online) learning" and "incremental learning" for this. Is there a subtle difference? Is one term used more frequently? Or does it depend on the research community?

Edit: The Bishop book (Pattern Recognition and Machine Learning) uses the terms on-line learning and sequential learning as synonyms but does not mention incremental learning.

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Interesting, I haven't read the Bishop book, but I've heard online learning and sequential learning treated as different concepts. I understand online learning to mean a learning problem in which a model is able to use new incoming data as a way to "grow" the training set and update the model dynamically. I've heard sequential learning applied to problems where the goal is to predict a sequence instead of individual instances, as in predicting part-of-speech sequences in POS tagging. $\endgroup$
    – kylerthecreator
    Jun 29, 2015 at 19:22
  • 1
    $\begingroup$ To add to the confusion with yet another phrase scikit-learn uses "partial fit". $\endgroup$
    – Finn Årup Nielsen
    Oct 9, 2015 at 13:59

3 Answers 3

Reset to default
7
$\begingroup$

So, the following answer is just based on different opinions of collegues and professors from the field. I want to try to summarize it briefly:

Sequential and online learning is mostly associated with Bayesian updating. The sequential learning is used widely for an order in time of the data, meaning that $x_1$ is coming always first, then $x_2$, then $x_3$ and so on. The Dataset each has a certain order in that sense. In contrast to that incremental may be a whole block of data at time x and another block of data at time y. While the block internally may be randomly ordered.

Concerning online learning, the people mostly referred to a data stream, hence a online learning is always incremental learning but incremental learning does not have to be online.

These are quite fuzzy definitions, and in my opinion there is not clear definition though. I still hope that helps.

Improve this answer
$\endgroup$
4
$\begingroup$

Although the definitions of the two concepts are fuzzy, there is a slight difference between online learning and incremental learning approaches.

In online learning approach, model is updated to adapt to the new data. It is possible that the model can forget the previously learned inferences which is called as Catastrophic Interference.

Whereas in the incremental approach, even as the model is updated, previous inferences are not forgotten.

Improve this answer
$\endgroup$
1
$\begingroup$

According to this article, both of the learning(Online and incremental) methods aim at learning (updating) a model when the data comes on the fly to obtain the same model as the one learned in a batch setting (i.e. on static data). The difference is that on-line learning learns a model when the training instances arrive sequentially one by one (1-by-1), whereas incremental learning updates a model when a new batch of data instances arrive. The comparisons between on-line learning and incremental learning are listed in Table 1. It should be noted that, all the existing on-line learning frameworks can be used for incremental learning as well, because on-line learning algorithms can process the batch of new data 1-by-1.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.