I read through the Internet and found this:

Most of Continuous Learning studies focus on a Multi-Task scenario, where the same model is required to learn incrementally a number of isolated tasks without forgetting the previous ones.

And they explain PermutedMNIST is Multi-Task. I have no idea why?

In other words,

Single-Incremental-Task considers a single task which is incremental in nature. In other words, we still add new classes sequentially but the classification problem is unique and when computing accuracy we need to distinguish among all the classes encountered so far.

However, I still can not clarify two these definitions? Could someone help me out and give me example of Multi-Task scenario and Single-Incremental-Task scenario.

TL;DR/Summary: The classes ($$y1$$, $$y2$$, $$y3$$ below) in multi-task can be anything (it may be that $$y1 \cap y2 = \emptyset$$, $$y1 \cap y2 = \emptyset$$, and so on). In Incremental we take the labels (and data from a common set, (i.e. $$y[:2] \subseteq y[:4] \subseteq y$$, by the definition of subsetting)

It is just a question of the interpretation of the definition. The definitions look very similar to each other but the devil's in the details.

Assuming that the model can distinguish the upper bond number of classes: For example, it is an ANN with $$N$$ neurons in the output layer and the number of classes ($$k$$) in the task with most classes is $$k <= N$$ (multi-task) or the total number of classes ($$k$$) is also $$k <= N$$ (single-incremental-task); we can say that:

Here we will train the model on different tasks over time, this is often called as reinforcement learning. In semi-python pseudo code (where .train already includes things like cross validation):

model = Whatever(...)

X1 = [[1, 0],
[2, 2],
[3, 0]]
y1 = [0, 1, 0]
model.train(X1, y1)

X2 = [[4, 4],
[5, 5],
[6, 0]]
y2 = [1, 2, 0]
model.train(X2, y2)

X3 = [[7, 0],
[8, 8],
[9, 9]]
y3 = [0, 1, 1]
model.train(X3, y3)
score = model.score(X3, y3)


Here the tasks may or may not be related. Or often are slightly related (e.g. identifying different types of objects in each training).

This is also training the model several times, here we have a single task in X but do not feed the entire dataset at once. In semi-python pseudo code:

model = Whatever(...)

X = [[1, 0],
[2, 2],
[3, 0],
[4, 4],
[5, 5],
[6, 0]]
y = [0, 1, 0, 3, 2, 0]

model.train(X[:2, :], y[:2])
score1 = model.score(X[:2, :], y[:2])
model.train(X[:4, :], y[:4])
score2 = model.score(X[:4, :], y[:4])
model.train(X, y)
score3 = model.score(X, y)


Here the task is one but it may be a big one. One place where this technique is used is to build a learning curve, which is one way of evaluating if we have enough data to understand the variation of the task.

Extra note: in the multitask case we said that $$y1 \cap y2 \cap y3 = \emptyset$$ could be (and most likely is) possible. One example would be: $$y1$$ are different models of cars and $$y2$$ are different models of ships. And the question is: Do understanding different models of cars help with differentiation different models of ships?

(P.S. ys will always be enumerated from 0 up to the number of classes, i.e. the numeric values of y will always be the same but their class meaning does not need to be).

• Although I upvoted your answer but I can not clearly understand the difference. Could you please summarize it briefly. You said  the devil's in the details but I do see any critical details. Sorry if its inconvenient. – Giang Nguyễn Jun 18 '19 at 13:22
• @GiangNguyễn - I gace a shot at making a summary with some extra notation. i.e. instead of plain code with some set operations. Plus some small changes to the values in the ys. Have a look if that makes more sense. – grochmal Jun 18 '19 at 13:35
• The Extra note really makes sense, now I know about multi-task case, how about single-incremental case, can you give one more clear example like in multi-task case. Thank you. – Giang Nguyễn Jun 18 '19 at 13:40
• Aha, now I read your comment again and may be I almost understand this. Thank you, I will read papers and confirm your comment in this post. – Giang Nguyễn Jun 18 '19 at 13:43
• By reading this: vlomonaco.github.io/core50/benchmarks.html, now I fully understand the difference, ty again! – Giang Nguyễn Jun 19 '19 at 6:04