Is this approach for Active Learning correct?

Question

Let's say that we have 10,000 unlabeled documents, and we want to use pool-based sampling with batch size of 5. And we will use least confidence (LC) for querying strategy.

Is this outline of active learning reasonable:

1) Randomly select 30 documents, and get them labelled:=> labelled: 30, unlabelled: 9970.

2) Train the model on 30 labelled data

3) Use the model to make predictions for 9970 unlabelled documents

4) Use LC to select 5 out of all predicted documents that the model is least certain about

5) Get labels for the 5 documents:=> labelled: 35, unlabelled: 9965

6) Retrain the model on labelled data

7) Keep doing steps 2 - 6 until, the labeling resource expires or other stopping criteria is met.

Is this correct? I am mostly concerned about step 3) Should we retrain on all the data on every iteration? It seems like it could get computationally slow, but on the other hand if we don't evaluate every document how would we know which one is the most helpful to get labelled?

Answer 1

Your flow is correct. Model is retrained on new labeled data. Otherwise, the next candidates for labeling will be selected from the same region as previous candidates on which the model is least certain. By training on new labeled data, model will move on to new regions about which is least certain. Here is the diagram for pool-based method from page 5 of this survey on active learning: