1
$\begingroup$

Let's say that we have 10,000 unlabeled documents, and we want to use pool-based sampling with batch size of 5. And we will use least confidence (LC) for querying strategy.

Is this outline of active learning reasonable:

1) Randomly select 30 documents, and get them labelled:=> labelled: 30, unlabelled: 9970.

2) Train the model on 30 labelled data

3) Use the model to make predictions for 9970 unlabelled documents

4) Use LC to select 5 out of all predicted documents that the model is least certain about

5) Get labels for the 5 documents:=> labelled: 35, unlabelled: 9965

6) Retrain the model on labelled data

7) Keep doing steps 2 - 6 until, the labeling resource expires or other stopping criteria is met.

Is this correct? I am mostly concerned about step 3) Should we retrain on all the data on every iteration? It seems like it could get computationally slow, but on the other hand if we don't evaluate every document how would we know which one is the most helpful to get labelled?

$\endgroup$
1
$\begingroup$

Your flow is correct. Model is retrained on new labeled data. Otherwise, the next candidates for labeling will be selected from the same region as previous candidates on which the model is least certain. By training on new labeled data, model will move on to new regions about which is least certain. Here is the diagram for pool-based method from page 5 of this survey on active learning:

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.