Let's say that we have 10,000 unlabeled documents, and we want to use pool-based sampling with batch size of 5. And we will use least confidence (LC) for querying strategy.
Is this outline of active learning reasonable:
1) Randomly select 30 documents, and get them labelled:=> labelled: 30, unlabelled: 9970.
2) Train the model on 30 labelled data
3) Use the model to make predictions for 9970 unlabelled documents
4) Use LC to select 5 out of all predicted documents that the model is least certain about
5) Get labels for the 5 documents:=> labelled: 35, unlabelled: 9965
6) Retrain the model on labelled data
7) Keep doing steps 2 - 6 until, the labeling resource expires or other stopping criteria is met.
Is this correct? I am mostly concerned about step 3) Should we retrain on all the data on every iteration? It seems like it could get computationally slow, but on the other hand if we don't evaluate every document how would we know which one is the most helpful to get labelled?