With active learning I hope to keep the annotation effort to a minimum, yet building still a good classifier.
My initial starting point is that I have about 20k images which can belong to ten different classes, and have 0 labeled images at the moment. After each active learning iteration, I hope to get the labels of e.g. 100 images. If it matters, unfortunately, the data is very likely imbalanced which means that five classes are probably very rare.
So how do I construct my test set for active learing?
Draw a random sample of a certain percentage right at the beginning, annotate it and keep the test set static throughout the whole project?
Grow the test set with each active learning iteration? (example: 10 of the 100 new labeled images are randomly added to the growing test set?)
Any other idea?
I was looking for this topic on Google and Google Scholar, but found no good hits regarding papers which elaborate on test set construction for active learning projects.
Any ideas, experiences or further readings welcome! Thank you!