1
$\begingroup$

With active learning I hope to keep the annotation effort to a minimum, yet building still a good classifier.

My initial starting point is that I have about 20k images which can belong to ten different classes, and have 0 labeled images at the moment. After each active learning iteration, I hope to get the labels of e.g. 100 images. If it matters, unfortunately, the data is very likely imbalanced which means that five classes are probably very rare.

So how do I construct my test set for active learing?

  1. Draw a random sample of a certain percentage right at the beginning, annotate it and keep the test set static throughout the whole project?

  2. Grow the test set with each active learning iteration? (example: 10 of the 100 new labeled images are randomly added to the growing test set?)

  3. Any other idea?

I was looking for this topic on Google and Google Scholar, but found no good hits regarding papers which elaborate on test set construction for active learning projects.

Any ideas, experiences or further readings welcome! Thank you!

Improve this question
$\endgroup$
2
  • $\begingroup$ By active learning, do you mean online learning? Usually in an online learning setup, there is no test set and the model trains and evaluates on live data and adjust its predictions as newer data becomes available. $\endgroup$
    – Jayaram Iyer
    Jun 5, 2021 at 9:33
  • $\begingroup$ @JayaramIyer By active learning, I mean a (supervised) setup in which labeled data is scarce and hardly any (or no) samples are labeled at the beginning. A so-called query strategy (QS) is responsible to select samples which are labeled next by an "oracle". This QS hopefully selects the most informative samples for the classifier to learn quickly with less labeled data. $\endgroup$
    – ziggyler
    Jun 6, 2021 at 20:14

1 Answer 1

Reset to default
0
$\begingroup$

I am working on how to apply active learning to testing. I understood that you have already a training dataset but you do not have a testing dataset and you would use active learning to label more samples for testing. You have two options:

Option 1: If the training dataset is large enough. Then, you can consider this dataset as your entire dataset and split it into (70%,30%) for training and testing. No need to use active learning to select testing samples. Split the dataset and train the model from scratch.

Option 2: If the training dataset is small, you might probably apply active learning to label more samples for testing. Which samples I should select for testing? the simple solution is to apply the same technique you used before in sampling the training dataset. Otherwise, if you would like to apply more special criteria for sampling e.g., representative, diverse, or hard samples, you could read this paper and implement the algorithms over there.

An Active Learning Approach with Uncertainty, Representativeness, and Diversity

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you for your answer and sorry for the late reply. It would need to be option 2 for sure. I have very few labeled samples to start with. So how would option 2 look like in practice? E.g. in each iteration, you let the query (selection) strategy select 8 samples for training, and 2 samples for test? --> retrain model --> query selection strategy selects 8 for training, 2 for test --> retrain model --> ... $\endgroup$
    – ziggyler
    Jul 13, 2021 at 20:55
  • $\begingroup$ You could label your training dataset until you are satisfied with the performance of the model using .e.g., marginal sampling. Then, retrain your model from scratch with the training dataset you labeled. Then, you can label new data (namely testing data) based on the trained model by using e.g., marginal sampling, you will find your model to select unseen patterns during labeling the training dataset. I do not recommend labeling your test dataset along the way with training as you suggested. Typically, the testing dataset should be more challenging for the model to test. $\endgroup$
    – user119783
    Jul 14, 2021 at 17:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.