I am facing a problem where I want to use active learning to improve my classifier. Basically, I can choose data from one (and only one) data set among a set of candidate data sets. The question is which one to choose?

In other words, given a set of candidate data sets that I can use improve my classification model, which one is going to improve the model most? Can I use the some metric (eg., average) from the objective functions of the batches? Do I need to normalise the objectives across all data sets? Have metrics to infer the magnitude of model improvement been proposed?

At this stage, my objective function looks like:

a  Uncertainty + (1-a) Diversity

where a is a weight factor, Uncertainty is the uncertainty in the model prediction for a given data point and Diversity is a measure of distance between the data point and the training population.

Any help would be greatly appreciated.




Your question is actually the whole point of active learning. You probably need to read about existing approaches in active learning in order to find the one which suits your needs.

I'm not up to date at all on the topic but a traditional approach was to train several models on available data, make them predict on all the unannotated instances then use majority voting: instances for which the models tend to agree are "easy" to predict, whereas those for which models make different predictions are "hard" so potentially more valuable to improve performance.

  • $\begingroup$ Thanks, @Erwan. I think my question differs to conventional active learning approaches where there is only one pool of data. In my case, I have a set of potential pools where I can draw samples from but I can only pick one pool. $\endgroup$ – WAF Aug 18 at 5:09
  • $\begingroup$ What do these pools represent?, different sources of data maybe? I assume that every pool contains the same kind of instances right? $\endgroup$ – Erwan Aug 18 at 9:14
  • $\begingroup$ Yes, same kind of data. $\endgroup$ – WAF Aug 18 at 11:08
  • $\begingroup$ Then I don't really see why you need to select a particular pool, you could try using all the instances together. Anyway you can use the method I mentioned for each pool, then select the pool for which the classifiers disagree the most (you could use inter-annotator agreement measures for instance) $\endgroup$ – Erwan Aug 18 at 22:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.