I am facing a problem where I want to use active learning to improve my classifier. Basically, I can choose data from one (and only one) data set among a set of candidate data sets. The question is which one to choose?

In other words, given a set of candidate data sets that I can use improve my classification model, which one is going to improve the model most? Can I use the some metric (eg., average) from the objective functions of the batches? Do I need to normalise the objectives across all data sets? Have metrics to infer the magnitude of model improvement been proposed?

At this stage, my objective function looks like:

a  Uncertainty + (1-a) Diversity

where a is a weight factor, Uncertainty is the uncertainty in the model prediction for a given data point and Diversity is a measure of distance between the data point and the training population.

Any help would be greatly appreciated.




Your question is actually the whole point of active learning. You probably need to read about existing approaches in active learning in order to find the one which suits your needs.

I'm not up to date at all on the topic but a traditional approach was to train several models on available data, make them predict on all the unannotated instances then use majority voting: instances for which the models tend to agree are "easy" to predict, whereas those for which models make different predictions are "hard" so potentially more valuable to improve performance.

  • $\begingroup$ Thanks, @Erwan. I think my question differs to conventional active learning approaches where there is only one pool of data. In my case, I have a set of potential pools where I can draw samples from but I can only pick one pool. $\endgroup$ – WAF Aug 18 '19 at 5:09
  • $\begingroup$ What do these pools represent?, different sources of data maybe? I assume that every pool contains the same kind of instances right? $\endgroup$ – Erwan Aug 18 '19 at 9:14
  • $\begingroup$ Yes, same kind of data. $\endgroup$ – WAF Aug 18 '19 at 11:08
  • $\begingroup$ Then I don't really see why you need to select a particular pool, you could try using all the instances together. Anyway you can use the method I mentioned for each pool, then select the pool for which the classifiers disagree the most (you could use inter-annotator agreement measures for instance) $\endgroup$ – Erwan Aug 18 '19 at 22:45

Uncertainty is fairly easy, if you have a probabilistic output. Just apply the model to unlabeled data sets and pick the one with highest average uncertainty. In the binary classification case, that's just lowest mean(abs(p - 0.5)). modAL (https://github.com/modAL-python/modAL) has some utilities that could be useful in the multi-class case where there are several possible definitions of 'uncertainty'. See https://modal-python.readthedocs.io/en/latest/content/query_strategies/uncertainty_sampling.html for example.

For diversity, I suppose you can measure the average distance or similarity between your train set and each data set. This assumes you have a meaningful distance or similarity metric. Up to you how to normalize them to make them comparable though.

Some models may already capture that points not near the training set are inherently less certain, in which case this might be redundant. Some may not (like max margin classifiers maybe), but if you suspect new datasets from different part of the input space behave differently to the training set, maybe those models aren't appropriate anyway.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.