# Batch-mode active learning: How to select the batch that will bring the largest model improvement

I am facing a problem where I want to use active learning to improve my classifier. Basically, I can choose data from one (and only one) data set among a set of candidate data sets. The question is which one to choose?

In other words, given a set of candidate data sets that I can use improve my classification model, which one is going to improve the model most? Can I use the some metric (eg., average) from the objective functions of the batches? Do I need to normalise the objectives across all data sets? Have metrics to infer the magnitude of model improvement been proposed?

At this stage, my objective function looks like:

a  Uncertainty + (1-a) Diversity


where a is a weight factor, Uncertainty is the uncertainty in the model prediction for a given data point and Diversity is a measure of distance between the data point and the training population.

Any help would be greatly appreciated.

Cheers

Franz

Your question is actually the whole point of active learning. You probably need to read about existing approaches in active learning in order to find the one which suits your needs.

I'm not up to date at all on the topic but a traditional approach was to train several models on available data, make them predict on all the unannotated instances then use majority voting: instances for which the models tend to agree are "easy" to predict, whereas those for which models make different predictions are "hard" so potentially more valuable to improve performance.

• Thanks, @Erwan. I think my question differs to conventional active learning approaches where there is only one pool of data. In my case, I have a set of potential pools where I can draw samples from but I can only pick one pool.
– WAF
Aug 18 '19 at 5:09
• What do these pools represent?, different sources of data maybe? I assume that every pool contains the same kind of instances right? Aug 18 '19 at 9:14
• Yes, same kind of data.
– WAF
Aug 18 '19 at 11:08
• Then I don't really see why you need to select a particular pool, you could try using all the instances together. Anyway you can use the method I mentioned for each pool, then select the pool for which the classifiers disagree the most (you could use inter-annotator agreement measures for instance) Aug 18 '19 at 22:45

Uncertainty is fairly easy, if you have a probabilistic output. Just apply the model to unlabeled data sets and pick the one with highest average uncertainty. In the binary classification case, that's just lowest mean(abs(p - 0.5)). modAL (https://github.com/modAL-python/modAL) has some utilities that could be useful in the multi-class case where there are several possible definitions of 'uncertainty'. See https://modal-python.readthedocs.io/en/latest/content/query_strategies/uncertainty_sampling.html for example.

For diversity, I suppose you can measure the average distance or similarity between your train set and each data set. This assumes you have a meaningful distance or similarity metric. Up to you how to normalize them to make them comparable though.

Some models may already capture that points not near the training set are inherently less certain, in which case this might be redundant. Some may not (like max margin classifiers maybe), but if you suspect new datasets from different part of the input space behave differently to the training set, maybe those models aren't appropriate anyway.