I am facing a problem where I want to use active learning to improve my classifier. Basically, I can choose data from one (and only one) data set among a set of candidate data sets. The question is which one to choose?
In other words, given a set of candidate data sets that I can use improve my classification model, which one is going to improve the model most? Can I use the some metric (eg., average) from the objective functions of the batches? Do I need to normalise the objectives across all data sets? Have metrics to infer the magnitude of model improvement been proposed?
At this stage, my objective function looks like:
a Uncertainty + (1-a) Diversity
a is a weight factor,
Uncertainty is the uncertainty in the model prediction for a given data point and
Diversity is a measure of distance between the data point and the training population.
Any help would be greatly appreciated.