Help me name my problem – Online Ensemble Realibration

Question

I have the following problem: k predictors (let's say A, B) . Each predicts a value and their confidence. Only one result must be selected. For each predictor, I have their past performance which is composed of their confidence scores, and also flag that indicates whether their prediction was correct or not. The (very simplified) training data could look something like:

predictor	prediction 1	prediction 2	prediction 3	prediction 4
predictor A	0.7, correct	0.6, correct	0.4, correct	0.3, correct
predictor B	0.8, correct	0.6, incorrect	0.2, incorrect	0.3, incorrect

Now, let's assume that there is an incoming prediction where A has confidence 0.6 and B has confidence 0.6. My goal is to decide which one to pick.

With each incoming sample, I would like to (online) update some model (it can be one model per member of ensemble) that would recalibrate the score of each member of the ensemble and to therefore enhance it with my knowledge of whether their prediction was correct or not.

How is my problem called? I feel like it's somewhat related to multi-arm bandit problem, but the setting is different. I have searched for something like "online learning ensemble calibration", but without much luck. Thanks.

Answer 1

This sounds most like a form of online learning where only some subset of modules are trained at each new datapoint. In your case, what you describe is similar to the weighing of updates in this paper (page 79) albeit that your "weighing" is "0" for all except one classifier.