I have the following problem:
k predictors (let's say
B) . Each predicts a value and their confidence. Only one result must be selected. For each predictor, I have their past performance which is composed of their confidence scores, and also flag that indicates whether their prediction was correct or not. The (very simplified) training data could look something like:
|predictor||prediction 1||prediction 2||prediction 3||prediction 4|
|predictor A||0.7, correct||0.6, correct||0.4, correct||0.3, correct|
|predictor B||0.8, correct||0.6, incorrect||0.2, incorrect||0.3, incorrect|
Now, let's assume that there is an incoming prediction where
A has confidence
B has confidence
0.6. My goal is to decide which one to pick.
With each incoming sample, I would like to (online) update some model (it can be one model per member of ensemble) that would recalibrate the score of each member of the ensemble and to therefore enhance it with my knowledge of whether their prediction was correct or not.
How is my problem called? I feel like it's somewhat related to multi-arm bandit problem, but the setting is different. I have searched for something like "online learning ensemble calibration", but without much luck. Thanks.