0
$\begingroup$

Which of the algorithms in the current literature for contextual bandits can be implemented for online learning and which ones can't? I'd really appreciate it if someone could provide a link to papers too! Thanks for the help!

$\endgroup$
  • 2
    $\begingroup$ I'm pretty sure all bandit algorithms are necessarily online learning algorithms $\endgroup$ – David Marx Jan 7 '18 at 21:55
  • $\begingroup$ I agree with David, generally the whole point of the bandit problem, and how it is framed in the literature, is to maximise reward (or minimise "regret") during an active learning process. "Offline contextual bandits" are essentially just a supervised learning/regression problem. $\endgroup$ – Neil Slater Jan 7 '18 at 22:02
  • $\begingroup$ So for example the epoch greedy algorithm mentioned here hunch.net/~jl/projects/interactive/sidebandits/bandit.pdf is online? The reason i ask is because the algorithm itself solves a supervised learning algorithm, so i'm interested if this is implementable practically? $\endgroup$ – Pavan Sangha Jan 8 '18 at 9:03
  • $\begingroup$ It does not look like epoch greedy is implemented e.g. in Vowpal Wabbit, but you can ask in an issue to the repo on github. Take note Vowpal Wabbit has a learning curve as usage is only sparsely documented. $\endgroup$ – matanster Jul 23 '18 at 8:53
  • $\begingroup$ just to note you typically prime an online model with a pre-prepared offline trained model, rather than unleash a fresh-and-naive untrained online model, reason being that you typically have little knowledge on whether your feature set is good enough for the model to fit your problem. Also because you can spare some aggregate real-world cost by unleashing a pre-trained model based on history, if you have any. Now replace 'typically' with 'sometimes' as it depends on the scenario and how confident you are regarding it e.g. from a-priori knowledge. $\endgroup$ – matanster Jul 23 '18 at 9:11
0
$\begingroup$

My answer can only be considered partial, I've not compiled a list, but I believe all algorithms implemented here, are, well, implemented for both offline and online mode. This one, can also be implemented for online mode.

Not trying to imply you should use that implementation, but this is kind of a living proof transcending deducing very analytically from articles. The thing to understand, is that certain CB algorithms are paired with rather benign algorithms for using offline-accumulated data for training them, in turn paired with mathematical proofs that the loss they incur in that offline training is a good predictor for the loss they'll incur in online mode (if the real world is still 'sufficiently similar' to the one logged from).

Some (other than mentioned above) algorithms may be only applicable to offline training, or at least I'm not aware of a theoretical refutation, that an algorithm may offline-train better in a way precluding direct use for online learning with the same algorithm. But many algorithms are encoded in software only for offline evaluations, as a lot of research dwells on offline. So I think it's a good question!

I think you should certainly email an author of any article that seems really helpful to you, to specifically ask them, if the article doesn't make that 100% clear, and they might even point you in rare cases at a solid online implementation! Do note online usage entails more production-readiness software considerations, and might be an extra mile in terms of the quality of the software expected as such ...

|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.