Say I'm attempting to improve click-through rates on videos on my website. I've been reading the literature on contextual bandits and came across the Microsoft MWT white paper. I believe this is the right method for this case. However, I'm a bit confused on the details of the policy exploration and policy training (Section 3.2 and 3.3).

My first question: Is there a difference between a policy and exploration policy? I want to say no, but want confirmation.

My second question pertains to learning a policy. As an example, say that I initially used an $\epsilon$-greedy exploration policy and collected user click-log data. The paper states that (offline) I want to find a policy out of all allowed policies that approximately maximizes the estimated expected reward (Inverse Propensity Scoring estimator) $\mu_{ips}(\pi)$ for policy $\pi$ -- Eqn. 4 in Section 3.3. The policy chooses an action $a\in A$ (e.g. a video to serve to the user) given a context $x$ (e.g. user attributes and perhaps attributes of the article/video currently being watched) and an outcome (click or no-click) is observed. The authors propose to reduce the policy training to a cost-sensitive classification problem (e.g. logistic regression, decision trees, or neural nets) where each policy $\pi$ is viewed as a classifier (i.e. for context $x_{i}$, a given policy $\pi$ chooses action $a_{i}$).

However, it's not clear to me what the classification task is. Does it mean that I can use something like a neural net whose output is the policy that minimizes the cost? Is this already performed in ML packages like VW? If so, how?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.