2
$\begingroup$

Say I'm attempting to improve click-through rates on videos on my website. I've been reading the literature on contextual bandits and came across the Microsoft MWT white paper. I believe this is the right method for this case. However, I'm a bit confused on the details of the policy exploration and policy training (Section 3.2 and 3.3).

My first question: Is there a difference between a policy and exploration policy? I want to say no, but want confirmation.

My second question pertains to learning a policy. As an example, say that I initially used an $\epsilon$-greedy exploration policy and collected user click-log data. The paper states that (offline) I want to find a policy out of all allowed policies that approximately maximizes the estimated expected reward (Inverse Propensity Scoring estimator) $\mu_{ips}(\pi)$ for policy $\pi$ -- Eqn. 4 in Section 3.3. The policy chooses an action $a\in A$ (e.g. a video to serve to the user) given a context $x$ (e.g. user attributes and perhaps attributes of the article/video currently being watched) and an outcome (click or no-click) is observed. The authors propose to reduce the policy training to a cost-sensitive classification problem (e.g. logistic regression, decision trees, or neural nets) where each policy $\pi$ is viewed as a classifier (i.e. for context $x_{i}$, a given policy $\pi$ chooses action $a_{i}$).

However, it's not clear to me what the classification task is. Does it mean that I can use something like a neural net whose output is the policy that minimizes the cost? Is this already performed in ML packages like VW? If so, how?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.