I am looking for an algorithm that allows me the following:

I have a webpage and I want to randomly show one "content" from a list of contents on that webpage depending on who (visitor) sees the webpage. I know my visitors' demographic features like age, gender, locale. I assume that visitors with similar demographics have a similar taste regarding content on my website. I also know that they liked my content when they share it in the end.

What I have:

  • A list of available contents, let's say: red, green, blue, purple, violet
  • A constant stream of events of visitors with specific demographic data that share a content

What I want:

  • First of all, all contents should be displayed completely randomly. Each user should randomly get one content without any preference
  • As soon as the first user with a specific demographic shares the content they got I want other visitors with similar demographic features that end up on my webpage to see this specific content with a higher probability.

So basically I want a self optimizing system that learns in realtime.


This sounds like a classic use for a contextual bandit solver.

In essence you can run a simple online model (pretty much any regression model, or even a simple classifier like logistic regression if your reward signal is binary success/fail such as in your case) that learns to associate your demographic data with expected reward from each possible action - for you the reward can simply be 1 for a share link created or 0 for no share link.

Whilst the model is learning, you select the next action according to predicted reward from the model. There are choices between different workable strategies. For instance you could use an $\epsilon$-greedy approach: Pick the action with maximum predicted expected reward (or randomly choose between shared maximum values), but sometimes - with probability $\epsilon$ - you choose random content. There are other approaches and options that you can discover by researching contextual bandits and the simpler multi-armed bandit problems.

As an example, you could use a logistic regression model to predict expected reward from user demographics, with one such model per possible action. For a version that picks evenly to start, but prefers items that have been shared more over time, you can use a Boltzmann distribution (also called Gibbs distribution) using the predicted rewards as the inverse "energies" for the actions, and lowering the temperature as you collect more data. You can also initialise the weights of your model to predict a small but optimistic positive reward to start with to encourage early exploration. Whenever a user views your page, you pick the action to take based on the predicted rewards, and afterwards take the user response (share or not share) as feedback to update the one model associated with that action.

In the above example, the logistic regression learning rate, temperature scheme and starting reward are hyper-parameters of your model, and you use them to trade off responsiveness to individual events versus long-term accuracy for selecting the best action.

| improve this answer | |
  • $\begingroup$ Hey thanks so much for that detailed explanation and the efforts! Sounds like a good approach. Do you have experience with the implementation of such services? $\endgroup$ – Matthias Mar 17 '17 at 8:14
  • $\begingroup$ @Matthias: I have implemented context-free bandit solvers as part of studying reinforcement learning, which is a strongly related topic. I have no experience of doing this for real on a live service. $\endgroup$ – Neil Slater Mar 17 '17 at 9:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.