# Reinforcement learning using univalent and multivalent heterogeneous features

## Problem introduction

I have a game in which human players play levels (just like the famous casual game candy crush) where each level has its own properties and its own difficulty. In said game, the only variable that I, myself, have control over is the level's difficulty. I also have means to empirically measure some metrics that reflect the engagement of a player based on the findings of 'Quantifying human engagement'.

## Problem formulation

Based on the data that I collect from the players, I am able to reproduce the above-cited paper's findings by empirically computing a player's engagement from my dataset as follows :

$$\bar{t_a}(n) = \frac{\bar{t_p^{emp}}(n)}{p_c(n)} = \text{Reward}$$

Where $$\bar{t_p^{emp}}(n)$$ is just the average number of attempts needed by players that passed level $$n$$ to pass it and $$p_c(n)$$ is the churn probability defined by the total number of players that abandoned at level $$n$$ divided by the total number of players that played level $$n$$.

The set of actions that reflect the difficulty of a level which I have control over is represented as :

$$\text{Action} \in \{A,B,C\}$$

where each of the three options $$A$$, $$B$$ or $$C$$ represent a degree of difficulty. I also have the option of making difficulty actions randomly at first in order to learn a player's preferences in real time and its impact on the above-defined reward. Based on my findings so far, the actions have an impact on the reward.

## Data collected

I collect data that represent features for thousands of players, each of which is given a unique $$id$$, in the following form :

• Univalent features (one value per row, can be numerical or categorical)
• Multivalent features (short time series have a fixed length of $$k$$ elements in the past)

An example of my data looks like this :

ID        X1               X2              X3          Reward      Action
1    [1,2,3,1,..]    [True, False, ...]    0.33        0.33         A
1    [5,1,2,3,..]    [True, True, ...]     0.6         0.5          B
2    [0.5,1,..]      [False, False,...]    0.6         0.5          C


I can have different numbers of observations for different $$id$$'s (so for example, $$12$$ observations for $$id_1$$ and $$50$$ for $$id_2$$) But each row can be a standalone learning instance that contain the full historical data for the same $$id$$ of course.

## Modeling approach

I would like to find a modeling approach that returns one of the three actions available in order to maximize the reward (human engagement) based on what has been given in the past. This makes me think that it is a reinforcement learning problem. However, I have yet to find any resources or tutorials on how to preprocess such data nor did I find any models that handle such cases.

The only paper that I came across that discusses this problem is deep learning for youtube recommendation where the authors talk about univalent and multivalent features (so tabular data and time series data mixed together, same as my case) then they use an embedding layer to transform the feature space into a single format input for a deep neural network. However, the authors do not share any code to reproduce their preprocessing techniques.

Any thoughts? Thanks!

• Can you elaborate on how the reward is computed? I mean, how does the reward value depend on your prediction? Jun 28 at 15:29
• Also, you really need to give some context. What is the actual problem? Are the answers ground truth answers? Here it seems that there is a mixture of a prediction problem and an optimization problem. If you could elaborate more on where do these data come from, do you have access to a simulator/environment and what the actual problem is we will be able to help you or point you to the relevant literature. Jun 29 at 4:06
• Thank you both for your questions. I have edited the question to include more context. In short, no I do not have access to a simulator but I do have an environment where I can give one of the three options randomly for each player. So the answer column is not ground truth since it can be given randomly at first in order to learn a player's preferences. Some sequence of answers given may lead to a low reward and others to a higher reward. The reward is computed empirically from the data and the aim is to give answers for each player to maximize the reward in the long run! I hope it helps. Jun 29 at 7:57
• So you want an agent that will output an action (A, B, C) which modulates the difficulty of a game that a human plays in order for the human to perform maximally at the game. Or, the "reward" is not an actual performance measure of how the human player performed but instead a pre-calculated quantity that depends on the action sequence the human generated while playing the game? Jun 29 at 19:03
• Yes exactly. I would like an agent that would take one of the three pre-defined actions to modulate difficulty in the game for human players in order to maximize engagement. The reward reflects an engagement that can be computed empirically from the data using the number of attempts and empirical churn for each level. Said action should, in theory, have an impact on this reward. Thank you for the advice, I will re-edit the post and introduce the problem first! Jun 30 at 7:49