# Given historical states and action/reward data, is reinforcement learning a reasonable approach?

To summarize my problem:

• I want to maximize my total reward over all timesteps
• I have 3 discrete actions at each time step.
• The state vector for each time step has 5 features. The features are numeric and continuous.
• The rewards for each action change at each timestep.
• The reward values are not known before the action is made.
• But we will know what the rewards would have been for all actions before the next action needs to be taken. (I believe this means we can do on-policy training).
• Edit: Selecting an action, shouldn't affect the next state or the future rewards.
• I believe that selecting the best action is quite hard.
• In my experiments, it is hard to beat a very simple strategy of picking an action based on some threshold of the first state/input feature, i.e;

$$\text{action}_t = \begin{cases} 1 & \text{if } x_{1,t} < 0.1 \\ 2 & 0.1 \leq x_{1,t} < 2.3 \\ 3 & x_{1,t} \geq 2.3 \end{cases}$$

Here, $$x_1$$ is probably the most important feature for deciding which action is superior. The other features are important for estimating the size of the reward.

I think it is very hard to predict the best action in any individual step. I want to find a (simple) policy that has good performance over many steps in reward space, not classification rate/accuracy space.

I have N>10,000 historical data points. My state data is a numpy array with shape (N, 5) and looks like this:

array([[ 0.091     , -0.2189    ,  0.97724431,  3.908     ,  3.999     ],
[ 0.22      ,  0.091     ,  0.94358974,  3.68      ,  3.9       ],
[-0.22      ,  0.22      ,  1.05978261,  3.9       ,  3.68      ],
...,
[ 0.34      ,  0.3       ,  0.96617286,  9.7111    , 10.0511    ],
[-0.6876    ,  0.34      ,  1.06875175, 10.6888    , 10.0012    ],
[ 0.3988    , -0.6876    ,  0.96051485,  9.7012    , 10.1       ]])

My reward data is an array with shape (N, 3) and looks like this:

array([[ 35055,  35631,  27413],
[ 37475,  36726,  24968],
[ 36908,  36315,  24974],
...,
[112152, 109845,  93747],
[101289, 102263,  94943],
[ 96325,  97717,  91522]])

I have tried posing this as a regular multiclass classification problem and whilst it may achieve better classification performance than a simple lag-1 strategy, it doesn't achieve as good cumulative reward. I believe this is because of the non-linearity between the (multiclass) logloss/accuracy metrics that the models optimise for and the actual reward space which is what we actually care about (I think Taleb describes this well here). I have also tried posting this as a multi output regression problem where I try to predict the reward value of each of the actions. This doesn't work that well either, it has no concept of maximising reward over time instead of each action. It fails to take the uncertainty of the estimates/predicted rewards into account.

I have constructed a gym using the Gymnasium Python library that captures the problem.

class CustomEnv(gym.Env):
def __init__(self, state_data, reward_data):
super(CustomEnv, self).__init__()

# Define the action and observation space
self.action_space = spaces.Discrete(3)
self.observation_space = spaces.Box(low=-np.inf, high=np.inf, shape=(5,), dtype=np.float32)

# Store the state and reward data
self.state_data = state_data
self.reward_data = reward_data

# Ensure the state and reward data have the same length
assert self.state_data.shape[0] == self.reward_data.shape[0]

# Initialize the current step
self.current_step = 0

def step(self, action):
# Ensure action is valid
assert self.action_space.contains(action)

# Get the reward for the action at the current step
reward = self.reward_data[self.current_step, action]

# Move to the next step
self.current_step += 1

# Check if the episode is done
done = self.current_step == self.state_data.shape[0]

# Get the next state
next_state = self.state_data[self.current_step] if not done else np.zeros(self.state_data.shape[1])

return next_state, reward, done, {}

def reset(self):
# Reset the environment to the beginning
self.current_step = 0
return self.state_data[self.current_step]

def render(self, mode='human', close=False):
# This environment doesn't have a visual representation, but you could add one if you want.
pass

My questions: I believe that that must be a better, more systematic way to find a 'good policy' than the ad-hoc policy I can come up with myself and described above...

• Is reinforcement learning the best way to approach this problem?
• If so, what methods or algorithms would be the best to try first?
• As far as I can tell, tabular Q learning won't work because I have continuous inputs.
• Given the noisy reward behaviour, and from my own experiments, I imagine a very simple model would work best.
• Have I set up my gym environment correctly?
• What other tools/methods should be considered? The simpler the better!