Given historical states and action/reward data, is reinforcement learning a reasonable approach?

Question

To summarize my problem:

• I want to maximize my total reward over all timesteps
• I have 3 discrete actions at each time step.
• The state vector for each time step has 5 features. The features are numeric and continuous.
• The rewards for each action change at each timestep.
• The reward values are not known before the action is made.
  • But we will know what the rewards would have been for all actions before the next action needs to be taken. (I believe this means we can do on-policy training).
• Edit: Selecting an action, shouldn't affect the next state or the future rewards.
• I believe that selecting the best action is quite hard.
  • In my experiments, it is hard to beat a very simple strategy of picking an action based on some threshold of the first state/input feature, i.e;

$$ \text{action}_t = \begin{cases} 1 & \text{if } x_{1,t} < 0.1 \\ 2 & 0.1 \leq x_{1,t} < 2.3 \\ 3 & x_{1,t} \geq 2.3 \end{cases} $$

Here, $x_1$ is probably the most important feature for deciding which action is superior. The other features are important for estimating the size of the reward.

I think it is very hard to predict the best action in any individual step. I want to find a (simple) policy that has good performance over many steps in reward space, not classification rate/accuracy space.

I have N>10,000 historical data points. My state data is a numpy array with shape (N, 5) and looks like this:

array([[ 0.091     , -0.2189    ,  0.97724431,  3.908     ,  3.999     ],
       [ 0.22      ,  0.091     ,  0.94358974,  3.68      ,  3.9       ],
       [-0.22      ,  0.22      ,  1.05978261,  3.9       ,  3.68      ],
       ...,
       [ 0.34      ,  0.3       ,  0.96617286,  9.7111    , 10.0511    ],
       [-0.6876    ,  0.34      ,  1.06875175, 10.6888    , 10.0012    ],
       [ 0.3988    , -0.6876    ,  0.96051485,  9.7012    , 10.1       ]])

My reward data is an array with shape (N, 3) and looks like this:

array([[ 35055,  35631,  27413],
       [ 37475,  36726,  24968],
       [ 36908,  36315,  24974],
       ...,
       [112152, 109845,  93747],
       [101289, 102263,  94943],
       [ 96325,  97717,  91522]])

I have tried posing this as a regular multiclass classification problem and whilst it may achieve better classification performance than a simple lag-1 strategy, it doesn't achieve as good cumulative reward. I believe this is because of the non-linearity between the (multiclass) logloss/accuracy metrics that the models optimise for and the actual reward space which is what we actually care about (I think Taleb describes this well here). I have also tried posting this as a multi output regression problem where I try to predict the reward value of each of the actions. This doesn't work that well either, it has no concept of maximising reward over time instead of each action. It fails to take the uncertainty of the estimates/predicted rewards into account.

I have constructed a gym using the Gymnasium Python library that captures the problem.

class CustomEnv(gym.Env):
    def __init__(self, state_data, reward_data):
        super(CustomEnv, self).__init__()

        # Define the action and observation space
        self.action_space = spaces.Discrete(3)
        self.observation_space = spaces.Box(low=-np.inf, high=np.inf, shape=(5,), dtype=np.float32)

        # Store the state and reward data
        self.state_data = state_data
        self.reward_data = reward_data

        # Ensure the state and reward data have the same length
        assert self.state_data.shape[0] == self.reward_data.shape[0]

        # Initialize the current step
        self.current_step = 0

    def step(self, action):
        # Ensure action is valid
        assert self.action_space.contains(action)

        # Get the reward for the action at the current step
        reward = self.reward_data[self.current_step, action]

        # Move to the next step
        self.current_step += 1

        # Check if the episode is done
        done = self.current_step == self.state_data.shape[0]

        # Get the next state
        next_state = self.state_data[self.current_step] if not done else np.zeros(self.state_data.shape[1])

        return next_state, reward, done, {}

    def reset(self):
        # Reset the environment to the beginning
        self.current_step = 0
        return self.state_data[self.current_step]

    def render(self, mode='human', close=False):
        # This environment doesn't have a visual representation, but you could add one if you want.
        pass

My questions: I believe that that must be a better, more systematic way to find a 'good policy' than the ad-hoc policy I can come up with myself and described above...

• Is reinforcement learning the best way to approach this problem?
• If so, what methods or algorithms would be the best to try first?
  • As far as I can tell, tabular Q learning won't work because I have continuous inputs.
  • Given the noisy reward behaviour, and from my own experiments, I imagine a very simple model would work best.
• Have I set up my gym environment correctly?
• What other tools/methods should be considered? The simpler the better!

Answer 1

If you can visit all states, dynamic programming (DP) will often find the optimtal solution.

A slightly more complex optional is value function approximation (VFA). It can be impossible to perform tabular Q-learning on continuous data. VFA replaces a look-up table with a function. If you have enough data, it is possible to use deep learning within VFA.