Can a classifier be trained with reinforcement learning without access to single classification results?

Question

Question: Can a classifier be trained with reinforcement learning without access to single classification results?

I want to train a classifier using reinforcement learning. However, there is one big restriction: the program does not have access to the score regularly, not even after every classification. Only after many classifications were completed (e.g. around 40-200 classifications, let's call them a batch) the final score of that batch is available. One batch can be executed rather quickly: it takes just around one second. Therefore, thousands of batches can be executed, each of them returning a score for its classifications. Every time a batch is executed, the current ML Model is given as input for the batch to use.

Other than that, of course, the feature vector is known (contains around 60 features) and the labels are known (around 6 labels).

I have never applied Reinforcement Learning before, therefore, I can not tell whether this can work. In theory, I think, it should: all data is available. The algorithm can choose some parameter values for the model, try them out, and get a score. Then try out different values and get the score again. This way it should be able to improve step by step.

Additional Notes: Although the text above should be enough to understand the problem and provide an answer (which can be general and not specific to a concrete use case), my personal use case and details about it are explained here. This might be useful to understand the problem in more detail.

Edit: Before, I used "Random Forest" as an example of a possible ML model that one could use. As a Random Forest seems to require supervised learning contrary to reinforcement learning, I have removed it from the text (not counting some special use cases, such as this one).

Answer 1

If every data point has a ground-truth label (i.e., one of the six labels), than any supervised learning technique can work, including Random Forest. If the labels come in batches, then the parameters of the model can be updated with each new batch. Either completely retrain the model with data up to the current time point or incrementally update the model parameters with the new data.

There is no reason to apply reinforcement learning unless it is necessary.

If every data point does not have a ground-truth label, then reinforcement learning might be appropriate. Then you can not use Random Forest or any other supervised learning technique. You'll need to pick a Reinforcement Learning technique that can handle sparsely labeled data.

Answer 2

If you want to train an AI agent, then you should use RL, but only if your agent has access to control of the game. It should be able to play the game (i.e. interact with the environment) to be able to optimize its policy. It doesn't matter if the reward is delayed, but interaction is crucial. So, basically two elements: interaction with the environment and reward gaining are needed for any RL model. The model has to be able to explore and exploit its policy in the process of improvement, that's why it has to receive feedback from its actions according to the current best policy, not some random games.

If you want to just classify some game data (units in your case) collected from the game, you should try different classification algorithms, but not RL. I won't be an AI agent, it will be just a classification model, so it won't be able to play the game.

One other option of learning how to play the game without actually playing for your agent would be to collect the data after each game, but not only the data about the units (i.e. states of the game environment), but also the possible actions in each state and actions taken in those states. Then you could hard code some heuristic search algorithm to look for the best possible action in each state of the game. But then again you won't be able to test it unless your agent can interact with the game. And it probably won't be as effective as RL trained agent. So, I'd advise finding a way for an agent to have access to the game and then to try the RL algorithms.