Question: Can a classifier be trained with reinforcement learning without access to single classification results?
I want to train a classifier using reinforcement learning. However, there is one big restriction: the program does not have access to the score regularly, not even after every classification. Only after many classifications were completed (e.g. around 40-200 classifications, let's call them a batch) the final score of that batch is available. One batch can be executed rather quickly: it takes just around one second. Therefore, thousands of batches can be executed, each of them returning a score for its classifications. Every time a batch is executed, the current ML Model is given as input for the batch to use.
Other than that, of course, the feature vector is known (contains around 60 features) and the labels are known (around 6 labels).
I have never applied Reinforcement Learning before, therefore, I can not tell whether this can work. In theory, I think, it should: all data is available. The algorithm can choose some parameter values for the model, try them out, and get a score. Then try out different values and get the score again. This way it should be able to improve step by step.
Additional Notes: Although the text above should be enough to understand the problem and provide an answer (which can be general and not specific to a concrete use case), my personal use case and details about it are explained here. This might be useful to understand the problem in more detail.
Edit: Before, I used "Random Forest" as an example of a possible ML model that one could use. As a Random Forest seems to require supervised learning contrary to reinforcement learning, I have removed it from the text (not counting some special use cases, such as this one).