The data comes from a simulator hence I have the possibility to generate unlimited data. The reward is 0 (no success) and success(1) if episode is successful. Now, the question is what metric to use for evaluation? For example, if I choose the average success rate as metric and hypothetically someone trains the RL model on one episode and the reward is 1 then the average will be 1. So I have to choose min. number of episodes. Should it be 10, 1000, 10000 and how do I calculate it from the data? Each episode is separate task so each episode might be different length than the previous one and might have different complexity.