1
$\begingroup$

I am new to reinforcement learning and I would like know how to generate a learning curve plot such as that shown below (taken from this blog post), that illustrates the reward (return) and its variance (shaded region). I would like to use Matplotlib or any other Python plotting framework.

$\endgroup$
1
$\begingroup$

To estimate the variance, you probably need to run your algorithm multiple times and keep track of the return for each of these runs. From these multiple returns, you can estimate the variance.

Once you have the standard deviation (or variance) of the return, you can plot something like your plots using matplotlib's fill_between function.

| improve this answer | |
$\endgroup$
  • $\begingroup$ So, do I need to run the algorithm for a specific number of episodes (e.g., 3 episodes) once or multiple times? $\endgroup$ – Cristian M Apr 18 at 13:11
  • $\begingroup$ @CristianM It depends on what you want to estimate or visualise. If you want to estimate the variance of all episodes, then, suppose you have $N$ episodes, then you will have $N$ returns from which you can compute an average return and a variance. $\endgroup$ – nbro Apr 18 at 13:12
  • $\begingroup$ I want to generate a plot similar to that presented above. As I understand, the plot shows the reward's mean value and variance for 1,2,3 to 200 episodes. In order to obtain these values, I need to compute the reward for each episode (1,2,...200) multiple times (e.g. 10 times) to obtain its corresponding reward mean and variance. That would imply 200 episodes * 10 runs/episode = 2000 runs. Please correct me if I am wrong. $\endgroup$ – Cristian M Apr 18 at 13:22
  • $\begingroup$ @CristianM As I said, it depends on what you want to do. You decide. Do you want to see the average return for all episodes? Or maybe do you want to see the average return for a single episode? If it's the second option, you will have to run an episode multiples times. If it's the first option, you just need to run each episode once, then compute the average of these returns. Again, you decide what you will do depending on what you want to do. $\endgroup$ – nbro Apr 18 at 13:25
  • $\begingroup$ @ nbro OK. Thank you! $\endgroup$ – Cristian M Apr 18 at 13:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.