Hindsight Experience Replay (HER) results obtained 50 times faster than original paper?

Question

I am reproducing the results from Hindsight Experience Replay by Andrychowicz et. al. In the original paper they present the results below, where the agent is trained for 200 epochs.

200 epochs * 800 episodes * 50 time steps = 8,000,000 total time steps.

I try to reproduce the results but instead of using 8 cpu cores, I am using 16 CPU cores.

---

Fetch Push

I train the FetchPush for 80 epochs, but with only 50 episodes per epoch. Therefore 80 * 50 * 50 = 200,000 iterations. I present the curve below, generated using two random seeds.

After 20 epochs = 50,000 iterations we solve this environment. In the paper above, it took the original authors 100 episodes = 4,000,000 iterations to do so.

How is my algorithm converging 50 times faster?

---

Pick and Place

I train the FetchPickAndPlace for 80 epochs, but with only 50 episodes per epoch. Therefore 80 * 50 * 50 = 200,000 iterations. I present the curve below, generated using three random seeds:

and logger output for the first two epochs, showing that indeed I have 50 episodes per epoch:

Now, as can be seen from my tensorboard plot, after 40 epochs we get a steady success rate, close to 1. 40 epochs * 50 episodes * 50 time steps = 100,000 iterations. Therefore it took the algorithm approximately 100,000 time steps to learn this environment.

The original paper took approximately 50 * 800 * 50 = 2,000,000 time steps to achieve the same goal.

How is it that in my case the environment was solved nearly 20 times faster? Are there any flaws in my workings above? Surely I am doing something wrong, right?

Results are also faster than another paper which also uses 19 MPI workers:

As stated in this paper: "We train for 50 epochs (one epoch consists of 19 2 50 = 1 900 full episodes), which amounts to a total of 4.75 x10^6 timesteps." It took around 2,000,000 timesteps to reach a median success rate of 0.9.

---

SUMMARY

---

Any suggestions on what I may be doing wrong would be appreciated.

Code Reddit thread

---

EDIT - Logging

Logging process shows that the rank 0 worker is reporting results. Inside her.py)

   if rank == 0:
        logger.dump_tabular()

The function responsible for writing all diagnostics is dumpkvs() inside logger.py:

def dumpkvs():
    """
    Write all of the diagnostics from the current iteration
    """
    Logger.CURRENT.dumpkvs()

Code can be found here:

https://github.com/openai/baselines/blob/master/baselines/logger.py

Answer 1

I am reading the results incorrectly.

1 epoch = 50 episodes = 2500 timesteps
50 epochs = 2500 episodse = 125,000 timesteps
125,000*16 CPU cores = 2 million time steps.

Will give a more detailed answer later.