I am training an RL agent using PPO2 algorithm. Iam using stable-baselines library. During the training process, my rewards are slowly increasing and stabilizing, but are falling down suddenly. I figured its because of the catastrophic forgetting of the network. So,I would like to save my model when the mean episodic reward becomes maximum.

Example of a training process. enter image description here

So here I would like to save the model when the training rewards are higer. My code

import pandas as pd
from stable_baselines.common.policies import MlpPolicy
from stable_baselines.common.policies import FeedForwardPolicy, register_policy
from stable_baselines.common.vec_env import DummyVecEnv
from stable_baselines import PPO2
from Env import Env
import matplotlib.pyplot as plt
from stable_baselines.logger import configure


# multiprocess environment
n_cpu = 8
env = DummyVecEnv([lambda: AHUenv() for i in range(n_cpu)])

# Custom MLP policy of three layers of size 128 each
class CustomPolicy(FeedForwardPolicy):
    def __init__(self, *args, **kwargs):
        super(AhuCustomPolicy, self).__init__(*args, **kwargs,
                                           net_arch=[dict(pi=[128, 128],
                                                          vf=[128, 128])],

# Register the policy, it will check that the name is not already taken
register_policy('CustomPolicy', CustomPolicy)

model = PPO2(CustomPolicy,env,gamma=0.9, n_steps=88, ent_coef=0.01,
             learning_rate=2.5e-4, vf_coef=0.5, max_grad_norm=0.5, lam=0.95,
             nminibatches=4, noptepochs=4, cliprange=0.1, cliprange_vf=None,
             verbose=0, tensorboard_log="./11_11_full_logs_4/", _init_setup_model=True,
             policy_kwargs=None, full_tensorboard_log=False)

model.learn(total_timesteps=2000000,callback=None, seed=None,
            log_interval=1, tb_log_name="Full Logs", reset_num_timesteps=True)


Here the model is saved after the 2 million steps. Is it possible to save the models in between?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.