Training RL Agents¶
Support is provided for a variety of Online RL Algorithms including PPO, SAC and DQN. To add support for a new RL Agent, see Creating a New RL Agent
RL Algorithms (Agents) are specified in the rl_algs/ directory. Each algorithm is self-contained in a <agent>.py file.
Common functions for creating an environment, loading data to a buffer, etc are stored in the rl_algs/rl_utils.py file.
RL Algorithm implementations follow the CleanRL paradigm.
Important
When training a RL Agent, the site_start_date and site_end_date should correspond to the crop_start_date and crop_end_date, otherwise there will be
crop observation variables missing which the RL Agent cannot handle.
Starting a Training Run¶
To train a RL Agent, run the following:
python3 train_agent.py --agent-type PPO --save-folder logs/ppo/
This trains a PPO Agent with the default hyperparameters and saves the output to logs/ppo/. Support for
Weights and Biases is included and can be toggled with:
python3 train_agent.py --agent_type <agent_type> --save-folder logs/test/ --track
Important
The --agent-type must match the name of the RL Agent file.py and name of the nn.Module class that selects actions.
For more information, see Creating a New RL Agent
Algorithm Configuration¶
Every <agent>.py file in rl_algs/ has an associated Args dataclass. See each .py file for the
corresponding hyperparameters.
To specify a hyperparameter, run the code:
python3 train_agent.py --agent-type PPO --save-folder logs/ppo/ --PPO.gamma 0.85
The above block of code trains a PPO agent with gamma value of 0.85.
Important
Ensure that the argumement specified, i.e. --PPO.gamma correspond to the agent type, in this case --agent-type PPO.
Otherwise, the argument will not be passed to the correct algorithm.
Note
As with any top-level script in WOFOST-Gym, a configuration can be loaded via the --config-fpath <Path to file> argument.
For training agents, this can be used to load a previously saved hyperparameter configuration, and/or can be used to load a previously saved
environment. See Command Line Arguments for more information.