Skip to content

code for Understanding the Effects of Second-Order Approximations in Natural Policy Gradient Reinforcement Learning

License

Notifications You must be signed in to change notification settings

gebob19/natural-policy-gradient-reinforcement-learning

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

2 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Understanding the Effects of Second-Order Approximations in Natural Policy Gradient Reinforcement Learning [ArXiv]

Setup

  • Python 3.8.0
  • pip install -r req.txt
  • Mujoco 200 license

Main Files

  • main.py: main run file for model training
  • models.py: neural networks for policy and critic models
  • optim.py: second-order approximations for realizing the natural gradient
  • utils.py: helper functions

Reproducing Experiments

  • scripts/: bash training scripts formatted for compute canada/SLURM jobs
  • visualize/json: training hyperparameters for each experiment
  • visualize/csv: training results in .csv format
  • visualize/performance.py: (after training) view results & create .csv results
    • best to run with VSCode ipython cells

Experiment Example

To run the baseline experiments:

  • Tune hparams: bash scripts/hparams/baseline.sh
    • runs will be saved in runs/hparams_baseline/...
  • Extract best hparams from runs: python baseline_hparams.py
    • the best hparams will be saved in visualize/json/baseline.json
  • Run training with hparams: bash scripts/baseline/diagonal.sh
    • runs will be saved in runs/5e6_baseline/...
  • Run speed tests: bash scripts/speed/baseline.sh
    • runs will be saved in runs/baseline_speed/...
  • View results: run interactive ipython in visualize/performance.py
# %%
runs_path = pathlib.Path("../runs/5e6_baseline/")
speed_runs_path = pathlib.Path("../runs/baseline_speed/")
name = "baseline"
baseline_data = analyze(runs_path, speed_runs_path)
baseline_df = mean_df(*baseline_data, name, save=True)

Second-order Approximation References

Implementations

Other

  • Code formatted with Black
  • Experiment runs format: runs/{experiment_name}/{env_name}/{approximation}_runs/{tensorboard folder}/...

About

code for Understanding the Effects of Second-Order Approximations in Natural Policy Gradient Reinforcement Learning

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published