GoRL: Generative Online Reinforcement Learning

Chubin Zhang*¹, Zhenglin Wan*², Feng Chen¹, Fuchao Yang¹, Lang Feng¹, Yaxin Zhou³, Xingrui Yu^4,5, Yang You², Ivor Tsang^1,4,5, Bo An¹

¹ Nanyang Technological University · ² National University of Singapore · ³ Carnegie Mellon University
⁴ CFAR, A*STAR · ⁵ IHPC, A*STAR

(*: Equal contribution)

     

🎉 GoRL has been accepted to ICML 2026!

GoRL (Generative Online Reinforcement Learning) separates tractable latent policy optimization from expressive flow-matching and diffusion action generation.

📖 Table of Contents

• Key Features
• Method
• Installation
• Usage
• Configuration
• Directory Structure
• Results
• Acknowledgement
• Citation
• License

✨ Key Features

Feature	Description
Latent optimization	PPO updates a tractable latent policy rather than the generative sampling chain.
Generative policies	Both flow-matching and diffusion decoders are supported.
Alternating training	Latent policy optimization alternates with fixed-prior decoder refinement.
Eight control tasks	Six DMControl tasks plus HumanoidStand and HumanoidRun, with PPO, FPO, and DPPO baselines.

🧠 Method

GoRL separates a generative policy into a PPO-trained latent encoder and a flow-matching or diffusion action decoder:

$$\pi(a\mid s) = \int p_\phi(a\mid s,z),\pi_\theta(z\mid s),\mathrm{d}z$$

• Encoder $\pi_\theta(z\mid s)$: a tractable latent policy optimized with PPO.
• Decoder $g_\phi(s,z)$: a flow-matching or diffusion model that maps latent decisions to actions.

Training alternates between collecting policy data, fitting the decoder, and optimizing the encoder with the decoder frozen. The six standard tasks start with an identity decoder. Humanoid tasks first train a fresh Brax PPO teacher. No pretrained checkpoint or dataset is required.

🛠️ Installation

Tested on Linux with Python 3.12 and NVIDIA CUDA 12.

git clone https://github.com/bennidict23/GoRL.git
cd GoRL
unset PYTHONPATH PYTHONHOME
export PYTHONNOUSERSITE=1
conda create -n gorl python=3.12 pip=25.0.1
conda activate gorl
python -m pip install -r requirements.txt
export XLA_PYTHON_CLIENT_PREALLOCATE=false
export MUJOCO_GL=egl

Run a small training experiment:

CUDA_VISIBLE_DEVICES=0 gorl train --task CheetahRun --method gorl_fm --seed 1 --smoke

The first run compiles JAX kernels and may be quiet for several minutes. --smoke checks execution, not benchmark performance.

🚀 Usage

GoRL

Remove --smoke to use the full training schedule:

# Flow matching
CUDA_VISIBLE_DEVICES=0 gorl train --task CheetahRun --method gorl_fm --seed 1

# Diffusion
CUDA_VISIBLE_DEVICES=0 gorl train --task CheetahRun --method gorl_diffusion --seed 1

The same command works for every task. Profiles are selected automatically.

Tasks	GoRL schedule
CheetahRun, FingerSpin, FingerTurnHard, FishSwim, HopperStand, WalkerWalk	60M / 60M / 30M / 30M
HumanoidStand	60M / 60M / 30M / 30M
HumanoidRun	60M backbone anchor + 60M / 60M latent stages

Humanoid GoRL additionally trains its teacher for 180M requested PPO steps (about 185.79M after update rounding), outside the nominal schedule above. Humanoid PPO uses Brax; the other PPO tasks use the FPO backend.

Baselines

Fetch the pinned baseline code once; Git is required:

gorl-fetch-dependencies --dependency all --destination-root external

CUDA_VISIBLE_DEVICES=0 gorl train --task CheetahRun --method ppo --seed 1
CUDA_VISIBLE_DEVICES=0 gorl train --task CheetahRun --method fpo --seed 1
CUDA_VISIBLE_DEVICES=0 gorl train --task CheetahRun --method dppo --seed 1

Baselines use a single 180M-step training budget.

Outputs and W&B

Runs are saved under runs/. Final return is the last evaluation, not the best.

W&B is optional and disabled by default:

wandb login
gorl train --task CheetahRun --method gorl_fm --seed 1 --wandb-mode online

Use --wandb-mode offline without an account. Curves stay continuous across stages.

⚙️ Configuration

Defaults are in configs/tasks/ and configs/methods/. Use --config FILE.toml for overrides and gorl train --help for options. Keep hyperparameters fixed when comparing seeds. All role seeds default to --seed.

To train both Humanoid variants using one freshly trained teacher:

CUDA_VISIBLE_DEVICES=0,1 gorl train-pair --task HumanoidRun \
  --teacher-seed 1 --fm-seed 1 --diffusion-seed 3 --gpus 0 1

Smoke runs are not a full-budget performance guarantee. HumanoidStand Diffusion can have large decoder-switch drops or non-finite training. Teacher training and collection consume additional interactions beyond the nominal schedule. Training-state resume is not supported.

📂 Directory Structure

.
├── configs/    Task, method, and dependency settings
├── docs/       Framework and results figures
├── scripts/    Baseline dependency setup
└── src/        GoRL algorithms and baseline adapters

📊 Results

The original six-task comparison is shown below, with GoRL-FM and GoRL-Diffusion alongside PPO, FPO, and DPPO. This release also includes HumanoidStand and HumanoidRun.

🙏 Acknowledgement

GoRL builds on FPO, Brax, and MuJoCo Playground. See third-party notices.

📝 Citation

If you find this code useful, please cite our paper:

@inproceedings{pmlr-v306-zhang26fu,
  title     = {Generative Online Reinforcement Learning},
  author    = {Zhang, Chubin and Wan, Zhenglin and Chen, Feng and Yang, Fuchao and Feng, Lang and Zhou, Yaxin and Yu, Xingrui and You, Yang and Tsang, Ivor and An, Bo},
  booktitle = {Proceedings of the 43rd International Conference on Machine Learning},
  year      = {2026},
  volume    = {306},
  pages     = {158816--158837},
  series    = {Proceedings of Machine Learning Research},
  publisher = {PMLR},
  url       = {https://proceedings.mlr.press/v306/zhang26fu.html}
}

📄 License

MIT License.