HERO

Learning Humanoid End-Effector Control for
Visual Whole-Body Open-Vocabulary Object Grasping

Runpei Dong · Ziyan Li · Arjun Gupta · Xialin He · Saurabh Gupta

CoRL 2026

Train HERO in Isaac Sim and try humanoid grasping in your browser with the Unitree G1 and Dex3 hands.

Models and benchmark data: Hugging Face collection.

Interactive demo

Select an object and click Pick & place. Physics and inference run locally in desktop Chrome or Edge. The bundled example ONNX model uses delta anchor to further improve end-effector tracking.

30-second loop · UIUC I → tray → return · Left-side Cheez-It (50 cm round table) → tray → return

Install and build the demo, then open build/demo/Tabletop_Lab.html. See checkpoint compatibility for other policies.

Installation

Demo and data tools

Use Python 3.10+ and Node.js 18+. From the repository root:

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e '.[demo,dev]'

Build the browser demo

Export the example policy and scenes, then package the frontend:

python scripts/fetch_ycb_cracker_box.py

python -m sim2sim.interactive_client.export_policy_assets \
  --hero checkpoints/example/model.onnx --parity-fixture
python -m sim2sim.interactive_client.export_scene \
  --hero checkpoints/example/model.onnx

cd sim2sim/interactive_client
npm ci
python prepare_assets.py --delivery ../../build/demo
npm run build
python package_standalone.py --output ../../build/demo

For development, run npm run dev here and open http://127.0.0.1:8768.

Training environment

Use Linux, an NVIDIA GPU, Isaac Sim 5.1, and Isaac Lab 2.3. Install Isaac Lab, activate its environment, then install:

python -m pip install -e third_party/holosoma --no-deps
python -m pip install -e '.[train,dev]'

Training data

The paper combines AMASS motions and generated IK reaches. Neither dataset is bundled.

Prepare AMASS

Obtain AMASS and retarget it to G1 before preparation:

python scripts/prepare_amass.py \
  --input /path/to/retargeted_amass_g1 \
  --output data/amass \
  --jobs 8

The input must contain G1 trajectories, not raw SMPL files. See data formats and manifests.

Generate IK reaches

Start with the example generator:

python scripts/generate_ik_example.py \
  --output data/ik_examples --num-clips 32 --seed 0

Extend its targets, orientations, and motion profiles for your corpus. See --help and the profile definitions.

Training

In the Isaac Lab environment, start with AMASS from the repository root:

python scripts/train.py --motion-dir data/amass

Use --config to select training variants: dual-actor PPO (default, with or without delta anchor), without_delta_ee (no residual end-effector feedback), or *_single (one whole-body actor). Outputs go to runs/.

For the paper's recipe, combine AMASS and IK data with explicit source weights:

python scripts/train.py \
  --motion-dir /path/to/amass_and_generated_ik \
  --allow-mixed-data

Export a trained policy

In the Isaac Lab environment, export ONNX with matching _hero.json metadata:

python scripts/export.py \
  --checkpoint /path/to/model_20000.pt \
  --motion-dir data/amass \
  --output checkpoints/my_policy

Use your policy in the browser demo

Export the policy and scenes from the same checkpoint:

python -m sim2sim.interactive_client.export_policy_assets \
  --hero checkpoints/my_policy/model_20000.onnx \
  --allow-custom-policy --parity-fixture
python -m sim2sim.interactive_client.export_scene \
  --hero checkpoints/my_policy/model_20000.onnx

Repeat the frontend build steps. Keep each model with its matching metadata.

Validation

Use the training installation (bundled backend and .[train,dev]):

python -m pytest tests -q
cd sim2sim/interactive_client
npm test
node policy_parity.mjs ../../build/demo/policies

Parity compares browser inference with the native reference from --parity-fixture.

Benchmark

hero_bench_v1 has 1,298 reaching clips: core 420, extended 618, and stress 260. Core includes the paper's 180-target protocol byte for byte. MuJoCo evaluates open-loop control and HERO replanning, reporting world-frame palm errors, success rates, and confidence intervals.

pip install -e ".[bench]"
python scripts/hero_bench.py fetch --url https://huggingface.co/datasets/RunpeiDong/hero_bench/resolve/main/hero_bench_v1_corpus_33542601.tar.gz --sha256 33542601ed456ac606f8177ab75892d2e616585ec562f01976cb6af238c9c5d0 --out data/hero_bench_v1
python scripts/hero_bench.py run --corpus data/hero_bench_v1 --onnx-dir checkpoints/example --sidecar checkpoints/example/model_hero.json --out results/example --tiers core
python scripts/hero_bench.py report --corpus data/hero_bench_v1 --results results/example --card

Reference results for the bundled checkpoints/example model (delta anchor, five-frame 1,000-dimensional actor input, 44,000 iterations) on hero_bench_v1 (protocol hash 139ea1d62b8ab1d1, MuJoCo, Dex3 hands, --odom so, --tiers all; fail-free 100 % in every cell). Errors are the reaching hand's world-frame palm error: open-loop hold and hold3 average the clip's hold window, final-1 s the last second before the padded clip end.

tier	open-loop hold (cm)	closed-loop hold3 (cm)	closed-loop final-1 s (cm)	S7.5	C3
core (420)	5.5	5.2	3.2	82.4 %	52.6 %
extended (618)	5.3	4.5	3.2	86.2 %	60.4 %
stress (260)	7.0	6.8	5.4	68.5 %	35.8 %

S7.5 = open-loop hold mean <= 7.5 cm and <= 15 deg; C3 = closed-loop final second <= 3 cm and <= 15 deg (Wilson 95 % intervals are in the results cards). Thresholds, the stop rule, and the odometry mode and seed rule are part of the corpus manifest.

See benchmark protocols and corpus setup.

Citation

Please cite HERO if you use it in your research:

@inproceedings{dong2026hero,
  title     = {{HERO}: Learning Humanoid End-Effector Control for Visual Whole-Body Open-Vocabulary Object Grasping},
  author    = {Dong, Runpei and Li, Ziyan and Gupta, Arjun and He, Xialin and Gupta, Saurabh},
  booktitle = {10th Annual Conference on Robot Learning},
  year      = {2026},
  url       = {https://openreview.net/forum?id=gbchkYm28k}
}

Acknowledgments and license

Built with HoloSoma, Isaac Lab, MuJoCo, Mink, ONNX Runtime, and Three.js.

Code: MIT. Third-party terms: THIRD_PARTY_NOTICES.md.