[riita10069]

Summary

Following the discussion in #221, #211 and #123, this PR does two things:

1. AlpaSim now gives the AutoE2E driver the inputs it was trained with. The driver ran on its camera-only track, because AlpaSim passes no HD map and the plugin ignored PredictionInput.route. It now draws the 14-channel map context and the 2-channel route mask with the nuPlan training rasterizer, from the scene's HD map and AlpaSim's route (option 2 in Closed-loop AlpaSim: nuPlan Epoch 5 drifts off the road from ego yaw-history feedback #221).
2. A minimal closed-loop RL loop, as an experiment under experiment/closed-loop/rl (outside wf_autoe2e): AlpaSim rollouts alternating with GRPO steps on the GRU planner, on one scene.

The loop works end to end, but on one scene and three small steps it does not change the driving yet (Results). I would be grateful for feedback on the reward and the exploration recipe before scaling it up.

1. Training-aligned inputs (Model/plugins/alpasim_driver)

• AlpaSim fork: a 4-line patch, experiment/closed-loop/alpasim.patch, applied by setup_alpasim.sh, adds PredictionInput.scene_id from the session's debug_info, so the driver can open the scene's USDZ. The driver config mounts AlpaSim's scene cache read-only. No proto change.
• Map context (scene_map.py): the scene's trajdata VectorMap drawn by the same rasterize_navigation_map as nuPlan packing (now factored out of data_parsing/nuplan/targets.py, pack path unchanged). Lanes stand in for nuPlan's roadblocks, wait lines for stop lines; lanes without edges are widened to 3.7 m as AlpaSim's offroad metric does. NuRec maps carry no intersection areas, so that channel stays empty.
• Route mask: channel 0 is the lanes under the route that point along it and their same-direction neighbours, behind the ego from its travelled path, ahead along unique successors to the raster's 120 m. Channel 1 is the end of the recorded drive, standing in for nuPlan's mission goal.
• replay_driver.py replays the logged route and scene id, so replays see the same inputs.

2. Minimal GRPO loop (experiment/closed-loop/rl)

• Rollouts: AlpaSim executes the planner's mean plan; grpo_step.py replays every decision from rollout.asl, keeps the planner's inputs and samples 8 plans per state. It stops if the replayed plan differs from the executed one by more than 5 cm or 0.01 rad, so updates stay on-policy.
• Exploration ([Question]: Should We Commit to a GRU Planner / Bézier Planner Before Defining the RL/GRPO Exploration Strategy? #211): the GRU planner is deterministic, so Gaussian noise goes on 4 low-frequency DCT coefficients of acceleration and of curvature. Samples differ by smooth trajectory-level changes, not per-step jitter. It is not a published recipe, and it applies to Bézier control points alike, so it does not settle [Question]: Should We Commit to a GRU Planner / Bézier Planner Before Defining the RL/GRPO Exploration Strategy? #211's planner question.
• Update: GRPO [Shao et al., 2024] with per-state groups. Advantages are centred without the std division (Dr. GRPO [Liu et al., 2025]). The step is the natural gradient in mean-plan space plus an anchor to the imitation planner over all 64 steps, with a TRPO-style line search [Schulman et al., 2015]: a step is accepted only if the surrogate falls and every state's plan change stays within a bound. The GRU planner detaches its recurrent state from its BEV lookup, so the parameter gradient is approximate. Only the planner is updated; the encoders stay frozen.
• Reward (Proposal / discussion: reward design for the stage-3 closed-loop RL policy #123) (reward.py): progress gated by safety, NC · DAC · DDC · V · EP · TTC · LK · C over 4 s of each plan.
  • The terms follow NAVSIM's PDMS/EPDMS [Dauner et al., 2024; Cao et al., 2025], multiplied with progress as in CaRL [Jaeger et al., 2025], so a plan that stops scores 0.
  • NC and TTC use NAVSIM's contact types and at-fault rules, DAC uses AlpaSim's own offroad rule, and the human filter is matched by place along the recorded path.
  • V keeps plans within 2.5-3 m of the recorded path, where the reconstruction is reliable. It measures distance to the path, so braking is not penalised, as in RAD [Gao et al., 2025].
  • Gated plans score -1 + safe_fraction: below every passing plan, with later failures above earlier ones.
  • Only the objects AlpaSim simulated are scored. Episodes end 3 m from the recorded path.

Results (experiment/closed-loop/RESULTS.md)

nuPlan Epoch 5, scene clipgt-01d503d4, one 11 s rollout per iteration (7 s on the log, then 4 s in closed loop), L40S. The attached videos are the front camera of iterations 0 to 3.

Iteration	Off-road (first)	Max dist to GT [m]	Driven [m]
0 (imitation)	3.8 s	5.92	30.2
3 (after 3 GRPO steps)	3.8 s	5.86	30.1

• Every step was on-policy and bounded. The replay error was 0.13 mm, the surrogate fell, and the plan change stayed within the trust region. A first, unbounded version diverged (KL 71, then 1852), which is why the line search is there.
• The driving does not change yet. The accepted steps are small, 0.02-0.09 of a bound of 1: the line search shrinks them until the surrogate falls, so the direction is the limit. The within-group reward spread is also small (std 0.02-0.03).
• About 60% of the decisions are gated: the imitation plan leaves the lanes within 4 s, the drift of Closed-loop AlpaSim: nuPlan Epoch 5 drifts off the road from ego yaw-history feedback #221.
• One GRPO step takes about 8 minutes for 37 decisions, with 4.3 GB of GPU memory.

Pipeline note

Entry points, data layers and lineage are unchanged. The RL loop stays in experiment/, and the targets.py change is a behaviour-preserving extraction. RL checkpoints have no Flyte lineage; if the loop proves useful, I would redesign it for wf_autoe2e in Design/ first.

Limitations and next steps

• The reward is a contextual bandit on visited states, with no credit assignment over the episode. Logged objects do not react, and there is no traffic-light term.
• Next: wider exploration (--sigma), more scenes and rollouts per step, and measuring what the planner's detach costs the update direction. After that, rollout-level groups and Proposal / discussion: reward design for the stage-3 closed-loop RL policy #123's phase-2 terms.

Testing

On the GPU host (AlpaSim venv): experiment/closed-loop/rl 35 passed; plugin and map tests 61 passed. Scoring the recorded drive itself gives 1.0 on every term.

References

• Shao et al., DeepSeekMath (GRPO), arXiv:2402.03300
• Liu et al., Understanding R1-Zero-Like Training (Dr. GRPO), arXiv:2503.20783
• Schulman et al., Trust Region Policy Optimization, ICML 2015
• Jaeger et al., CaRL: Learning Scalable Planning Policies with Simple Rewards, arXiv:2504.17838
• Dauner et al., NAVSIM, arXiv:2406.15349
• Cao et al., Pseudo-Simulation for Autonomous Driving (EPDMS), arXiv:2506.04218
• Gao et al., RAD: Training an End-to-End Driving Policy via Large-Scale 3DGS-based RL, arXiv:2502.13144

closed-loop-rl-r3-iteration-0.mp4

closed-loop-rl-r3-iteration-1.mp4

closed-loop-rl-r3-iteration-2.mp4

closed-loop-rl-r3-iteration-3.mp4