[Question]: Should We Commit to a GRU Planner / Bézier Planner Before Defining the RL/GRPO Exploration Strategy? #211

I want to raise a design question before we commit too strongly to the current Bézier planner.
The current GRU / Bézier planner works naturally for behavior cloning / imitation learning: the model predicts control points, and those control points define a trajectory. My concern is what happens later when we want to move from BC to RL post-training, particularly something like GRPO.

GRPO needs multiple meaningful trajectory rollouts for the same scene so that they can receive different rewards and be compared relative to one another. In our current GRU formulation, the planner is effectively deterministic, so we do not naturally have an exploration mechanism.

One possibility would be to perturb the predicted Bézier control points. However, I am not sure what the right recipe for doing this would be. Completely random perturbations may generate trajectories that are too far from the pretrained/BC policy, physically unrealistic, or consistently poor. If most sampled trajectories receive similarly bad rewards, I am not sure we would get sufficiently useful relative gradients during RL to optimize the policy.

Ideally, we would want structured exploration around the current policy: trajectories that are different enough to explore alternative behaviors, but still plausible and close enough to the pretrained policy to provide useful reward differences.

This is one reason I think Flow Matching may be worth considering before finalizing the planner design.
Flow Matching already gives us a stochastic generative process for trajectories. However, even with Flow Matching, simply changing the initial noise seed is probably not sufficient for RL exploration.
During imitation learning, if a scene has only one demonstrated trajectory, the learned conditional trajectory distribution will tend to concentrate around that demonstration. If similar scenes contain multiple demonstrated behaviors, Flow Matching can represent those modes, but it is still primarily learning the modes present in the demonstrations.

So, in my view:
Multimodality learned from demonstrations is not necessarily the same thing as useful RL exploration.
Qwen-Drive provides an interesting recipe for addressing this. Instead of relying only on different initial Flow Matching seeds, they make the final few sampling transitions stochastic during RL. They inject structured low-frequency perturbations so that the sampled trajectories differ through smooth trajectory-level changes rather than waypoint-level jitter, while also using a restoring term to keep exploration close to the pretrained Flow Matching policy.
I could not find an equivalent recipe in Alpamayo-R1 for directly applying GRPO to the continuous Flow-Matching Action Expert.
I am therefore wondering whether we should decide on the BC → RL post-training strategy before locking in the planner representation.
Some questions I think we should answer:

• Do we have a principled way to make the Bézier planner stochastic?
• Is there a good method for perturbing Bézier control points while keeping trajectories physically meaningful?
• Should we predict a distribution over Bézier control points instead of deterministic control points?
• How do we make sure exploration stays close enough to the BC policy to generate useful GRPO reward differences?
• Would Flow Matching give us a cleaner path from imitation learning to RL post-training?
• Could we adapt something similar to Qwen-Drive's structured stochastic sampling for our planner?

I am not suggesting that Bézier is necessarily the wrong choice. I mainly want to make sure we understand the implications for RL/post-training before committing to it as the final trajectory representation.
If there is already a good recipe for structured exploration over Bézier control points, I would be very interested in exploring that as well.