feat(trainer): IPO+KL — double-sided mask, KL only on kept tokens

[samsja]

Summary

Two modifications to the existing IPO+KL default loss (DPPO-Binary TV PG term + Kimi-K2.5 KL term):

1. Double-sided importance-ratio masking (advantage-sign-agnostic): tokens whose ratio $r_t = \pi_{\text{train}}/\pi_{\text{infer}}$ falls outside $[\alpha, \beta]$ are dropped. Replaces the original advantage-conditioned probability-difference mask.
2. KL only on kept tokens. Previously kl_loss = loss_mask * log_ratio**2 — KL was paid even on tokens excluded from the PG term. Now it's gated by keep_mask, so masked tokens contribute nothing to either term.

The PG-term shape (DPPO-Binary TV) and the KL form (Kimi-K2.5, $(\log r_t)^2$) are unchanged. adv_tau / teacher_tau (on-policy distillation) are also unchanged.

$$\mathcal{L}(\theta) = \sum_t \mathbb{1}[r_t \in [\alpha, \beta]] \cdot \mathbb{1}[\text{loss_mask}] \cdot \big(-\widehat{A}_t \cdot r_t + \tau_{\text{kl}} \cdot (\log r_t)^2\big)$$

Breaking config changes (trainer.loss)

Field	Was	Now
dppo_mask_low (default 0.2)	prob-diff threshold	importance-ratio lower bound α
dppo_mask_high (default 0.2 → 5.0)	prob-diff threshold	importance-ratio upper bound β

kl_tau (1e-3), adv_tau (1.0), teacher_tau (0.0) untouched.

Notes

• Sibling PR feat(trainer): replace DPPO+KL default loss with IcePop #2401 (feat/icepop-loss) is pure IcePop, no KL term at all. This PR keeps the existing KL — useful if the KL signal turns out to add stability on top of the double-sided mask alone.
• Field names kept (dppo_mask_*) to keep the diff against main minimal; only the semantic and one default change.
• Test file untouched — existing smoke tests pass dppo_mask_high=10.0 as a kwarg, which still resolves to a valid (now ratio-based) value.

🤖 Generated with Claude Code