feat(trainer): replace DPPO+KL default loss with IcePop

[samsja]

Summary

Override the default RL loss with IcePop from the INTELLECT-3 technical report (eq. 1):

$$J_{\text{IcePop}}(\theta) = \mathbb{E}\left[ \frac{1}{\sum_i |y_i|} \sum_{i,t} \mathcal{M}!\left(\frac{\pi_{\text{train}}(y_{i,t}\mid x, y_{i,<t};\theta)}{\pi_{\text{infer}}(y_{i,t}\mid x, y_{i,<t};\theta_{\text{old}})};\alpha,\beta\right) \widehat{A}_{i,t} \right]$$

$$\mathcal{M}(k; \alpha, \beta) = \begin{cases} k & k \in [\alpha, \beta] \ 0 & \text{otherwise} \end{cases}$$

Tokens whose trainer/inference importance ratio falls outside $[\alpha, \beta]$ get zero policy-gradient weight — they are dropped, not clipped. There is no separate KL penalty — the double-sided ratio mask is what keeps the update inside the trust region.

Breaking config changes (trainer.loss)

Removed	Added	Default
dppo_mask_low	icepop_ratio_low (α)	0.2
dppo_mask_high	icepop_ratio_high (β)	5.0
kl_tau	—	(term removed)

adv_tau and teacher_tau are unchanged. mismatch_kl / is_masked* metrics are retained for observability (semantics now ratio-based, not advantage-conditioned).

Notes

• Token-level mask is detached from autograd: gradient flows through importance_ratio only for in-range tokens.
• Replaces, rather than adds, a loss type — the user asked for the new behavior to override the existing default.

🤖 Generated with Claude Code