The pure ORPO reduction from the two length-normalized log-probs ([B] each):
L_NLL + λ·L_OR with L_OR = mean(softplus(-log_odds)). The SFT term:
when nllFull is ABSENT (sft_scope:"response"), L_NLL = mean(-ℓw) —
bit-exact to the pre-sft_scope behavior; when PRESENT (sft_scope:"full"),
L_NLL = meanAll(nllFull), the separately-computed full-scope chosen NLL
([1]/[B]/scalar all reduce correctly). The odds-ratio term uses the
response-only ℓw/ℓr in BOTH modes (matching TRL). Model-free and
graph-preserving — the forwards live in orpoLoss, the math (and its grad
path) lives here, so the loss can be unit-tested against a hand reference
with synthetic lw/lr. Does NOT dispose lw/lr/nllFull (the caller owns them);
returns the scalar loss.
The pure ORPO reduction from the two length-normalized log-probs ([B] each):
L_NLL + λ·L_ORwithL_OR = mean(softplus(-log_odds)). The SFT term: whennllFullis ABSENT (sft_scope:"response"),L_NLL = mean(-ℓw)— bit-exact to the pre-sft_scope behavior; when PRESENT (sft_scope:"full"),L_NLL = meanAll(nllFull), the separately-computed full-scope chosen NLL ([1]/[B]/scalar all reduce correctly). The odds-ratio term uses the response-only ℓw/ℓr in BOTH modes (matching TRL). Model-free and graph-preserving — the forwards live inorpoLoss, the math (and its grad path) lives here, so the loss can be unit-tested against a hand reference with synthetic lw/lr. Does NOT dispose lw/lr/nllFull (the caller owns them); returns the scalar loss.