Function tokenizePair

tokenizePair(
    prompt: string,
    response: string,
    tok: LoadedTokenizer,
    maxLength: number,
): { ids: number[]; promptLen: number }
Tokenize prompt+response, returning (ids, promptLen) — port of dpo.py _tokenize_pair, including the left-truncation that preserves the response when prompt+response exceeds maxLength.
Parameters
- prompt: string
- response: string
- tok: LoadedTokenizer
- maxLength: number
Returns { ids: number[]; promptLen: number }
- Defined in packages/training/src/dataset.ts:266