Function packRows

packRows(
sampleTokens: number[][],
opts: PplDataOptions,
): Int32Array<ArrayBufferLike>[]
Tokenize samples (seeded shuffle), concatenate, and pack into non-overlapping rows of sequenceLength tokens (reference load_data).
Parameters
- sampleTokens: number[][]
- opts: PplDataOptions
Returns Int32Array<ArrayBufferLike>[]
- Defined in packages/inference/src/scoring/perplexity.ts:82