Interface FillTokenizerLike

The tokenizer surface row compilation needs (LoadedTokenizer satisfies it).

interface FillTokenizerLike {
    bosTokenId: number | null;
    decode(ids: number[], skipSpecialTokens?: boolean): string;
    encode(text: string, addSpecialTokens?: boolean): number[];
}

Index

Properties

Methods

Properties

`Readonly`bosTokenId

bosTokenId: number | null

Methods

decode

decode(ids: number[], skipSpecialTokens?: boolean): string
Parameters
- ids: number[]
- OptionalskipSpecialTokens: boolean
Returns string
- Defined in packages/inference/src/generation/fill/schema-rows.ts:46

encode

encode(text: string, addSpecialTokens?: boolean): number[]
Parameters
- text: string
- OptionaladdSpecialTokens: boolean
Returns number[]
- Defined in packages/inference/src/generation/fill/schema-rows.ts:45