Class WhisperTokenizer

constructor

• new WhisperTokenizer(
      tok: LoadedTokenizer,
      numLanguages: number,
  ): WhisperTokenizer

  Parameters

  • tok: LoadedTokenizer
  • numLanguages: number

  Returns WhisperTokenizer

ReadonlyallLanguageCodes

ReadonlyallLanguageTokens

Readonlyeot

ReadonlynoSpeech

ReadonlynoTimestamps

ReadonlynumLanguages

Readonlysot

ReadonlysotLm

ReadonlysotPrev

ReadonlytimestampBegin

Readonlytranscribe

Readonlytranslate

nonSpeechTokens

• get nonSpeechTokens(): number[]

  tokenizer.non_speech_tokens — punctuation/symbol tokens suppressed at every step (mlx-whisper tokenizer.py).

  Returns number[]

decode

• decode(ids: number[]): string

  decode() drops timestamp tokens (ids ≥ timestampBegin) like the oracle.

  Parameters

  • ids: number[]

  Returns string

decodeText

• decodeText(ids: number[]): string

  Text tokens only (id < eot).

  Parameters

  • ids: number[]

  Returns string

decodeWithTimestamps

• decodeWithTimestamps(ids: number[]): string

  Parameters

  • ids: number[]

  Returns string

encode

• encode(text: string): number[]

  Parameters

  • text: string

  Returns number[]

languageToken

• languageToken(language: string): number

  Parameters

  • language: string

  Returns number

sotSequence

• sotSequence(language: string | null, task: WhisperTask): number[]

  [sot, <|lang|>, <|task|>]

  Parameters

  • language: string | null
  • task: WhisperTask

  Returns number[]

splitToWordTokens

• splitToWordTokens(
      tokens: number[],
      language: string,
  ): { words: string[]; wordTokens: number[][] }

  Split text tokens into words (split_to_word_tokens): unicode-boundary split for CJK/Thai/Lao/Myanmar, space split otherwise.

  Parameters

  • tokens: number[]
  • language: string

  Returns { words: string[]; wordTokens: number[][] }