Token filters the optimized decode steps apply on device.
Ids masked at the first sampled position (blank + eot), or null.
Ids masked at every step (non-speech + task/special tokens).
Token filters the optimized decode steps apply on device.