Class MlxPrefillCohort

Ordinary sampling/cache policy over the shared target preparation driver.

Hierarchy (View Summary)

MlxPrefillRows<PrefillState>
- MlxPrefillCohort

Index

Constructors

Properties

host operations

Accessors

canAdmit rows supportsMixedWork tokenWeight

Methods

admit advance dispose

Constructors

constructor

new MlxPrefillCohort(host: PrefillHost): MlxPrefillCohort
Parameters
- host: PrefillHost
Returns MlxPrefillCohort
Overrides MlxPrefillRows.constructor
- Defined in packages/inference/src/execution/prefill-cohort.ts:40

Properties

`Readonly`host

host: PrefillHost

`Readonly`operations

operations: MlxPrefillRowsHost<PrefillState>

Accessors

canAdmit

get canAdmit(): boolean
Returns boolean
Overrides MlxPrefillRows.canAdmit
- Defined in packages/inference/src/execution/prefill-cohort.ts:36

rows

get rows(): readonly Row[]
Returns readonly Row[]
Inherited from MlxPrefillRows.rows
- Defined in packages/inference/src/execution/prefill-rows.ts:79

supportsMixedWork

get supportsMixedWork(): boolean
Returns boolean
- Defined in packages/inference/src/execution/prefill-cohort.ts:39

tokenWeight

get tokenWeight(): number
Keep a request's prefill weight until retirement so a nearly completed request does not repeatedly attract another long prefill into its tail.

Returns number
Inherited from MlxPrefillRows.tokenWeight
- Defined in packages/inference/src/execution/prefill-rows.ts:83

Methods

admit

admit(row: Row): void
Parameters
- row: Row
Returns void
Inherited from MlxPrefillRows.admit
- Defined in packages/inference/src/execution/prefill-rows.ts:85

advance

advance(workLimit?: MlxPreparationWork): Promise<boolean>
Parameters
- OptionalworkLimit: MlxPreparationWork
Returns Promise<boolean>
Inherited from MlxPrefillRows.advance
- Defined in packages/inference/src/execution/prefill-rows.ts:131

dispose

dispose(): void
Returns void
Inherited from MlxPrefillRows.dispose
- Defined in packages/inference/src/execution/prefill-rows.ts:257