Class QwenResidualDeltaLayer

A residual/normalization kernel between the recurrent mixer and its MLP.

Hierarchy (View Summary)

QwenDeltaLayer
- QwenResidualDeltaLayer

Index

Constructors

Properties

inputNorm mixer mlp postAttnNorm

Methods

forward forwardAttn forwardMlp

Constructors

constructor

new QwenResidualDeltaLayer(
    ...args: [
        weights: Weights,
        config: ModelConfig,
        prefix: string,
        mixer: {
            forward(
                x: MlxArray,
                cache: GatedDeltaCache,
                phase: ReadPhase,
            ): MlxArray;
        },
        mlp: { forward(x: MlxArray): MlxArray },
    ],
): QwenResidualDeltaLayer
Parameters
- ...args: [
      weights: Weights,
      config: ModelConfig,
      prefix: string,
      mixer: {
          forward(
              x: MlxArray,
              cache: GatedDeltaCache,
              phase: ReadPhase,
          ): MlxArray;
      },
      mlp: { forward(x: MlxArray): MlxArray },
  ]
Returns QwenResidualDeltaLayer
Overrides QwenDeltaLayer.constructor
- Defined in packages/inference/src/models/qwen/blocks.ts:627

Properties

`Readonly`inputNorm

inputNorm: RMSNorm

`Readonly`mixer

mixer: {
forward(x: MlxArray, cache: GatedDeltaCache, phase: ReadPhase): MlxArray;
}

`Readonly`mlp

mlp: { forward(x: MlxArray): MlxArray }

`Readonly`postAttnNorm

postAttnNorm: RMSNorm

Methods

forward

forward(x: MlxArray, cache: Cache, phase: ReadPhase): MlxArray
Parameters
- x: MlxArray
- cache: Cache
- phase: ReadPhase
Returns MlxArray
Overrides QwenDeltaLayer.forward
- Defined in packages/inference/src/models/qwen/blocks.ts:631

forwardAttn

forwardAttn(x: MlxArray, cache: Cache, phase: ReadPhase): MlxArray
Parameters
- x: MlxArray
- cache: Cache
- phase: ReadPhase
Returns MlxArray
Inherited from QwenDeltaLayer.forwardAttn
- Defined in packages/inference/src/models/qwen/blocks.ts:582

forwardMlp

forwardMlp(x: MlxArray): MlxArray
Parameters
- x: MlxArray
Returns MlxArray
Inherited from QwenDeltaLayer.forwardMlp
- Defined in packages/inference/src/models/qwen/blocks.ts:587