The treatment of module base (no .weight suffix) under policy. The trellis axis is the ROTATED axis: R1 acts on the input dim of gate/up and on the output dim of down_proj, so the coded sequence runs along the direction the Hadamard gaussianized.
base
.weight
policy
The treatment of module
base(no.weightsuffix) underpolicy. The trellis axis is the ROTATED axis: R1 acts on the input dim of gate/up and on the output dim of down_proj, so the coded sequence runs along the direction the Hadamard gaussianized.