Why does Linear(3, 2) store a (2, 3) matrix?
The argument order and the weight shape looked contradictory when I connected the feed-forward layers.
The intuition
The arguments describe a transformation from 3 inputs to 2 outputs. Each output needs three weights, so there are two rows of three weights. With row-vector inputs, forward uses the transpose.
(..., 3) @ (3, 2) -> (..., 2)Writing out one multiplication was more useful than memorizing the convention. It also made the two expansion projections and the contraction projection in SwiGLU much easier to check.
Related reading · PyTorch: Linear