The attention mechanism, worked out by hand
Queries, keys and values with real numbers. Why the scaling factor is sqrt(d_k), what multi-head attention buys, and how causal masking works.
Tag
Every article tagged Deep Learning. For a structured path, browse Learn.
Queries, keys and values with real numbers. Why the scaling factor is sqrt(d_k), what multi-head attention buys, and how causal masking works.
A working decoder-only transformer in 120 lines. Pre-norm, residual streams, why the MLP is 4x wide, weight tying, stable initialisation.