Glossary

Concepts

GQA — Grouped-Query Attention

Several query heads share one key/value head. Shrinks the KV cache by the grouping factor while staying close to MHA quality.

MLA — Multi-head Latent Attention

Projects keys and values into a shared low-rank latent that is cached instead of the full KV. Cache per token drops far below GQA at the same context.

DSA — DeepSeek Sparse Attention

A sparse attention stage on top of MLA: a lightning indexer selects a small token subset per query, so long-context cost grows sub-linearly.

KDA — Kimi Delta Attention

A gated delta-rule linear attention: recurrent per-channel state updated with a gated delta rule. Constant-memory per-token cost, no KV cache.

MoE — Mixture of Experts

The FFN is replaced by many expert FFNs; a router activates a small top-k subset per token. Total parameters grow, per-token compute does not.

mHC — Manifold-constrained Hyper-Connections

Replaces the plain residual add with several parallel residual streams mixed through a constrained combiner, improving gradient flow.

NoPE

No positional encoding on some attention paths (e.g. MLA Q/K in GLM-5.3-Flash): position information is not injected there.

MTP — Multi-Token Prediction

An extra prediction layer trains the model to emit several future tokens per step; used for training signal and speculative decoding.