Mamba-2: SSM and the attention mechanism

We continue our exploration of State Space Model–based architectures and move on to Mamba-2 — an architecture that evolved from the idea of Structured State-Space Duality (SSD), which shows that SSMs and the attention mechanism can be viewed as two perspectives on the same structure. This approach combines the strengths of both directions: the efficiency and linear scalability of SSMs with the expressiveness and flexibility of attention mechanisms.

We will examine how the new Mamba-2 layer is structured, how it differs from the first version, and how SSD allows computations to be reduced to matrix operations that are highly optimized for GPUs. It is precisely this design that makes Mamba-2 several times faster than Mamba-1 while remaining competitive with transformers in language modeling tasks.