Mamba Explained: A New Paradigm for Efficient Long-Sequence Modeling Based on State Space Models

Core Claim

Mamba, introduced by Albert Gu and Tri Dao, is the first SSM-based architecture scaled to large language model (LLM) scale. Its key innovation is a hardware-aware selective mechanism that dynamically gates state updates per token—retaining contextual expressivity while eliminating Transformer’s quadratic complexity bottleneck.

Key Technical Features

• Computational Complexity: For sequence length N, Mamba achieves O(N) training/inference time and memory usage, versus O(N²) for Transformers; at 16K context, Mamba-3B is 5× faster and uses 2.8× less GPU memory than Llama-2-3B.
• Selective Scan Mechanism: Replaces fixed-parameter convolution in classical SSMs with input-dependent gating of state transitions—endowing the model with attention-like context sensitivity while preserving causality and parallel trainability.
• Hardware-Aware Optimization: Leverages fused CUDA kernels (via mamba-ssm library), eliminates redundant memory copies and kernel launch overhead, achieving 125 tokens/sec (FP16) on A100 for Mamba-1.3B at 8K sequence length—significantly outperforming comparably sized Transformers.

Empirical Performance

• On language modeling benchmarks (The Pile, WikiText), Mamba-3B matches Llama-2-3B’s perplexity; it shows up to +12.7% relative improvement on long-range tasks including NarrativeQA (long-document QA) and HumanEval (code completion).
• Open-source implementation is available on Hugging Face (state-spaces/mamba) and GitHub (https://github.com/state-spaces/mamba), supporting PyTorch 2.0+ and FlashAttention integration; the foundational paper is published on arXiv:2312.00752 (Mamba: Linear-Time Sequence Modeling with Selective State Spaces).