FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness


FlashAttention-2: Faster Attention with Better Parallelism


FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision


Ring Attention with Blockwise Transformers for Near-Infinite Context


Striped Attention: Faster Ring Attention for Causal Transformers


DeepSeek-V2 (Multi-head Latent Attention)


Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer LLM


Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention


Longformer: The Long-Document Transformer


Fast Transformer Decoding: Multi-Query Attention


GQA: Generalized Multi-Query Transformer Models


Sparser is Faster and Less is More: Efficient Sparse Attention


SpargeAttention: Training-free Sparse Attention


Infini-attention: Efficient Infinite Context Transformers


You Only Cache Once: Decoder-Decoder Architectures (YOCO)


Perceiver: General Perception with Iterative Attention


Perceiver IO: A General Architecture for Structured I/O


Gated Linear Attention Transformers with Hardware-Efficient Training


Griffin: Mixing Gated Linear Recurrences with Local Attention


Outrageously Large Neural Networks: Sparsely-Gated Mixture-of-Experts Layer


Switch Transformers: Scaling to Trillion Parameter Models


GShard: Scaling Giant Models with Conditional Computation


ST-MoE: Designing Stable and Transferable Sparse Expert Models


MegaBlocks: Efficient Sparse Training with Mixture-of-Experts


Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts


OLMoE: Open Mixture-of-Experts Language Models


Mixture-of-Experts Meets Instruction Tuning


Mixture-of-Depths: Dynamically allocating compute in transformers


Diffusion-LM Improves Controllable Text Generation


Structured Denoising Diffusion Models in Discrete State-Spaces (D3PM)


Discrete Diffusion Modeling by Estimating Ratios of Data Distribution (SEDD)


Simple and Effective Masked Diffusion Language Models (MDLM)


Block Diffusion: Interpolating Between Autoregressive and Diffusion LMs


Transfusion: Predict the Next Token and Diffuse Images


CodeFusion: A Pre-trained Diffusion Model for Code Generation


DPLM-2: A Multimodal Diffusion Protein Language Model


Exploring Diffusion Transformer Designs via Grafting


Mamba: Linear-Time Sequence Modeling with Selective State Spaces


Transformers are SSMs: Structured State Space Duality (Mamba-2)


Jamba: A Hybrid Transformer-Mamba Language Model


Retentive Network: A Successor to Transformer (RetNet)


Eagle and Finch: RWKV with Matrix-Valued States


Megalodon: Efficient LLM Pretraining with Unlimited Context


KAN 2.0: Kolmogorov-Arnold Networks Meet Science


Titans: Learning to Memorize at Test Time


LongRoPE2: Near-Lossless LLM Context Window Scaling


LongBench: Bilingual Multitask Benchmark for Long Context


RULER: What's the Real Context Size of Your Long-Context LLMs?


The Era of 1-bit LLMs: All LLMs are in 1.58 Bits (BitNet b1.58)


BitNet: Scaling 1-bit Transformers for Large Language Models


BitNet v2: Native 4-bit Activations with Hadamard Transformation


MatFormer: Nested Transformer for Elastic Inference


LLM2Vec: LLMs Are Secretly Powerful Text Encoders


Mixture-of-Recursions: Learning Dynamic Recursive Depths


Relaxed Recursive Transformers: Parameter Sharing with Layer-wise LoRA


Fast Inference from Transformers via Speculative Decoding


Accelerating Large Language Model Decoding with Speculative Sampling


Medusa: LLM Inference Acceleration with Multiple Decoding Heads


EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty


EAGLE-2: Faster Inference with Dynamic Draft Trees


EAGLE-3: Scaling up Inference Acceleration via Training-Time Test


SpecInfer: Tree-based Speculative Inference and Verification


REST: Retrieval-Based Speculative Decoding


LayerSkip: Early Exit Inference and Self-Speculative Decoding


Jakiro: Boosting Speculative Decoding with Decoupled Multi-Head via MoE


Scaling up Test-Time Compute with Latent Reasoning: Recurrent Depth


Large Language Monkeys: Scaling Inference Compute with Repeated Sampling


Inference Scaling Laws: Compute-Optimal Inference


Scaling of Search and Learning: A Roadmap to Reproduce o1


Training LLMs to Reason in a Continuous Latent Space (Coconut)


DeepSeek-R1: Incentivizing Reasoning via Reinforcement Learning


Mutual Reasoning Makes Smaller LLMs Stronger (rStar)


Stop Overthinking: A Survey on Efficient Reasoning for LLMs


MMLU (Massive Multitask Language Understanding)