Skip to content

brahimfarhat/Awesome-LLM-Compression

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Awesome LLM Compression

Awesome LLM compression research papers and tools to accelerate LLM training and inference.

Contents

Papers

Survey

  • A Survey on Model Compression for Large Language Models
    TACL [Paper]

  • The Cost of Compression: Investigating the Impact of Compression on Parametric Knowledge in Language Models
    EMNLP 2023 [Paper] [Code]

  • The Efficiency Spectrum of Large Language Models: An Algorithmic Survey
    Arxiv 2023 [Paper]

  • Efficient Large Language Models: A Survey
    TMLR [Paper] [GitHub Page]

  • Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems
    ICML 2024 Tutorial [Paper] [Tutorial]

  • Understanding LLMs: A Comprehensive Overview from Training to Inference
    Arxiv 2024 [Paper]

  • Faster and Lighter LLMs: A Survey on Current Challenges and Way Forward
    IJCAI 2024 (Survey Track) [Paper] [GitHub Page]

  • A Survey of Resource-efficient LLM and Multimodal Foundation Models
    Arxiv 2024 [Paper]

  • A Survey on Hardware Accelerators for Large Language Models
    Arxiv 2024 [Paper]

  • A Comprehensive Survey of Compression Algorithms for Language Models
    Arxiv 2024 [Paper]

  • A Survey on Transformer Compression
    Arxiv 2024 [Paper]

  • Model Compression and Efficient Inference for Large Language Models: A Survey
    Arxiv 2024 [Paper]

  • A Survey on Knowledge Distillation of Large Language Models
    Arxiv 2024 [Paper] [GitHub Page]

  • Efficient Prompting Methods for Large Language Models: A Survey
    Arxiv 2024 [Paper]

Quantization

🌟 Feel free to explore the subpage for LLM quantization

  • ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers
    NeurIPS 2022 [Paper] [Code (DeepSpeed)]

  • LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
    NeurIPS 2022 [Paper] [Code]

  • Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models
    NeurIPS 2022 [Paper] [Code]

  • LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models
    Arxiv 2022 [Paper]

  • SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
    ICML 2023 [Paper] [Code]

  • FlexRound: Learnable Rounding based on Element-wise Division for Post-Training Quantization
    ICML 2023 [Paper] [Code (DeepSpeed)]

  • Understanding INT4 Quantization for Transformer Models: Latency Speedup, Composability, and Failure Cases
    ICML 2023 [Paper] [Code]

  • The case for 4-bit precision: k-bit Inference Scaling Laws
    ICML 2023 [Paper]

  • GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
    ICLR 2023 [Paper] [Code]

  • PreQuant: A Task-agnostic Quantization Approach for Pre-trained Language Models
    ACL 2023 [Paper]

  • Boost Transformer-based Language Models with GPU-Friendly Sparsity and Quantization
    ACL 2023 [Paper]

  • QLoRA: Efficient Finetuning of Quantized LLMs
    NeurIPS 2023 [Paper] [Code]

  • The Quantization Model of Neural Scaling
    NeurIPS 2023 [Paper]

  • Quantized Distributed Training of Large Models with Convergence Guarantees
    ICML 2023 [Paper]

  • RPTQ: Reorder-based Post-training Quantization for Large Language Models
    Arxiv 2023 [Paper] [Code]

  • ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation
    AAAI 2024 [Paper] [Code]

  • Integer or Floating Point? New Outlooks for Low-Bit Quantization on Large Language Models
    Arxiv 2023 [Paper]

  • Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantization
    NeurIPS 2023 [Paper]

  • Compress, Then Prompt: Improving Accuracy-Efficiency Trade-off of LLM Inference with Transferable Prompt
    Arxiv 2023 [Paper]

  • AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
    MLSys 2024 (Best Paper 🏆) [Paper] [Code]

  • LLM-QAT: Data-Free Quantization Aware Training for Large Language Models
    ACL Findings 2024 [Paper] [Code]

  • SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression
    ICLR 2024 [Paper] [Code]

  • OWQ: Lessons learned from activation outliers for weight quantization in large language models
    AAAI 2024 [Paper]

  • SqueezeLLM: Dense-and-Sparse Quantization
    ICML 2024 [Paper] [Code]

  • INT2.1: Towards Fine-Tunable Quantized Large Language Models with Error Correction through Low-Rank Adaptation
    Arxiv 2023 [Paper]

  • LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuning
    ICLR 2024 [Paper]

  • INT-FP-QSim: Mixed Precision and Formats For Large Language Models and Vision Transformers
    Arxiv 2023 [Paper] [Code]

  • QIGen: Generating Efficient Kernels for Quantized Inference on Large Language Models
    Arxiv 2023 [Paper] [Code]

  • Do Emergent Abilities Exist in Quantized Large Language Models: An Empirical Study
    COLING 2024 [Paper]

  • ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formats
    Arxiv 2023 [Paper] [Code (DeepSpeed)]

  • OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantization
    ISCA 2023 [Paper]

  • NUPES : Non-Uniform Post-Training Quantization via Power Exponent Search
    Arxiv 2023 [Paper]

  • GPT-Zip: Deep Compression of Finetuned Large Language Models
    ICML 2023 Workshop ES-FoMO [Paper]

  • Generating Efficient Kernels for Quantized Inference on Large Language Models
    ICML 2023 Workshop ES-FoMO [Paper]

  • Gradient-Based Post-Training Quantization: Challenging the Status Quo
    Arxiv 2023 [Paper]

  • FineQuant: Unlocking Efficiency with Fine-Grained Weight-Only Quantization for LLMs
    Arxiv 2023 [Paper]

  • OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models
    ICLR 2024 [Paper] [Code]

  • FPTQ: Fine-grained Post-Training Quantization for Large Language Models
    Arxiv 2023 [Paper]

  • eDKM: An Efficient and Accurate Train-time Weight Clustering for Large Language Models
    IEEE Computer Architecture Letters 2023 [Paper]

  • QuantEase: Optimization-based Quantization for Language Models -- An Efficient and Intuitive Algorithm
    Arxiv 2023 [Paper]

  • Norm Tweaking: High-performance Low-bit Quantization of Large Language Models
    AAAI 2024 [Paper]

  • Understanding the Impact of Post-Training Quantization on Large-scale Language Models
    Arxiv 2023 [Paper]

  • MEMORY-VQ: Compression for Tractable Internet-Scale Memory
    NAACL 2024 [Paper]

  • Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs
    Arxiv 2023 [Paper] [Code]

  • Efficient Post-training Quantization with FP8 Formats
    MLSys 2024 [Paper] [Code (Intel® Neural Compressor)]

  • QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models
    ICLR 2024 [Paper] [Code]

  • Rethinking Channel Dimensions to Isolate Outliers for Low-bit Weight Quantization of Large Language Models
    ICLR 2024 [Paper] [Code]

  • ModuLoRA: Finetuning 3-Bit LLMs on Consumer GPUs by Integrating with Modular Quantizers
    TMLR (Featured Certification 🌟) [Paper]

  • PB-LLM: Partially Binarized Large Language Models
    ICLR 2024 [Paper] [Code]

  • Dual Grained Quantization: Efficient Fine-Grained Quantization for LLM
    Arxiv 2023 [Paper]

  • QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models
    ICLR 2024 [Paper] [Code]

  • LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models
    ICLR 2024 [Paper] [Code]

  • QFT: Quantized Full-parameter Tuning of LLMs with Affordable Resources
    Arxiv 2023 [Paper]

  • TEQ: Trainable Equivalent Transformation for Quantization of LLMs
    Arxiv 2023 [Paper] [Code (Intel® Neural Compressor)]

  • BitNet: Scaling 1-bit Transformers for Large Language Models
    Arxiv 2023 [Paper] [Code]

  • FP8-LM: Training FP8 Large Language Models
    Arxiv 2023 [Paper] [Code]

  • QUIK: Towards End-to-End 4-Bit Inference on Generative Large Language Models
    Arxiv 2023 [Paper] [Code]

  • AFPQ: Asymmetric Floating Point Quantization for LLMs
    ACL Findings 2024 [Paper] [Code]

  • AWEQ: Post-Training Quantization with Activation-Weight Equalization for Large Language Models
    Arxiv 2023 [Paper]

  • Atom: Low-bit Quantization for Efficient and Accurate LLM Serving
    MLSys 2024 [Paper] [Code]

  • QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models
    Arxiv 2023 [Paper]

  • Dissecting the Runtime Performance of the Training, Fine-tuning, and Inference of Large Language Models
    Arxiv 2023 [Paper]

  • How Does Calibration Data Affect the Post-training Pruning and Quantization of Large Language Models?
    Arxiv 2023 [Paper]

  • A Speed Odyssey for Deployable Quantization of LLMs
    Arxiv 2023 [Paper]

  • Enabling Fast 2-bit LLM on GPUs: Memory Alignment, Sparse Outlier, and Asynchronous Dequantization
    Arxiv 2023 [Paper]

  • Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing
    NeurIPS 2023 [Paper] [Code]

  • Efficient LLM Inference on CPUs
    NeurIPS 2023 on Efficient Natural Language and Speech Processing [Paper] [Code]

  • The Cost of Compression: Investigating the Impact of Compression on Parametric Knowledge in Language Models
    EMNLP Findings 2023 [Paper]

  • Zero-Shot Sharpness-Aware Quantization for Pre-trained Language Models
    EMNLP 2023 [Paper]

  • Revisiting Block-based Quantisation: What is Important for Sub-8-bit LLM Inference?
    EMNLP 2023 [Paper] [Code]

  • Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling
    EMNLP 2023 [Paper]

  • Watermarking LLMs with Weight Quantization
    EMNLP 2023 [Paper] [Code]

  • Enhancing Computation Efficiency in Large Language Models through Weight and Activation Quantization
    EMNLP 2023 [Paper]

  • LLM-FP4: 4-Bit Floating-Point Quantized Transformers
    EMNLP 2023 [Paper] [Code]

  • Agile-Quant: Activation-Guided Quantization for Faster Inference of LLMs on the Edge
    AAAI 2024 [Paper]

  • SmoothQuant+: Accurate and Efficient 4-bit Post-Training WeightQuantization for LLM
    Arxiv 2023 [Paper]

  • CBQ: Cross-Block Quantization for Large Language Models
    Arxiv 2023 [Paper]

  • ZeroQuant(4+2): Redefining LLMs Quantization with a New FP6-Centric Strategy for Diverse Generative Tasks
    Arxiv 2023 [Paper]

  • QuIP: 2-Bit Quantization of Large Language Models With Guarantees
    NeurIPS 2023 [Paper] [Code]

  • A Performance Evaluation of a Quantized Large Language Model on Various Smartphones
    Arxiv 2023 [Paper]

  • DeltaZip: Multi-Tenant Language Model Serving via Delta Compression
    Arxiv 2023 [Paper] [Code]

  • FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGA
    FPGA 2024 [Paper]

  • Extreme Compression of Large Language Models via Additive Quantization
    ICML 2024 [Paper]

  • Quantized Side Tuning: Fast and Memory-Efficient Tuning of Quantized Large Language Models
    Arxiv 2024 [Paper]

  • Inferflow: an Efficient and Highly Configurable Inference Engine for Large Language Models
    Arxiv 2024 [Paper]

  • FP6-LLM: Efficiently Serving Large Language Models Through FP6-Centric Algorithm-System Co-Design
    USENIX ATC 2024 [Paper]

  • KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
    Arxiv 2024 [Paper]

  • Can Large Language Models Understand Context?
    Arxiv 2024 [Paper]

  • EdgeQAT: Entropy and Distribution Guided Quantization-Aware Training for the Acceleration of Lightweight LLMs on the Edge
    Arxiv 2024 [Paper] [Code]

  • Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs
    Arxiv 2024 [Paper]

  • LQER: Low-Rank Quantization Error Reconstruction for LLMs
    ICML 2024 [Paper]

  • KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
    ICML 2024 [Paper] [Code]

  • BiLLM: Pushing the Limit of Post-Training Quantization for LLMs
    Arxiv 2024 [Paper] [Code]

  • QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks
    ICML 2024 [Paper] [Code]

  • L4Q: Parameter Efficient Quantization-Aware Training on Large Language Models via LoRA-wise LSQ
    Arxiv 2024 [Paper]

  • TP-Aware Dequantization
    Arxiv 2024 [Paper]

  • ApiQ: Finetuning of 2-Bit Quantized Large Language Model
    Arxiv 2024 [Paper]

  • Accurate LoRA-Finetuning Quantization of LLMs via Information Retention
    Arxiv 2024 [Paper] [Code]

  • BitDelta: Your Fine-Tune May Only Be Worth One Bit
    Arxiv 2024 [Paper] [Code]

  • QDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model Tuning
    AAAI EIW Workshop 2024 [Paper]

  • Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs
    ICML 2024 [Paper]

  • BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillation
    ACL 2024 [Paper] [Code]

  • OneBit: Towards Extremely Low-bit Large Language Models
    Arxiv 2024 [Paper]

  • DB-LLM: Accurate Dual-Binarization for Efficient LLMs
    Arxiv 2024 [Paper]

  • WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains More
    Arxiv 2024 [Paper]

  • GPTVQ: The Blessing of Dimensionality for LLM Quantization
    Arxiv 2024 [Paper] [Code]

  • APTQ: Attention-aware Post-Training Mixed-Precision Quantization for Large Language Models
    DAC 2024 [Paper]

  • A Comprehensive Evaluation of Quantization Strategies for Large Language Models
    DAC 2024 [Paper]

  • No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization
    Arxiv 2024 [Paper]

  • Evaluating Quantized Large Language Models
    Arxiv 2024 [Paper]

  • FlattenQuant: Breaking Through the Inference Compute-bound for Large Language Models with Per-tensor Quantization
    Arxiv 2024 [Paper]

  • LLM-PQ: Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantization
    Arxiv 2024 [Paper]

  • IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intact
    Arxiv 2024 [Paper]

  • On the Compressibility of Quantized Large Language Models
    Arxiv 2024 [Paper]

  • EasyQuant: An Efficient Data-free Quantization Algorithm for LLMs
    Arxiv 2024 [Paper]

  • QAQ: Quality Adaptive Quantization for LLM KV Cache
    Arxiv 2024 [Paper] [Code]

  • GEAR: An Efficient KV Cache Compression Recipefor Near-Lossless Generative Inference of LLM
    Arxiv 2024 [Paper]

  • What Makes Quantization for Large Language Models Hard? An Empirical Study from the Lens of Perturbation
    Arxiv 2024 [Paper]

  • SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression
    Arxiv 2024 [Paper] [Code]

  • AffineQuant: Affine Transformation Quantization for Large Language Models
    ICLR 2024 [Paper] [Code]

  • Oh! We Freeze: Improving Quantized Knowledge Distillation via Signal Propagation Analysis for Large Language Models
    ICLR Practical ML for Low Resource Settings Workshop 2024 [Paper]

  • Accurate Block Quantization in LLMs with Outliers
    Arxiv 2024 [Paper]

  • QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
    Arxiv 2024 [Paper] [Code]

  • Minimize Quantization Output Error with Bias Compensation
    Arxiv 2024 [Paper] [Code]

  • Cherry on Top: Parameter Heterogeneity and Quantization in Large Language Models
    Arxiv 2024 [Paper]

  • Increased LLM Vulnerabilities from Fine-tuning and Quantization
    Arxiv 2024 [Paper]

  • Quantization of Large Language Models with an Overdetermined Basis
    Arxiv 2024 [Paper]

  • How Good Are Low-bit Quantized LLaMA3 Models? An Empirical Study
    Arxiv 2024 [Paper] [Code] [Model]

  • How to Parameterize Asymmetric Quantization Ranges for Quantization-Aware Training
    Arxiv 2024 [Paper]

  • Mitigating the Impact of Outlier Channels for Language Model Quantization with Activation Regularization
    Arxiv 2024 [Paper] [Code]

  • KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization
    Arxiv 2024 [Paper]

  • When Quantization Affects Confidence of Large Language Models?
    NAACL 2024 [Paper]

  • QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving
    Arxiv 2024 [Paper] [Code]

  • Learning from Students: Applying t-Distributions to Explore Accurate and Efficient Formats for LLMs
    ICML 2024 [Paper]

  • LLM-QBench: A Benchmark Towards the Best Practice for Post-training Quantization of Large Language Models
    Arxiv 2024 [Paper] [Code]

  • SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Models
    Arxiv 2024 [Paper]

  • Combining multiple post-training techniques to achieve most efficient quantized LLMs
    Arxiv 2024 [Paper]

  • Edge Intelligence Optimization for Large Language Model Inference with Batching and Quantization
    Arxiv 2024 [Paper]

  • Unlocking Data-free Low-bit Quantization with Matrix Decomposition for KV Cache Compression
    Arxiv 2024 [Paper]

  • SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models
    Arxiv 2024 [Paper] [Code]

  • OAC: Output-adaptive Calibration for Accurate Post-training Quantization
    Arxiv 2024 [Paper]

  • PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compression
    Arxiv 2024 [Paper]

  • SpinQuant -- LLM quantization with learned rotations
    Arxiv 2024 [Paper]

  • Compressing Large Language Models using Low Rank and Low Precision Decomposition
    Arxiv 2024 [Paper] [Code]

  • Athena: Efficient Block-Wise Post-Training Quantization for Large Language Models Using Second-Order Matrix Derivative Information
    Arxiv 2024 [Paper]

  • Exploiting LLM Quantization
    Arxiv 2024 [Paper]

  • One QuantLLM for ALL: Fine-tuning Quantized LLMs Once for Efficient Deployments
    Arxiv 2024 [Paper]

  • LCQ: Low-Rank Codebook based Quantization for Large Language Models
    Arxiv 2024 [Paper]

  • LoQT: Low Rank Adapters for Quantized Training
    Arxiv 2024 [Paper] [Code]

  • CLAQ: Pushing the Limits of Low-Bit Post-Training Quantization for LLMs
    Arxiv 2024 [Paper] [Code]

  • I-LLM: Efficient Integer-Only Inference for Fully-Quantized Low-Bit Large Language Models
    Arxiv 2024 [Paper]

  • Outliers and Calibration Sets have Diminishing Effect on Quantization of Modern LLMs
    Arxiv 2024 [Paper]

  • Rotation and Permutation for Advanced Outlier Management and Efficient Quantization of LLMs
    Arxiv 2024 [Paper] [Code]

  • QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overhead
    Arxiv 2024 [Paper] [Code]

  • ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization
    Arxiv 2024 [Paper] [Code]

  • Low-Rank Quantization-Aware Training for LLMs
    Arxiv 2024 [Paper]

  • TernaryLLM: Ternarized Large Language Model
    Arxiv 2024 [Paper]

  • Examining Post-Training Quantization for Mixture-of-Experts: A Benchmark
    Arxiv 2024 [Paper] [Code]

  • Delta-CoMe: Training-Free Delta-Compression with Mixed-Precision for Large Language Models
    Arxiv 2024 [Paper]

  • QQQ: Quality Quattuor-Bit Quantization for Large Language Models
    Arxiv 2024 [Paper] [Code]

  • QTIP: Quantization with Trellises and Incoherence Processing
    Arxiv 2024 [Paper]

  • Prefixing Attention Sinks can Mitigate Activation Outliers for Large Language Model Quantization
    Arxiv 2024 [Paper]

  • Mixture of Scales: Memory-Efficient Token-Adaptive Binarization for Large Language Models
    Arxiv 2024 [Paper]

  • Tender: Accelerating Large Language Models via Tensor Decomposition and Runtime Requantization
    ISCA 2024 [Paper]

  • SDQ: Sparse Decomposed Quantization for LLM Inference
    Arxiv 2024 [Paper]

  • Attention-aware Post-training Quantization without Backpropagation
    Arxiv 2024 [Paper]

  • EDGE-LLM: Enabling Efficient Large Language Model Adaptation on Edge Devices via Layerwise Unified Compression and Adaptive Layer Tuning and Voting
    Arxiv 2024 [Paper] [Code]

  • Compensate Quantization Errors: Make Weights Hierarchical to Compensate Each Other
    Arxiv 2024 [Paper]

  • Layer-Wise Quantization: A Pragmatic and Effective Method for Quantizing LLMs Beyond Integer Bit-Levels
    Arxiv 2024 [Paper] [Code]

  • CDQuant: Accurate Post-training Weight Quantization of Large Pre-trained Models using Greedy Coordinate Descent
    Arxiv 2024 [Paper]

  • OutlierTune: Efficient Channel-Wise Quantization for Large Language Models
    Arxiv 2024 [Paper]

  • T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge
    Arxiv 2024 [Paper] [Code]

  • GPTQT: Quantize Large Language Models Twice to Push the Efficiency
    ICORIS 2024 [Paper]

  • Improving Conversational Abilities of Quantized Large Language Models via Direct Preference Alignment
    Arxiv 2024 [Paper]

  • How Does Quantization Affect Multilingual LLMs?
    Arxiv 2024 [Paper]

  • RoLoRA: Fine-tuning Rotated Outlier-free LLMs for Effective Weight-Activation Quantization
    Arxiv 2024 [Paper] [Code]

  • Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients
    Arxiv 2024 [Paper] [Code]

  • FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision
    Arxiv 2024 [Paper] [Code]

  • Accuracy is Not All You Need
    Arxiv 2024 [Paper]

  • BitNet b1.58 Reloaded: State-of-the-art Performance Also on Smaller Networks
    Arxiv 2024 [Paper]

  • LeanQuant: Accurate Large Language Model Quantization with Loss-Error-Aware Grid
    Arxiv 2024 [Paper]

  • Fast Matrix Multiplications for Lookup Table-Quantized LLMs
    Arxiv 2024 [Paper] [Code]

  • EfficientQAT: Efficient Quantization-Aware Training for Large Language Models
    Arxiv 2024 [Paper] [Code]

  • LRQ: Optimizing Post-Training Quantization for Large Language Models by Learning Low-Rank Weight-Scaling Matrices
    Arxiv 2024 [Paper] [Code]

  • Exploring Quantization for Efficient Pre-Training of Transformer Language Models
    Arxiv 2024 [Paper] [Code]

  • Spectra: A Comprehensive Study of Ternary, Quantized, and FP16 Language Models
    Arxiv 2024 [Paper] [Code]

  • Mamba-PTQ: Outlier Channels in Recurrent Large Language Models
    Arxiv 2024 [Paper]

  • PQCache: Product Quantization-based KVCache for Long Context LLM Inference
    Arxiv 2024 [Paper]

  • Compensate Quantization Errors+: Quantized Models Are Inquisitive Learners
    Arxiv 2024 [Paper]

  • Accurate and Efficient Fine-Tuning of Quantized Large Language Models Through Optimal Balance
    Arxiv 2024 [Paper] [Code]

  • STBLLM: Breaking the 1-Bit Barrier with Structured Binary LLMs
    Arxiv 2024 [Paper]

  • Advancing Multimodal Large Language Models with Quantization-Aware Scale Learning for Efficient Adaptation
    ACM MM 2024 [Paper]

  • ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models
    Arxiv 2024 [Paper]

  • MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models
    Arxiv 2024 [Paper] [Code (Marlin)] [Code (Sparse Marlin)]

  • Matmul or No Matmal in the Era of 1-bit LLMs
    Arxiv 2024 [Paper]

  • MobileQuant: Mobile-friendly Quantization for On-device Language Models
    Arxiv 2024 [Paper] [Code]

Pruning and Sparsity

🌟 Feel free to explore the subpage for LLM pruning

  • The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers
    ICLR 2023 [Paper]

  • Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time
    ICML 2023 [Paper] [Code]

  • LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation
    ICML 2023 [Paper] [Code]

  • LLM-Pruner: On the Structural Pruning of Large Language Models
    NeurIPS 2023 [Paper] [Code]

  • ZipLM: Inference-Aware Structured Pruning of Language Models
    NeurIPS 2023 [Paper] [Code]

  • H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
    NeurIPS 2023 [Paper] [Code]

  • Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time
    NeurIPS 2023 [Paper]

  • The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Matter
    NeurIPS 2023 [Paper] [Code]

  • Learning to Compress Prompts with Gist Tokens
    NeurIPS 2023 [Paper]

  • Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformers
    NeurIPS 2023 [Paper]

  • Prune and Tune: Improving Efficient Pruning Techniques for Massive Language Models
    ICLR 2023 TinyPapers [Paper]

  • SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot
    ICML 2023 [Paper] [Code]

  • AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning
    ICLR 2023 [Paper]

  • Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion Scale
    ACL 2023 [Paper] [Code]

  • Structured Pruning for Efficient Generative Pre-trained Language Models
    ACL 2023 [Paper]

  • A Simple and Effective Pruning Approach for Large Language Models
    ICLR 2024 [Paper] [Code]

  • Pruning Meets Low-Rank Parameter-Efficient Fine-Tuning
    ACL Findings 2024 [Paper]

  • Structural pruning of large language models via neural architecture search
    AutoML 2023 [Paper]

  • Pruning Large Language Models via Accuracy Predictor
    ICASSP 2024 [Paper]

  • Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity
    VLDB 2024 [Paper] [Cde]

  • Compressing LLMs: The Truth is Rarely Pure and Never Simple
    ICLR 2024 [Paper]

  • Pruning Small Pre-Trained Weights Irreversibly and Monotonically Impairs "Difficult" Downstream Tasks in LLMs
    ICML 2024 [Paper] [Code]

  • Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs
    ICLR 2024 [Paper]

  • Compresso: Structured Pruning with Collaborative Prompting Learns Compact Large Language Models
    Arxiv 2023 [Paper] [Code]

  • Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity
    Arxiv 2023 [Paper] [Code]

  • Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning
    Arxiv 2023 [Paper] [Code]

  • Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs
    ICLR 2024 [Paper] [Code]

  • One-Shot Sensitivity-Aware Mixed Sparsity Pruning for Large Language Models
    ICASSP 2024 [Paper]

  • Survival of the Most Influential Prompts: Efficient Black-Box Prompt Search via Clustering and Pruning
    EMNLP 2023 Findings [Paper]

  • The Cost of Compression: Investigating the Impact of Compression on Parametric Knowledge in Language Models
    EMNLP Findings 2023 [Paper]

  • Divergent Token Metrics: Measuring degradation to prune away LLM components -- and optimize quantization
    Arxiv 2023 [Paper]

  • LoRAShear: Efficient Large Language Model Structured Pruning and Knowledge Recovery
    Arxiv 2023 [Paper]

  • ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models
    Arxiv 2023 [Paper]

  • E-Sparse: Boosting the Large Language Model Inference through Entropy-based N:M Sparsity
    Arxiv 2023 [Paper]

  • Beyond Size: How Gradients Shape Pruning Decisions in Large Language Models
    Arxiv 2023 [Paper] [Code]

  • On the Impact of Calibration Data in Post-training Quantization and Pruning
    ACL 2024 [Paper]

  • BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocation
    OpenReview [Paper] [Code]

  • PUSHING GRADIENT TOWARDS ZERO: A NOVEL PRUNING METHOD FOR LARGE LANGUAGE MODELS
    OpenReview 2023 [Paper]

  • An Efficient Plug-and-Play Post-Training Pruning Strategy in Large Language Models
    Preprints 2023 [Paper]

  • Lighter, yet More Faithful: Investigating Hallucinations in Pruned Large Language Models for Abstractive Summarization
    Arxiv 2023 [Paper] [Code]

  • LORAPRUNE: PRUNING MEETS LOW-RANK PARAMETER-EFFICIENT FINE-TUNING
    Arxiv 2023 [Paper]

  • Mini-GPTs: Efficient Large Language Models through Contextual Pruning
    Arxiv 2023 [Paper] [Code]

  • The LLM Surgeon
    Arxiv 2023 [Paper]

  • Fluctuation-based Adaptive Structured Pruning for Large Language Models
    AAAI 2024 [Paper]

  • How to Prune Your Language Model: Recovering Accuracy on the "Sparsity May Cry'' Benchmark
    CPAL 2024 [Paper]

  • PERP: Rethinking the Prune-Retrain Paradigm in the Era of LLMs
    Arxiv 2023 [Paper]

  • Fast and Optimal Weight Update for Pruned Large Language Models
    Arxiv 2024 [Paper]

  • APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference
    Arxiv 2024 [Paper]

  • Scaling Sparse Fine-Tuning to Large Language Models
    Arxiv 2024 [Paper]

  • SliceGPT: Compress Large Language Models by Deleting Rows and Columns
    ICLR 2024 [Paper] [Code]

  • Shortened LLaMA: A Simple Depth Pruning for Large Language Models
    Arxiv 2024 [Paper]

  • Everybody Prune Now: Structured Pruning of LLMs with only Forward Passes
    Arxiv 2024 [Paper] [Code]

  • NutePrune: Efficient Progressive Pruning with Numerous Teachers for Large Language Models
    Arxiv 2024 [Paper]

  • LaCo: Large Language Model Pruning via Layer Collapse
    Arxiv 2024 [Paper]

  • Why Lift so Heavy? Slimming Large Language Models by Cutting Off the Layers
    Arxiv 2024 [Paper]

  • EBFT: Effective and Block-Wise Fine-Tuning for Sparse LLMs
    Arxiv 2024 [Paper] [Code]

  • Data-free Weight Compress and Denoise for Large Language Models
    Arxiv 2024 [Paper]

  • Gradient-Free Adaptive Global Pruning for Pre-trained Language Models
    Arxiv 2024 [Paper]

  • ShortGPT: Layers in Large Language Models are More Redundant Than You Expect
    Arxiv 2024 [Paper]

  • LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models
    Arxiv 2024 [Paper] [Code]

  • Compressing Large Language Models by Streamlining the Unimportant Layer
    Arxiv 2024 [Paper]

  • LoRAP: Transformer Sub-Layers Deserve Differentiated Structured Compression for Large Language Models
    Arxiv 2024 [Paper]

  • Shears: Unstructured Sparsity with Neural Low-rank Adapter Search
    NAACL 2024 [Paper]

  • Eigenpruning
    NAACL 2024 Abstract [Paper]

  • OpenBA-V2: Reaching 77.3% High Compression Ratio with Fast Multi-Stage Pruning
    Arxiv 2024 [Paper]

  • Pruning as a Domain-specific LLM Extractor
    NAACL 2024 Findings [Paper] [Code]

  • Differentiable Model Scaling using Differentiable Topk
    ICML 2024 [Paper]

  • COPAL: Continual Pruning in Large Language Generative Models
    ICML 2024 [Paper]

  • Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for Large Language Models
    ICML 2024 [Paper] [Code]

  • Feature-based Low-Rank Compression of Large Language Models via Bayesian Optimization
    ACL Findings 2024 [Paper]

  • Surgical Feature-Space Decomposition of LLMs: Why, When and How?
    ACL 2024 [Paper]

  • Pruning Large Language Models to Intra-module Low-rank Architecture with Transitional Activations
    ACL Findings 2024 [Paper]

  • Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruning
    ACL Findings 2024 [Paper] [Code]

  • Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
    ICML 2024 [Paper] [Code]

  • MoreauPruner: Robust Pruning of Large Language Models against Weight Perturbations
    Arxiv 2024 [Paper] [Code]

  • ALPS: Improved Optimization for Highly Sparse One-Shot Pruning for Large Language Models
    Arxiv 2024 [Paper]

  • HiP Attention: Sparse Sub-Quadratic Attention with Hierarchical Attention Pruning
    Arxiv 2024 [Paper]

  • Optimization-based Structural Pruning for Large Language Models without Back-Propagation
    Arxiv 2024 [Paper]

  • BlockPruner: Fine-grained Pruning for Large Language Models
    Arxiv 2024 [Paper] [Code]

  • Rethinking Pruning Large Language Models: Benefits and Pitfalls of Reconstruction Error Minimization
    Arxiv 2024 [Paper]

  • RankAdaptor: Hierarchical Dynamic Low-Rank Adaptation for Structural Pruned LLMs
    Arxiv 2024 [Paper]

  • What Matters in Transformers? Not All Attention is Needed
    Arxiv 2024 [Paper] [Code]

  • Pruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Merging
    Arxiv 2024 [Paper]

  • ShadowLLM: Predictor-based Contextual Sparsity for Large Language Models
    Arxiv 2024 [Paper] [Code]

  • Finding Transformer Circuits with Edge Pruning
    Arxiv 2024 [Paper] [Code]

  • Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs
    Arxiv 2024 [Paper] [Code]

  • MINI-LLM: Memory-Efficient Structured Pruning for Large Language Models
    Arxiv 2024 [Paper]

  • Reconstruct the Pruned Model without Any Retraining
    Arxiv 2024 [Paper]

  • A deeper look at depth pruning of LLMs
    ICML TF2M Workshop 2024 [Paper] [Code]

  • Greedy Output Approximation: Towards Efficient Structured Pruning for LLMs Without Retraining
    Arxiv 2024 [Paper]

  • Pruning Large Language Models with Semi-Structural Adaptive Sparse Training
    Arxiv 2024 [Paper]

  • A Convex-optimization-based Layer-wise Post-training Pruner for Large Language Models
    Arxiv 2024 [Paper]

  • ThinK: Thinner Key Cache by Query-Driven Pruning
    Arxiv 2024 [Paper]

  • LLM-Barber: Block-Aware Rebuilder for Sparsity Mask in One-Shot for Large Language Models
    Arxiv 2024 [Paper] [Code]

  • LLM Pruning and Distillation in Practice: The Minitron Approach
    Arxiv 2024 [Paper] [Models]

  • Training-Free Activation Sparsity in Large Language Models
    Arxiv 2024 [Paper]

  • PAT: Pruning-Aware Tuning for Large Language Models
    Arxiv 2024 [Paper] [Code]

Distillation

  • Lifting the Curse of Capacity Gap in Distilling Language Models
    ACL 2023 [Paper] [Code]

  • Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step
    ACL 2023 [Paper]

  • Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes
    ACL 2023 [Paper]

  • SCOTT: Self-Consistent Chain-of-Thought Distillation
    ACL 2023 [Paper]

  • DISCO: Distilling Counterfactuals with Large Language Models
    ACL 2023 [Paper] [Code]

  • LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions
    Arxiv 2023 [Paper] [Code]

  • How To Train Your (Compressed) Large Language Model
    Arxiv 2023 [Paper]

  • The False Promise of Imitating Proprietary LLMs
    Arxiv 2023 [Paper]

  • GPT4All: Training an Assistant-style Chatbot with Large Scale Data Distillation from GPT-3.5-Turbo
    Arxiv 2023 [Paper] [Code]

  • PaD: Program-aided Distillation Specializes Large Models in Reasoning
    Arxiv 2023 [Paper]

  • MiniLLM: Knowledge Distillation of Large Language Models
    ICLR 2024 [Paper] [Code]

  • On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
    ICLR 2024 [Paper]

  • GKD: Generalized Knowledge Distillation for Auto-regressive Sequence Models
    ICLR 2024 [Paper]

  • Chain-of-Thought Prompt Distillation for Multimodal Named Entity and Multimodal Relation Extraction
    Arxiv 2023 [Paper]

  • Task-agnostic Distillation of Encoder-Decoder Language Models
    Arxiv 2023 [Paper]

  • Sci-CoT: Leveraging Large Language Models for Enhanced Knowledge Distillation in Small Models for Scientific QA
    Arxiv 2023 [Paper]

  • Baby Llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty
    CoNLL 2023 [Paper] [Code]

  • Can a student Large Language Model perform as well as it's teacher?
    Arxiv 2023 [Paper]

  • Multistage Collaborative Knowledge Distillation from Large Language Models
    ACL 2024 [Paper] [Code]

  • Lion: Adversarial Distillation of Closed-Source Large Language Model
    EMNLP 2023 [Paper] [Code]

  • MCC-KD: Multi-CoT Consistent Knowledge Distillation
    EMNLP 2023 [Paper]

  • PromptMix: A Class Boundary Augmentation Method for Large Language Model Distillation
    EMNLP 2023 [Paper]

  • YODA: Teacher-Student Progressive Learning for Language Models
    Arxiv 2023 [Paper]

  • Knowledge Fusion of Large Language Models
    ICLR 2024 [Paper] [Code]

  • Knowledge Distillation for Closed-Source Language Models
    Arxiv 2024 [Paper]

  • TinyLLM: Learning a Small Student from Multiple Large Language Models
    Arxiv 2024 [Paper]

  • Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs
    Arxiv 2024 [Paper]

  • Revisiting Knowledge Distillation for Autoregressive Language Models
    ACL 2024 [Paper]

  • Sinkhorn Distance Minimization for Knowledge Distillation
    COLING 2024 [Paper]

  • Divide-or-Conquer? Which Part Should You Distill Your LLM?
    Arxiv 2024 [Paper]

  • Learning to Maximize Mutual Information for Chain-of-Thought Distillation
    ACL 2024 Findings [Paper]

  • DistiLLM: Towards Streamlined Distillation for Large Language Models
    ICML 2024 [Paper] [Code]

  • Efficiently Distilling LLMs for Edge Applications
    NAACL 2024 [Paper]

  • Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models
    Arxiv 2024 [Paper]

  • Distilling Algorithmic Reasoning from LLMs via Explaining Solution Programs
    Arxiv 2024 [Paper]

  • Compact Language Models via Pruning and Knowledge Distillation
    Arxiv 2024 [Paper] [Code]

  • LLM Pruning and Distillation in Practice: The Minitron Approach
    Arxiv 2024 [Paper] [Models]

  • The Mamba in the Llama: Distilling and Accelerating Hybrid Models
    Arxiv 2024 [Paper]

Efficient Prompting

  • Did You Read the Instructions? Rethinking the Effectiveness of Task Definitions in Instruction Learning
    ACL 2023 [Paper] [Code]

  • Batch Prompting: Efficient Inference with Large Language Model APIs
    EMNLP 2023 [Paper] [Code]

  • Adapting Language Models to Compress Contexts
    EMNLP 2023 [Paper] [Code]

  • Compressing Context to Enhance Inference Efficiency of Large Language Models
    EMNLP 2023 [Paper] [Code]

  • LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models
    EMNLP 2023 [Paper] [Code]

  • Vector-Quantized Prompt Learning for Paraphrase Generation
    EMNLP 2023 Findings [Paper]

  • Efficient Prompting via Dynamic In-Context Learning
    Arxiv 2023 [Paper]

  • Learning to Compress Prompts with Gist Tokens
    NeurIPS 2023 [Paper] [Code]

  • In-context Autoencoder for Context Compression in a Large Language Model
    ICLR 2024 [Paper]

  • Discrete Prompt Compression with Reinforcement Learning
    Arxiv 2023 [Paper] [Code]

  • BatchPrompt: Accomplish more with less
    Arxiv 2023 [Paper]

  • (Dynamic) Prompting might be all you need to repair Compressed LLMs
    Arxiv 2023 [Paper]

  • RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation
    Arxiv 2023 [Paper] [Code]

  • LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression
    ACL 2023 [Paper] [Code]

  • Extending Context Window of Large Language Models via Semantic Compression
    Arxiv 2023 [Paper]

  • Boosting LLM Reasoning: Push the Limits of Few-shot Learning with Reinforced In-Context Pruning
    Arxiv 2023 [Paper]

  • The Impact of Reasoning Step Length on Large Language Models
    ACL 2024 Findings [Paper]

  • Compressed Context Memory For Online Language Model Interaction
    ICLR 2024 [Paper] [Code]

  • Learning to Compress Prompt in Natural Language Formats
    Arxiv 2024 [Paper]

  • Say More with Less: Understanding Prompt Learning Behaviors through Gist Compression
    Arxiv 2024 [Paper] [Code]

  • StreamingDialogue: Prolonged Dialogue Learning via Long Context Compression with Minimal Losses
    Arxiv 2024 [Paper]

  • LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression
    Arxiv 2024 [Paper] [Code]

  • PCToolkit: A Unified Plug-and-Play Prompt Compression Toolkit of Large Language Models
    Arxiv 2024 [Paper] [Code]

  • PROMPT-SAW: Leveraging Relation-Aware Graphs for Textual Prompt Compression
    Arxiv 2024 [Paper]

  • Prompts As Programs: A Structure-Aware Approach to Efficient Compile-Time Prompt Optimization
    Arxiv 2024 [Paper] [Code]

  • Adapting LLMs for Efficient Context Processing through Soft Prompt Compression
    Arxiv 2024 [Paper]

  • Compressing Long Context for Enhancing RAG with AMR-based Concept Distillation
    Arxiv 2024 [Paper]

  • Unifying Demonstration Selection and Compression for In-Context Learning
    Arxiv 2024 [Paper]

  • SelfCP: Compressing Long Prompt to 1/12 Using the Frozen Large Language Model Itself
    Arxiv 2024 [Paper]

  • Fundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Models
    Arxiv 2024 [Paper]

  • QUITO: Accelerating Long-Context Reasoning through Query-Guided Context Compression
    Arxiv 2024 [Paper] [Code]

  • 500xCompressor: Generalized Prompt Compression for Large Language Models
    Arxiv 2024 [Paper]

Other

  • FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
    Arxiv 2022 [Paper]

  • TensorGPT: Efficient Compression of the Embedding Layer in LLMs based on the Tensor-Train Decomposition
    Arxiv 2023 [Paper]

  • Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformers
    Arxiv 2023 [Paper]

  • SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inference
    Arxiv 2023 [Paper]

  • Scaling In-Context Demonstrations with Structured Attention
    Arxiv 2023 [Paper]

  • Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline
    Arxiv 2023 [Paper] [Code]

  • CPET: Effective Parameter-Efficient Tuning for Compressed Large Language Models
    Arxiv 2023 [Paper]

  • Ternary Singular Value Decomposition as a Better Parameterized Form in Linear Mapping
    Arxiv 2023 [Paper]

  • LLMCad: Fast and Scalable On-device Large Language Model Inference
    Arxiv 2023 [Paper]

  • vLLM: Efficient Memory Management for Large Language Model Serving with PagedAttention
    Arxiv 2023 [Paper]

  • LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models
    Arxiv 2023 [Paper] [Code]

  • LORD: Low Rank Decomposition Of Monolingual Code LLMs For One-Shot Compression
    Arxiv 2023 [Paper] [Code]

  • Mixture of Tokens: Efficient LLMs through Cross-Example Aggregation
    Arxiv 2023 [Paper]

  • Efficient Streaming Language Models with Attention Sinks
    Arxiv 2023 [Paper] [Code]

  • Efficient Large Language Models Fine-Tuning On Graphs
    Arxiv 2023 [Paper]

  • SparQ Attention: Bandwidth-Efficient LLM Inference
    Arxiv 2023 [Paper]

  • Rethinking Compression: Reduced Order Modelling of Latent Features in Large Language Models
    Arxiv 2023 [Paper]

  • PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU
    Arxiv 2023 [Paper] [Code]

  • Dataset Quantization
    ICCV 2023 [Paper] [Code]

  • Text Alignment Is An Efficient Unified Model for Massive NLP Tasks
    NeurIPS 2023 [Paper] [Code]

  • Context Compression for Auto-regressive Transformers with Sentinel Tokens
    EMNLP 2023 [Paper] [Code]

  • TCRA-LLM: Token Compression Retrieval Augmented Large Language Model for Inference Cost Reduction
    EMNLP 2023 Findings [Paper]

  • Retrieval-based Knowledge Transfer: An Effective Approach for Extreme Large Language Model Compression
    EMNLP 2023 Findings [Paper]

  • FFSplit: Split Feed-Forward Network For Optimizing Accuracy-Efficiency Trade-off in Language Model Inference
    Arxiv 2024 [Paper]

  • LoMA: Lossless Compressed Memory Attention
    Arxiv 2024 [Paper]

  • Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
    Arxiv 2024 [Paper] [Code]

  • BiTA: Bi-Directional Tuning for Lossless Acceleration in Large Language Models
    Arxiv 2024 [Paper] [Code]

  • CompactifAI: Extreme Compression of Large Language Models using Quantum-Inspired Tensor Networks
    Arxiv 2024 [Paper]

  • MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases
    ICML 2024 [Paper] [Code]

  • BAdam: A Memory Efficient Full Parameter Training Method for Large Language Models
    Arxiv 2024 [Paper] [Code]

  • NoMAD-Attention: Efficient LLM Inference on CPUs Through Multiply-add-free Attention
    Arxiv 2024 [Paper]

  • Not all Layers of LLMs are Necessary during Inference
    Arxiv 2024 [Paper]

  • GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
    Arxiv 2024 [Paper]

  • Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference
    Arxiv 2024 [Paper]

  • Smart-Infinity: Fast Large Language Model Training using Near-Storage Processing on a Real System
    HPCA 2024 [Paper]

  • Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference
    MLSys 2024 [Paper]

  • ALoRA: Allocating Low-Rank Adaptation for Fine-tuning Large Language Models
    Arxiv 2024 [Paper]

  • Parameter Efficient Quasi-Orthogonal Fine-Tuning via Givens Rotation
    Arxiv 2024 [Paper]

  • Training LLMs over Neurally Compressed Text
    Arxiv 2024 [Paper]

  • TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding
    Arxiv 2024 [Paper] [Code]

  • SnapKV: LLM Knows What You are Looking for Before Generation
    Arxiv 2024 [Paper] [Code]

  • Characterizing the Accuracy - Efficiency Trade-off of Low-rank Decomposition in Language Models
    Arxiv 2024 [Paper]

  • KV-Runahead: Scalable Causal LLM Inference by Parallel Key-Value Cache Generation
    ICML 2024 [Paper]

  • Token-wise Influential Training Data Retrieval for Large Language Models
    ACL 2024 [Paper] [Code]

  • PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference
    ACL 2024 [Paper]

  • ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification
    Arxiv 2024 [Paper]

  • MiniCache: KV Cache Compression in Depth Dimension for Large Language Models
    Arxiv 2024 [Paper]

  • Basis Selection: Low-Rank Decomposition of Pretrained Large Language Models for Target Applications
    Arxiv 2024 [Paper]

  • PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling
    Arxiv 2024 [Paper]

  • A Simple and Effective L2 Norm-Based Strategy for KV Cache Compression
    Arxiv 2024 [Paper]

  • LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inference
    Arxiv 2024 [Paper]

  • RazorAttention: Efficient KV Cache Compression Through Retrieval Heads
    Arxiv 2024 [Paper]

  • AdaCoder: Adaptive Prompt Compression for Programmatic Visual Question Answering
    Arxiv 2024 [Paper]

  • Palu: Compressing KV-Cache with Low-Rank Projection
    Arxiv 2024 [Paper] [Code]

  • Finch: Prompt-guided Key-Value Cache Compression
    Arxiv 2024 [Paper]

  • Zero-Delay QKV Compression for Mitigating KV Cache and Network Bottlenecks in LLM Inference
    Arxiv 2024 [Paper]

  • CaM: Cache Merging for Memory-efficient LLMs Inference
    ICML 2024 [Paper] [Code]

  • Eigen Attention: Attention in Low-Rank Space for KV Cache Compression
    Arxiv 2024 [Paper] [Code]

  • MoDeGPT: Modular Decomposition for Large Language Model Compression
    Arxiv 2024 [Paper]

Tools

  • BMCook: Model Compression for Big Models [Code]

  • llama.cpp: Inference of LLaMA model in pure C/C++ [Code]

  • LangChain: Building applications with LLMs through composability [Code]

  • GPTQ-for-LLaMA: 4 bits quantization of LLaMA using GPTQ [Code]

  • Alpaca-CoT: An Instruction Fine-Tuning Platform with Instruction Data Collection and Unified Large Language Models Interface [Code]

  • vllm: A high-throughput and memory-efficient inference and serving engine for LLMs [Code]

  • LLaMA Efficient Tuning: Fine-tuning LLaMA with PEFT (PT+SFT+RLHF with QLoRA) [Code]

  • gpt-fast: Simple and efficient pytorch-native transformer text generation in <1000 LOC of python. [Code]

  • Efficient-Tuning-LLMs: (Efficient Finetuning of QLoRA LLMs). QLoRA, LLama, bloom, baichuan-7B, GLM [Code]

  • bitsandbytes: 8-bit CUDA functions for PyTorch [Code]

  • ExLlama: A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights. [Code]

  • lit-gpt: Hackable implementation of state-of-the-art open-source LLMs based on nanoGPT. Supports flash attention, 4-bit and 8-bit quantization, LoRA and LLaMA-Adapter fine-tuning, pre-training. [Code]

  • Lit-LLaMA: Implementation of the LLaMA language model based on nanoGPT. Supports flash attention, Int8 and GPTQ 4bit quantization, LoRA and LLaMA-Adapter fine-tuning, pre-training. [Code]

  • lama.onnx: LLaMa/RWKV onnx models, quantization and testcase [Code]

  • fastLLaMa: An experimental high-performance framework for running Decoder-only LLMs with 4-bit quantization in Python using a C/C++ backend. [Code]

  • Sparsebit: A model compression and acceleration toolbox based on pytorch. [Code]

  • llama2.c: Inference Llama 2 in one file of pure C [Code]

  • Megatron-LM: Ongoing research training transformer models at scale [Code]

  • ggml: Tensor library for machine learning [Code]

  • LLamaSharp: C#/.NET binding of llama.cpp, including LLaMa/GPT model inference and quantization, ASP.NET core integration and UI [Code]

  • rwkv.cpp: NT4/INT5/INT8 and FP16 inference on CPU for RWKV language model [Code]

  • Can my GPU run this LLM?: Calculate GPU memory requirement & breakdown for training/inference of LLM models. Supports ggml/bnb quantization [Code]

  • TinyChatEngine: On-Device LLM Inference Library [Code]

  • TensorRT-LLM: TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. [Code]

  • IntLLaMA: A fast and light quantization solution for LLaMA [Code]

  • EasyLLM: Built upon Megatron-Deepspeed and HuggingFace Trainer, EasyLLM has reorganized the code logic with a focus on usability. While enhancing usability, it also ensures training efficiency [Code]

  • GreenBit LLaMA: Advanced Ultra-Low Bitrate Compression Techniques for the LLaMA Family of LLMs [Code]

  • Intel® Neural Compressor: An open-source Python library supporting popular model compression techniques on all mainstream deep learning frameworks (TensorFlow, PyTorch, ONNX Runtime, and MXNet) [Code]

  • LLM-Viewer: Analyze the inference of Large Language Models (LLMs). Analyze aspects like computation, storage, transmission, and hardware roofline model in a user-friendly interface. [Code]

  • LLaMA3-Quantization: A repository dedicated to evaluating the performance of quantizied LLaMA3 using various quantization methods. [Code]

  • LLamaSharp: A C#/.NET library to run LLM models (🦙LLaMA/LLaVA) on your local device efficiently. [Code]

  • Green-bit-LLM: A toolkit for fine-tuning, inferencing, and evaluating GreenBitAI's LLMs. [Code] [Model]

  • Bitorch Engine: Streamlining AI with Open-Source Low-Bit Quantization. [Code]

  • llama-zip: LLM-powered lossless compression tool [Code]

  • LLaMA-Factory: Unify Efficient Fine-Tuning of 100+ LLMs [Code]

  • LLMC: A tool designed for LLM Compression. [Code]

  • BitBLAS: BitBLAS is a library to support mixed-precision matrix multiplications, especially for quantized LLM deployment. [Code]

  • AutoFP8: Open-source FP8 quantization library for producing compressed checkpoints for running in vLLM [Code]

  • AutoGGUF: automatically quant GGUF models [Code]

  • Transformer Compression: For releasing code related to compression methods for transformers, accompanying our publications [Code]

Contributing

This is an active repository and your contributions are always welcome! Before you add papers/tools into the awesome list, please make sure that:

  • The paper or tools is related to Large Language Models (LLMs). If the compression algorithms or tools are only evaluated on small-scale language models (e.g., BERT), they should not be included in the list.
  • The paper should be inserted in the correct position in chronological order (publication/arxiv release time).
  • The link to [Paper] should be the arxiv page, not the pdf page if this is a paper posted on arxiv.
  • If the paper is accpeted, please use the correct publication venue instead of arxiv

Thanks again for all the awesome contributors to this list!

Star History

Star History Chart

About

Awesome LLM compression research papers and tools.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published