SoK: All You Need to Know About On-Device ML Model Extraction - The Gap Between Research and Practice
This repo contains existing paper works on ML Model Extraction categorized into four distinct types of threat models, from the attacker’s and defender’s perspectives, respectively.
On-device ML is increasingly used in different applications. It brings convenience to offline tasks and avoids sending user-private data through the network. On-device ML models are valuable and may suffer from model extraction attacks from different categories. Existing studies lack a deep understanding of on-device ML model security, which creates a gap between research and practice. Our paper and this repository provide a systematization approach to classify existing model extraction attacks and defenses based on different threat models. We evaluated well-known research projects from existing work with real-world ML models and discussed their reproducibility, computation complexity, and power consumption. We identified the challenges for research projects in wide adoption in practice. We also provided directions for future research in ML model extraction security. We further evaluated well-known research projects from existing work with real-world ML models and discussed their reproducibility, computation complexity, and power consumption (more in evaluate.md)
The scope of this survey focuses only on ML model extraction or stealing and is mainly defined by the following aspects.
Attacker’s perspective
- App-based attacks: attackers assume they can gain access to the application files and they perform application de-packaging or decompiling, and extract the model files.
- Device-based attacks: attackers assume they can access the IoT devices and gain access to the memory and force a vulnerable application to launch and load ML models into memory or consistently scan the memory to wait for models to be loaded.
- Communication-based attacks: attackers can intercept communication between various memory regions and hardware architectures on an IoT device. These data can help to recover partial or complete details of ML models.
- Model-based attacks: attackers assume to be able to send (selective) input queries, and receive inference results to assess the functionality of models and fine-tune the data to send in subsequent steps and then go through the process back and forth to train substitute models.
Defender’s perspective
- App-based defense: defenders apply techniques, including encryption, obfuscation, or customized protection to model files in an app package.
- Device-based defense: defenders apply techniques, such as secure hardware, to prevent arbitrary memory access. Defenders can also customize hardware to support computation on encrypted data so that memory extraction will not reveal plaintext models.
- Communication-based defense: defenders apply data transformation, encryption, and randomization techniques to prevent side-channel information leakage and enable further calculation based on the transformed data in the memory components.
- Model-based defense: defenders apply weight obfuscation, misinformation, and differential privacy to increase the effort of attackers in training equivalent student models.
@inproceedings {nayan2024sok,
author = {Tushar Nayan and Qiming Gao and Mohammed Al Duniawi and Marcus Botacin and Selcuk Uluagac and Ruimin Sun},
title = {SoK: All You Need to Know About On-Device ML Model Extraction - The Gap Between Research and Practice},
booktitle = {33rd {USENIX} Security Symposium ({USENIX} Security 24)},
year = {2024},
url = {https://www.usenix.org/conference/usenixsecurity24/presentation/nayan},
publisher = {{USENIX} Association},
month = aug,
}
- App-based attacks
- A First Look at Deep Learning Apps on Smartphones [Paper] [Code]
- Smart App Attack: Hacking Deep Learning Models in Android Apps [Paper] [Code]
- Mind Your Weight(s): A Large-scale Study on Insufficient Machine Learning Model Protection in Mobile Apps [Paper] [Code]
- Understanding Real-world Threats to Deep Learning Models in Android Apps [Paper]
- Device-based attacks
- Communication-based attacks
- Security Analysis of Deep Neural Networks Operating in the Presence of Cache Side-Channel Attacks [Paper] [Code]
- CSI NN: Reverse Engineering of Neural Network Architectures Through Electromagnetic Side Channel [Paper]
- Cache Telepathy: Leveraging Shared Resource Attacks to Learn DNN Architectures [Paper]
- Open DNN Box by Power Side-Channel Attack [Paper]
- Reverse Engineering Convolutional Neural Networks Through Side-channel Information Leaks [Paper]
- GANRED: GAN-based Reverse Engineering of DNNs via Cache Side-Channel [Paper]
- DeepEM: Deep Neural Networks Model Recovery through EM Side-Channel Information Leakage [Paper]
- Stealing Neural Networks via Timing Side Channels [Paper]
- HuffDuff: Stealing Pruned DNNs from Sparse Accelerators [Paper]
- Hermes Attack: Steal DNN Models with Lossless Inference Accuracy [Paper]
- Leaky DNN: Stealing Deep-Learning Model Secret with GPU Context-Switching Side-Channel [Paper]
- Stealing Neural Network Models through the Scan Chain: A New Threat for ML Hardware [Paper]
- DeepSniffer: A DNN Model Extraction Framework Based on Learning Architectural Hints [Paper] [Code]
- DeepSteal: rowhammer-based side channel for ML model weight stealing [Paper] [Code]
- Model-based attacks
- ML-Doctor: Holistic Risk Assessment of Inference Attacks Against Machine Learning Models [Paper] [Code]
- Stealing Hyperparameters in Machine Learning [Paper]
- Towards Reverse-Engineering Black-Box Neural Networks [Paper] [Code]
- ActiveThief: Model Extraction Using Active Learning and Unannotated Public Data [Paper] [Code]
- ML-Stealer: Stealing Prediction Functionality of Machine Learning Models with Mere Black-Box Access [Paper]
- Knockoff Nets: Stealing Functionality of Black-Box Models [Paper] [Code]
- Simulating Unknown Target Models for Query-Efficient Black-box Attacks [Paper] [Code]
- App-based defense
- Device-based defense
- MyTEE: Own the Trusted Execution Environment on Embedded Devices [Paper] [Code]
- SANCTUARY: ARMing TrustZone with User-space Enclaves [Paper] [Code]
- DarkneTZ: Towards Model Privacy at the Edge using Trusted Execution Environments [Paper] [Code]
- Graviton: Trusted Execution Environments on GPUs [Paper]
- ShadowNet: A Secure and Efficient On-device Model Inference System for Convolutional Neural Networks [Paper]
- Communication-based defense
- ObfuNAS: A Neural Architecture Search-based DNN Obfuscation Approach [Paper] [Code]
- ShadowNet: A Secure and Efficient On-device Model Inference System for Convolutional Neural Networks [Paper]
- Slalom: Fast, Verifiable and Private Execution of Neural Networks in Trusted Hardware [Paper] [Code]
- Secure Outsourced Matrix Computation and Application to Neural Networks [Paper]
- NPUFort: a secure architecture of DNN accelerator against model inversion attack [Paper]
- NeurObfuscator: A Full-stack Obfuscation Tool to Mitigate Neural Architecture Stealing [Paper]
- NNReArch: A Tensor Program Scheduling Framework Against Neural Network Architecture Reverse Engineering [Paper]
- Model-based defense
- MindSpore [Code]
- Defending Against Model Stealing Attacks with Adaptive Misinformation [Paper] [Code]
- Prediction Poisoning: Towards Defenses Against DNN Model Stealing Attacks [Paper] [Code]
- PRADA: Protecting against DNN Model Stealing Attacks [Paper] [Code]
- SteerAdversary [Paper]
- Latent Dirichlet Allocation Model Training with Differential Privacy [Paper]
In case of feedback, suggestions, or issues, please contact the Authors.