evaluate.md

SoK: All You Need to Know About On-Device ML Model Extraction - The Gap Between Research and Practice

---

This repository contains the open-source projects we evaluated in our paper: SoK: All You Need to Know About On-Device ML Model Extraction - The Gap Between Research and Practice. Our goal is to identify challenges in applying these projects to real-world ML models in practice.

---

This repo contains the following:

Models – a folder containing several sample ML models. A larger collection of real-world ML models can be shared by request for academic purposes.

Evaluation – the link to the original PCM repository, our README file giving an example of using PCM to measure the power consumption of a project.

ModelXRay – the original code repository of ModelXRay, along with the author's original README file which we used for evaluation on our model files.

AES - TFSecured – the original code repository of TFSecured, along with the author's original README file which we used for evaluation on our model files.

ShadowNet – the original code repository of ShadowNet, along with the author's original README file which we used for evaluation on our model files.

DeepSniffer the original code repository of DeepSniffer, our setup.py file for DeepSniffer to create the virtual environment, original README renamed to orig_README, and our new README providing steps for conducting the evaluation and power consumption analysis, and other useful documentation.

ML-DOCTOR the original code repository of ML-DOCTOR, our setup.pyfile for ML-DOCTOR to create the virtual environment, original README renamed to orig_README, and our new README providing steps for conducting the evaluation and power consumption analysis, and other useful documentation.

Adaptive Misinformation the original code repository of Adaptive Misinformation, our setup.py file for Adaptive Misinformation to create the virtual environment, original README renamed to orig_README, and our new README providing script file for conducting the evaluation and power consumption analysis, and other useful documentation.

Prediction-Poisoning the original code repository of Prediction-Poisoning, our setup.py file for Prediction-Poisoning to create the virtual environment, original README renamed to orig_README, and our new README providing script file for conducting the evaluation and power consumption analysis, and other useful documentation.

NOTE:

• For ease of study, we recommend creating virtual environments for the following projects - adaptive-misinformation, deepSniffer, prediction-poisoning, and ml-doctor - using the script file env-setup.sh provided in the repository ./ML_Extraction_Sok. We recommend navigating to the project directories for further information on project execution and testing for these projects.

Citing our paper

@inproceedings {nayan2024sok,
author = {Tushar Nayan and Qiming Gao and Mohammed Al Duniawi and Marcus Botacin and Selcuk Uluagac and Ruimin Sun},
title = {SoK: All You Need to Know About On-Device ML Model Extraction - The Gap Between Research and Practice},
booktitle = {33rd {USENIX} Security Symposium ({USENIX} Security 24)},
year = {2024},
url = {https://www.usenix.org/conference/usenixsecurity24/presentation/nayan},
publisher = {{USENIX} Association},
month = aug,
}

Contact

In case of feedback, suggestions, or issues, please contact Authors.