M1gp Shimizu - Darwin phone
-
Upload
kazuto-shimizu -
Category
Technology
-
view
312 -
download
1
Transcript of M1gp Shimizu - Darwin phone
![Page 1: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/1.jpg)
DARWIN PHONES: THE EVOLUTION OFSENSING AND INFERENCEON MOBILE PHONESEMILIANO MILUZZO, CORY T. CORNELIUS, ASHWIN RAMASWAMY,TANZEEM CHOUDHURY, ZHIGANG LIU, ANDREW T. CAMPBELL
MOBISYS 2010
PRESENTER: KAZUTO SHIMIZU SEZAKI LAB, DEPT. OF IST, UNIV. OF TOKYO
![Page 2: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/2.jpg)
INTRODUCTION
Fortunately,
the presentation the author used at Mobisys 2010 is available on the web site.
http://www.cs.dartmouth.edu/~miluzzo/publications.html
![Page 3: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/3.jpg)
INTRODUCTION
Fortunately,
the presentation the author used at Mobisys 2010 is available on the web site.
http://www.cs.dartmouth.edu/~miluzzo/publications.html
So,
![Page 4: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/4.jpg)
INTRODUCTION
Fortunately,
the presentation the author used at Mobisys 2010 is available on the web site.
http://www.cs.dartmouth.edu/~miluzzo/publications.html
Experience Top Conference Quality from Now!!
![Page 5: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/5.jpg)
Darwin Phones: the Evolution of Sensing and Inference on
Mobile Phones
Emiliano Miluzzo*, Cory T. Cornelius*, Ashwin Ramaswamy*, Tanzeem Choudhury*, Zhigang Liu**,
Andrew T. Campbell*
* CS Department – Dartmouth College** Nokia Research Center – Palo Alto
![Page 6: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/6.jpg)
[email protected] Miluzzo
![Page 7: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/7.jpg)
[email protected] Miluzzo
![Page 8: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/8.jpg)
[email protected] Miluzzo
![Page 9: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/9.jpg)
[email protected] Miluzzo
![Page 10: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/10.jpg)
[email protected] Miluzzo
![Page 11: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/11.jpg)
[email protected] Miluzzo
![Page 14: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/14.jpg)
[email protected] Miluzzo
accelerometer
digital compass
microphone
WiFi/bluetooth GPS
….
light sensor/camera
sensing
![Page 15: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/15.jpg)
[email protected] Miluzzo
accelerometer
digital compass
microphone
WiFi/bluetooth GPS
light sensor/camera
gyroscope
air quality /pollution sensor
sensing….
![Page 17: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/17.jpg)
[email protected] Miluzzo
- 600 MHz CPU
- up to 1GB application memory
hardware
computation capability is increasing
![Page 18: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/18.jpg)
[email protected] Miluzzo
application distribution
collect huge amount of data for research
purposes
deploy apps onto millions of phones at
the blink of an eye
![Page 21: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/21.jpg)
[email protected] Miluzzo
cloud infrastructure
cloud - backend support
we want to push intelligence to the
phone
![Page 22: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/22.jpg)
[email protected] Miluzzo
cloud infrastructure
cloud - backend support
preserve the phone user experience
(battery lifetime, ability to make calls, etc.)
![Page 23: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/23.jpg)
[email protected] Miluzzo
cloud infrastructure
cloud - backend support
- sensing
- run machine learning algorithms locally
(feature extraction + inference)
![Page 24: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/24.jpg)
[email protected] Miluzzo
cloud infrastructure
cloud - backend support
- sensing
- run machine learning algorithms locally
(feature extraction + inference)
run machine learningalgorithms (learning)
![Page 25: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/25.jpg)
[email protected] Miluzzo
cloud infrastructure
cloud - backend support
store and crunch big data(fusion)
run machine learningalgorithms (learning)
- sensing
- run machine learning algorithms locally
(feature extraction + inference)
![Page 28: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/28.jpg)
[email protected] Miluzzo
societal scale sensing
global mobile sensor network
reality mining using mobile phones
will play a big role in the future
![Page 31: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/31.jpg)
[email protected] Miluzzo
microphone
camera
GPS/WiFi/cellular
air quality pollution
sensing apps
social context
audio / pollution / RF fingerprinting
image / video manipulation
darwin applies distributed computing and collaborative inference concepts to
mobile sensing systems
darwin
- classification model evolution
- classification model pooling
- collaborative inference
![Page 33: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/33.jpg)
why darwin?
[email protected] Miluzzo
deploy classifier X
deploy classifier X’
mobile phone sensing today
train classification model X’ in the lab
train classification model X in the lab
![Page 34: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/34.jpg)
why darwin?
[email protected] Miluzzo
train classification model X in the lab deploy classifier X
deploy classifier X’
a fully supervised approach doesn’t
scale!
mobile phone sensing today
train classification model X’ in the lab
![Page 35: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/35.jpg)
why darwin? a same classifier does not scale to multiple
environments (e.g., quiet and noisy env)
[email protected] Miluzzo
![Page 36: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/36.jpg)
why darwin? a same classifier does not scale to multiple
environments (e.g., quiet and noisy env)
[email protected] Miluzzo
![Page 37: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/37.jpg)
why darwin? a same classifier does not scale to multiple
environments (e.g., quiet and noisy env)
[email protected] Miluzzo
![Page 38: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/38.jpg)
why darwin? a same classifier does not scale to multiple
environments (e.g., quiet and noisy env)
[email protected] Miluzzo
darwin creates new classification models transparently from the user
(classification model evolution)
![Page 40: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/40.jpg)
[email protected] Miluzzo
why darwin?
ability for an application torapidly scale to many devices
darwin re-uses classification models when possible
(classification model pooling)
![Page 42: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/42.jpg)
[email protected] Miluzzo
why darwin?
leverage the large ensemble of in-situ resources
darwin exploits spatial diversity and co-operate to alleviate the “sensing context”
problem(collaborative inference)
![Page 47: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/47.jpg)
[email protected] Miluzzo
darwin phases
initial training (derive model seed)
classification model evolution
supervised
unsupervised
![Page 48: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/48.jpg)
[email protected] Miluzzo
darwin phases
initial training (derive model seed)
classification model evolution
classification model pooling
supervised
unsupervised
![Page 49: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/49.jpg)
[email protected] Miluzzo
darwin phases
initial training (derive model seed)
classification model evolution
classification model pooling
collaborative inference
supervised
unsupervised
![Page 51: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/51.jpg)
[email protected] Miluzzo
classification model training
sensed event
filtering (silence suppression +
voicing)
![Page 52: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/52.jpg)
[email protected] Miluzzo
classification model training
sensed event
filtering (silence suppression +
voicing)
featureextraction(MFCC)
![Page 53: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/53.jpg)
[email protected] Miluzzo
classification model training
filtering (silence suppression +
voicing)
featureextraction(MFCC)
modeltraining(GMM)
model
baseline
sensed event
send model + baseline back to phone
send MFCC tobackend to train the model
backend
![Page 56: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/56.jpg)
[email protected] Miluzzo
classification model evolution
phone: determines when to evolve
training sampled
![Page 57: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/57.jpg)
[email protected] Miluzzo
classification model evolution
phone: determines when to evolve
match?
YES
do not evolve
![Page 58: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/58.jpg)
[email protected] Miluzzo
classification model evolution
phone: determines when to evolve
match?
NO
evolve(train new model using
backend as before)
![Page 63: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/63.jpg)
[email protected] Miluzzo
classification model pooling
Speaker A’s model
Phone A Phone B
Phone C
Speaker C’s model
Speaker B’s modelSpeaker B’s model
Speaker C’s model
![Page 64: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/64.jpg)
[email protected] Miluzzo
classification model pooling
Speaker A’s model
Phone A Phone B
Phone C
Speaker C’s model
Speaker B’s modelSpeaker B’s model
Speaker C’s model
we have two options
1. train a new classifier for each speaker (costly for power, inference delay)
2. re-use already available classifiers
![Page 65: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/65.jpg)
[email protected] Miluzzo
classification model pooling
Speaker A’s model
Phone A Phone B
Phone C
Speaker C’s model
Speaker B’s modelSpeaker B’s model
Speaker C’s model
![Page 66: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/66.jpg)
[email protected] Miluzzo
classification model pooling
Speaker A’s model
Phone A Phone B
Phone C
Speaker B’s model
Speaker C’s model
Speaker C’s model
Speaker A’s model
Speaker B’s model
Speaker B’s model
Speaker A’s model
Speaker C’s model
![Page 67: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/67.jpg)
[email protected] Miluzzo
classification model pooling
Speaker A’s model
Phone A Phone B
Phone C
Speaker B’s model
Speaker C’s model
Speaker C’s model
Speaker A’s model
Speaker B’s model
Speaker B’s model
Speaker A’s model
Speaker C’s model
ready to run the collaborative inference algorithm
- local inference first- final inference later
![Page 69: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/69.jpg)
[email protected] Miluzzo
collaborative inference
1. local inference (running independently in parallel on each mobile phone)
two phases
![Page 70: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/70.jpg)
[email protected] Miluzzo
collaborative inference
1. local inference (running independently in parallel on each mobile phone)
two phases
2. final inference (after collecting Local Inference results, to get better confidence about the final classification result)
![Page 72: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/72.jpg)
[email protected] Miluzzo
collaborative inference
Phone A Phone B
Phone C
speaker A speaking!!!local inference (LI)
![Page 73: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/73.jpg)
[email protected] Miluzzo
collaborative inference
Phone A Phone B
Phone C
speaker A speaking!!!local inference (LI)
A’s LI results:Prob(A speaking) = 0.65Prob(B speaking) = 0.25Prob(C speaking) = 0.10
C’s LI results:Prob(A speaking) = 0.30Prob(B speaking) = 0.67Prob(C speaking) = 0.03
B’s LI results:Prob(A speaking) = 0.79Prob(B speaking) = 0.11Prob(C speaking) = 0.10
![Page 74: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/74.jpg)
[email protected] Miluzzo
collaborative inference
Phone A Phone B
Phone C
speaker A speaking!!!local inference (LI)
A’s LI results:Prob(A speaking) = 0.65Prob(B speaking) = 0.25Prob(C speaking) = 0.10
C’s LI results:Prob(A speaking) = 0.30Prob(B speaking) = 0.67Prob(C speaking) = 0.03
B’s LI results:Prob(A speaking) = 0.79Prob(B speaking) = 0.11Prob(C speaking) = 0.10
![Page 75: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/75.jpg)
[email protected] Miluzzo
collaborative inference
Phone A Phone B
Phone C
speaker A speaking!!!
A’s LI results:Prob(A speaking) = 0.65Prob(B speaking) = 0.25Prob(C speaking) = 0.10
local inference (LI)
C’s LI results:Prob(A speaking) = 0.30Prob(B speaking) = 0.67Prob(C speaking) = 0.03
B’s LI results:Prob(A speaking) = 0.79Prob(B speaking) = 0.11Prob(C speaking) = 0.10
![Page 76: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/76.jpg)
[email protected] Miluzzo
collaborative inference
Phone A Phone B
Phone C
speaker A speaking!!!
A’s LI results:Prob(A speaking) = 0.65Prob(B speaking) = 0.25Prob(C speaking) = 0.10
local inference (LI)
C’s LI results:Prob(A speaking) = 0.30Prob(B speaking) = 0.67Prob(C speaking) = 0.03
B’s LI results:Prob(A speaking) = 0.79Prob(B speaking) = 0.11Prob(C speaking) = 0.10
![Page 77: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/77.jpg)
[email protected] Miluzzo
collaborative inference
Phone A Phone B
Phone C
speaker A speaking!!!local inference (LI)
A’s LI results:Prob(A speaking) = 0.65Prob(B speaking) = 0.25Prob(C speaking) = 0.10
C’s LI results:Prob(A speaking) = 0.30Prob(B speaking) = 0.67Prob(C speaking) = 0.03
B’s LI results:Prob(A speaking) = 0.79Prob(B speaking) = 0.11Prob(C speaking) = 0.10
![Page 78: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/78.jpg)
[email protected] Miluzzo
collaborative inference
Phone A Phone B
Phone C
speaker A speaking!!!local inference (LI)
A’s LI results:Prob(A speaking) = 0.65Prob(B speaking) = 0.25Prob(C speaking) = 0.10
C’s LI results:Prob(A speaking) = 0.30Prob(B speaking) = 0.67Prob(C speaking) = 0.03
B’s LI results:Prob(A speaking) = 0.79Prob(B speaking) = 0.11Prob(C speaking) = 0.10
individual classification can be misleading!
![Page 79: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/79.jpg)
final inference (FI)
[email protected] Miluzzo
collaborative inference
Phone A Phone B
Phone C
each phone gathers LI results
A’s LI results
C’s LI results
B’s LI results
A’s LI results A’s LI results
C’s LI results C’s LI results
B’s LI resultsB’s LI results
![Page 80: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/80.jpg)
final inference (FI)
[email protected] Miluzzo
collaborative inferenceon each phone
A’s LI results:Prob(A speaking) = 0.65Prob(B speaking) = 0.25Prob(C speaking) = 0.10
C’s LI results:Prob(A speaking) = 0.30Prob(B speaking) = 0.67Prob(C speaking) = 0.03
B’s LI results:Prob(A speaking) = 0.79Prob(B speaking) = 0.11Prob(C speaking) = 0.10
![Page 81: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/81.jpg)
[email protected] Miluzzo
A’s LI results:Prob(A speaking) = 0.65Prob(B speaking) = 0.25Prob(C speaking) = 0.10
C’s LI results:Prob(A speaking) = 0.30Prob(B speaking) = 0.67Prob(C speaking) = 0.03
B’s LI results:Prob(A speaking) = 0.79Prob(B speaking) = 0.11Prob(C speaking) = 0.10
xxx
xxx
final inference (FI)
collaborative inferenceon each phone
![Page 82: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/82.jpg)
[email protected] Miluzzo
A’s LI results:Prob(A speaking) = 0.65Prob(B speaking) = 0.25Prob(C speaking) = 0.10
C’s LI results:Prob(A speaking) = 0.30Prob(B speaking) = 0.67Prob(C speaking) = 0.03
B’s LI results:Prob(A speaking) = 0.79Prob(B speaking) = 0.11Prob(C speaking) = 0.10
xxx
xxx
FI results (normalized):Confidence (A speaking) = 1 Confidence (B speaking) =
0.12Confidence (C speaking) =
0.002
=
final inference (FI)
collaborative inferenceon each phone
![Page 83: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/83.jpg)
[email protected] Miluzzo
A’s LI results:Prob(A speaking) = 0.65Prob(B speaking) = 0.25Prob(C speaking) = 0.10
C’s LI results:Prob(A speaking) = 0.30Prob(B speaking) = 0.67Prob(C speaking) = 0.03
B’s LI results:Prob(A speaking) = 0.79Prob(B speaking) = 0.11Prob(C speaking) = 0.10
xxx
xxx
=FI results (normalized):Confidence (A speaking) = 1 Confidence (B speaking) =
0.12Confidence (C speaking) =
0.002
final inference (FI)
collaborative inferenceon each phone
![Page 84: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/84.jpg)
[email protected] Miluzzo
A’s LI results:Prob(A speaking) = 0.65Prob(B speaking) = 0.25Prob(C speaking) = 0.10
C’s LI results:Prob(A speaking) = 0.30Prob(B speaking) = 0.67Prob(C speaking) = 0.03
B’s LI results:Prob(A speaking) = 0.79Prob(B speaking) = 0.11Prob(C speaking) = 0.10
xxx
xxx
=
collaborative inference compensates the inaccuracies of individual
inferences
FI results (normalized):Confidence (A speaking) = 1 Confidence (B speaking) =
0.12Confidence (C speaking) =
0.002
final inference (FI)
collaborative inferenceon each phone
![Page 86: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/86.jpg)
[email protected] Miluzzo
evaluation
C/C++ &
implemented on Nokia N97 andiPhone in support of a speaker
recognition app
![Page 87: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/87.jpg)
[email protected] Miluzzo
evaluation
C/C++ &
unix server
lightweight reliable protocol to transfer models from the server
and between phones
implemented on Nokia N97 andiPhone in support of a speaker
recognition app
![Page 88: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/88.jpg)
[email protected] Miluzzo
evaluation
C/C++ &
UDP multicast protocol to distribute
local inference results between phones
implemented on Nokia N97 andiPhone in support of a speaker
recognition app
![Page 89: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/89.jpg)
[email protected] Miluzzo
experimental scenarios
up to eight people in conversation in three different scenarios (quiet indoor, down the
street, in a restaurant)
![Page 95: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/95.jpg)
[email protected] Miluzzo
indoor quiet scenario
8 people talking around a table
collaborative inference + classification model evolution
boost the performance of a mobile sensing app
![Page 98: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/98.jpg)
[email protected] Miluzzo
impact of the number of mobile phones
the larger the number of mobile phones collaborating, the better the final inference result
![Page 101: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/101.jpg)
[email protected] Miluzzo
battery lifetime Vs inference responsiveness
smart duty-cycling techniques and machine learning algorithms with better performance in
terms of energy usage on mobile phones need to be identified
![Page 102: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/102.jpg)
PERSONAL OPINION
Contribution-Implemented the modality of unsupervised labeling
-Built & implemented concept of collaborative sensing
Merit-Drastic improve of accuracy
-Shorten learning time
Future work-Energy management
-Machine resource
![Page 103: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/103.jpg)
THANK YOU
REFERENCEEmiliano Miluzzo, Cory T. Cornelius, Ashwin Ramaswamy, Tanzeem Choudhury, Zhigang Liu, Andrew T. Campbell.
“Darwin Phone:the Evolution of Sensing and Inference on Mobile Phones,”
http://www.cs.dartmouth.edu/~miluzzo/publications.html
Talk(ppt), pdf, video, press available
![Page 104: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/104.jpg)
APPENDIX
![Page 105: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/105.jpg)
Emiliano Miluzzo (Ph.D)
Andrew T. Campbell (Professor) etc…
Mobile Sensing Group, Dartmouth College, Hanover, NH, USA
http://sensorlab.cs.dartmouth.edu/index.html
AUTHOR BACKGROUND
![Page 106: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/106.jpg)
RELATED WORK
Sensor node co-operation
Semi-supervised machine learning
Heterogeneous sensing device collaboration
Sensing applications on mobile phone
Paper # 24,33,36-38,45,49,52
28,41,50 15,25 8-10,13,19,21,27,29,31,35
Existing Static sensor network
On PC Only borrow data Individual device
Darwin Mobile sensor network
Applied to mobile phone
Execute and share individual inference
Multi device collaboration
![Page 107: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/107.jpg)
MACHINE PERFORMANCE
![Page 108: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/108.jpg)
MACHINE PERFORMANCE
![Page 109: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/109.jpg)
MACHINE PERFORMANCE
![Page 110: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/110.jpg)
SAMPLE APPLICATION
Speaker Model Computation
→MFCC feature extraction (Mel Frequency Cepstram Coefficient, メル周波数ケプストラム係数 )
• Leading approach for speech feature extraction [16,17,42]• Emphasize the part human use
Machine learning algorithm
→GMM (Gaussian Mixture Model)• Common to unsupervised machine learning
![Page 111: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/111.jpg)
PRIVACY & SECURITY
- Store and share not raw data but model & feature (of course protected)
- User can opt in and out anytime
- Darwin meets
1. Run on trusted device
2. Subscribe to trusted system
3. Run on trusted application i.e. pre-installed or downloaded from trusted third party.
![Page 112: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/112.jpg)
COLLABORATIVE INFERENCE
Individual Inference
LI = {Speaker1, Speaker2, .. ,Speaker_k}
Final Inference
![Page 113: M1gp Shimizu - Darwin phone](https://reader038.fdocuments.us/reader038/viewer/2022102901/55506377b4c90574428b54f1/html5/thumbnails/113.jpg)
EVALUATION SETTING
Situation• 5 phones• 8 people used• Several hours a day• 2 weeks
Voice chunk• Manually labeled to compare