Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.
-
Upload
leslie-sims -
Category
Documents
-
view
220 -
download
0
Transcript of Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.
![Page 1: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/1.jpg)
Introduction to Mobile Sensing with Smartphones
Uichin LeeApril 22, 2013
![Page 2: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/2.jpg)
iPhone 4 - Sensors
![Page 3: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/3.jpg)
Applications
• Transportation– Traffic conditions (MIT VTrack, Nokia/Berkeley
Mobile Millennium) • Social Networking– Sensing presence (Dartmouth CenceMe)
• Environmental Monitoring– Measuring pollution (UCLA PIER)
• Health and Well Being– Promoting personal fitness (UbiFit Garden)
![Page 4: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/4.jpg)
Citysense
MacroSense
CabSense
![Page 5: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/5.jpg)
Eco-system Players
• Multiple vendors– Apple AppStore– Google Play (Android Market)– Microsoft Mobile Marketplace
• Developers– Startups– Academia– Small Research laboratories– Individuals
• Critical mass of users
![Page 6: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/6.jpg)
Scale of Mobile Sensing
![Page 7: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/7.jpg)
Sensing Paradigm
• Participatory: active sensor data collection by users– Example: managing garbage cans by taking photos – Advantages: supports complex operations– Challenges:
• Quality of data is dependent on participants
• Opportunistic: automated sensor data collection– Example: collecting location traces from users– Advantages: lowers burden placed on the user– Challenges:
• Technically hard to build – people underutilized• Phone context problem (dynamic environments)
![Page 8: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/8.jpg)
SENSE
LEARN
INFORM, SHARE AND PERSUASION
Mobile Sensing Architecture
Mobile Computing Cloud
![Page 9: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/9.jpg)
Sense
• Programmability– Managing smartphone sensors with system APIs– Challenges: fine-grained control of sensors, portability
• Continuous sensing– Resource demanding (e.g., computation, battery)– Energy efficient algorithms– Trade-off between accuracy and energy consumption
• Phone context– Dynamic environments affect sensor data quality– Some solutions:
• Collaborative multi-phone inference• Admission controls for removing noisy data
SENSE
LEARN
INFORM, SHARE, PERSUASION
![Page 10: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/10.jpg)
Learn
• Integrating sensor data– Data mining and statistical analysis
• Learning algorithms – Supervised: data are hand-labeled (e.g., cooking, driving)– Semi-supervised: some of the data are labeled– Unsupervised: none of the data are labeled
• Human behavior and context modeling• Activity classification• Mobility pattern analysis (place logging)• Noise mapping in urban environments
SENSE
LEARN
INFORM, SHARE, PERSUASION
![Page 11: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/11.jpg)
Learn: Scaling Models
• Scaling model to everyday uses – Dynamic environments; personal differences – Large scale deployment (e.g., millions of people)
• Models must be adaptive and incorporate people into the process
• If possible, exploit social networks (community guided learning) to improve data classification and solutions
• Challenges:– Lack of common machine learning toolkits for smartphones– Lack of large-scale public data sets – Lack of public repository for sharing data sets, code, and tools
SENSE
LEARN
INFORM, SHARE, PERSUASION
![Page 12: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/12.jpg)
Inform, Share, Persuasion
• Sharing– Data visualization, community awareness, and social networks
• Personalized services– Profile user preferences, recommendations, persuasion
• Persuasive technology – systems that provide tailored feedback with the goal of changing user’s behavior– Motivation to change human behavior (e.g., healthcare,
environmental awareness)– Methods: games, competitions, goal setting– Interdisciplinary research combining behavioral and social
psychology with computer science
SENSE
LEARN
INFORM, SHARE, PERSUASION
![Page 13: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/13.jpg)
Privacy Issues
• Respecting the privacy of the user is the most fundamental responsibility of a mobile sensing system
• Current solutions– Cryptography– Privacy-preserving data mining– Processing data locally versus cloud services– Group sensing applications is based on user
membership and/or trust relationships
![Page 14: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/14.jpg)
Privacy Issues
• Reconstruction type attacks– Reverse engineering collected data to obtain invasive
information • Second-hand smoke problem– How can the privacy of third parties be effectively protected
when other people wearing sensors are nearby?– How can mismatched privacy policies be managed when
two different people are close enough to each other for their sensors to collect information?
• Stronger techniques for protecting people’s privacy are needed
![Page 15: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/15.jpg)
Understanding Smartphone Sensors:accelerometer, compass, gyroscope,
location, etc
![Page 16: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/16.jpg)
Smart Phone/Pad Sensors
GalaxyNexus iPhone4 iPhone5 Samsung
Galaxy S3SamsungGalaxy S4
Galaxy Tab/ iPad2
Accelerometer O O O O O O
Magnetometer (Compass) O O O O O O
Gyroscope O O O O O O
Light O O O O O O
Proximity O O O O O O
Camera O O O O O O
Voice O O O O O O
Pressure (Barometer) O O O
Humidity/IR/Temperature O
![Page 17: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/17.jpg)
Smartphone Sensors
• Accelerometer• Magnetometer (digital compass)• Gyroscope• Light• Pressure (Barometer)• Proximity• Camera• Voice• Temperature/Humidity/IR Gesture (Galaxy S4)
![Page 18: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/18.jpg)
Android APIs
• Package: android.hardware• Classes:– SensorManager – android service– Sensor – specific sensor– SensorEvent – specific event of the sensor = data
![Page 19: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/19.jpg)
SensorManager
• Most sensors interfaced through SensorManager or LocationManager – Obtain pointer to android service using
Context.getSystemService(name)
– For name, use constant defined by Context class • SENSOR_SERVICE for SensorManager • LOCATION_SERVICE for LocationManager
• Check for available sensors using List<Sensor> getSensorList(int type)
– Type constants provided in Sensor class documentation
![Page 20: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/20.jpg)
SensorManager
• Use getDefaultSensor(int type) to get a pointer to the default sensor for a particular type Sensor accel =
sensorManager.getDefaultSensor( Sensor.TYPE_ACCELEROMETER);
• Register for updates of sensor values using registerListener(SensorEventListener, Sensor, rate)
– Rate is an int, using one of the following 4 constants • SENSOR_DELAY_NORMAL (delay: 200ms)• SENSOR_DELAY_UI (delay: 60ms)• SENSOR_DELAY_GAME (delay: 20ms)• SENSOR_DELAY_FASTEST (delay: 0ms)
– Use the lowest rate necessary to reduce power usage• Registration will power up sensor:
mSensorService.enableSensor(l, name, handle, delay);
![Page 21: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/21.jpg)
SensorManager
• Unregister for sensor events using unregisterListener(SensorEventListener, Sensor) or unregisterListener(SensorEventListener)
• Undregistering will power down sensors:mSensorService.enableSensor(l, name, handle, SENSOR_DISABLE)
• Perform register in OnResume() and unregister in OnPause() to prevent using resources while your activity is not visible
• SensorListener is deprecated, use SensorEventListener instead – See documentation for Sensor, SensorManager, SensorEvent and
SensorEventListener
![Page 22: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/22.jpg)
SensorEventListener
• Must implement two methods onAccuracyChanged(Sensor sensor, int accuracy) onSensorChanged(SensorEvent event)
• SensorEvent – int accuracy – Sensor sensor – long timestamp • Time in nanoseconds at which event happened
– float[] values • Length and content of values depends on sensor type
![Page 23: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/23.jpg)
API – Setup
public class MainActivity extends Activity implements SensorEventListener {..
private SensorManager sm = null;…
public void onCreate(Bundle savedInstanceState) {..
sm = (SensorManager) getSystemService(SENSOR_SERVICE);}
protected void onResume() {..List<Sensor> typedSensors = sm.getSensorList(Sensor.TYPE_LIGHT);// also: TYPE_ALLif (typedSensors == null || typedSensors.size() <= 0) … error…sm.registerListener(this, typedSensors.get(0),
SensorManager.SENSOR_DELAY_GAME);// Rates: SENSOR_DELAY_FASTEST, SENSOR_DELAY_GAME,// SENSOR_DELAY_NORMAL, SENSOR_DELAY_UI
}}
![Page 24: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/24.jpg)
API – Processing Events
It is recommended not to update UI directly!
public class MainActivity extends Activity implements SensorEventListener {..private float currentValue;private long lastUpdate;…public void onSensorChanged(SensorEvent event) {
currentValue = event.values[0];lastUpdate = event.timestamp;
}..
}
![Page 25: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/25.jpg)
API – Cleanup
public class MainActivity extends Activity implements SensorEventListener {…
protected void onPause() {…
sm.unregisterListener(this);}
…
protected void onStop() {…
sm.unregisterListener(this);}
..}
![Page 26: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/26.jpg)
Accelerometer
Mass on spring
Gravity Free Fall Linear Acceleration Linear Accelerationplus gravity1g = 9.8m/s2
-1g1g
![Page 27: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/27.jpg)
Accelerometer
STMicroelectronics STM331DLH three-axis accelerometer (iPhone4)
![Page 28: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/28.jpg)
Accelerometer
• Sensor.TYPE_ACCELEROMETER• Values[3] = m/s2, measure the acceleration
applied to the phone minus the force of gravity (x, y, z)– GRAVITY_EARTH, GRAVITY_JUPITER,– GRAVITY_MARS, GRAVITY_MERCURY,– GRAVITY_MOON, GRAVITY_NEPTUNE
![Page 29: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/29.jpg)
Compass
• Magnetic field sensor (magnetometer)
ZX
Y
XY
Z
3-Axis Compass?
Magnetic inclination
Horizontal
Gravity
Magneticfield
vector
Magnetic declination
Magneticnorth
Geographicnorth
![Page 30: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/30.jpg)
Compass
Hall Effect 3-Axis Compass
![Page 31: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/31.jpg)
Compass
• Sensor.TYPE_MAGNETIC_FIELD• values[3] = in micro-Tesla (uT), magnetic field
in the X, Y and Z axis• SensorManager’s constants– MAGNETIC_FIELD_EARTH_MAX: 60.0– MAGNETIC_FIELD_EARTH_MIN: 30.0
![Page 32: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/32.jpg)
Orientation Sensor• Sensor.TYPE_ORIENTATION
– A fictitious sensor: orientation is acquired by using accelerometer and compass • Deprecated
– Now use getOrientation (float[] R, float[] result)• Values[3] – (Azimuth, Pitch, Roll)
– Azimuth, rotation around the Z axis• 0 <= azimuth <= 360, 0 = North, 90 = East, 180 = South, 270 = West
– Pitch, rotation around the X axis• -180 <= pitch <= 180• 0 = sunny side up• 180, -180 = up side down• -90 = head up (perpendicular) • 90 = head down (perpendicular)
– Roll, rotation around the Y axis• -90<=roll <= 90• Positive values when the z-axis moves toward the x-axis.
![Page 33: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/33.jpg)
Orientation: Why Both Sensors?
• We need two vectors to fix its orientation! (gravity and magnetic field vectors)
Tutorial: http://cache.freescale.com/files/sensors/doc/app_note/AN4248.pdf
![Page 34: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/34.jpg)
Gyroscope
• Angular velocity sensor– Coriolis effect – “fictitious force” that acts upon a freely moving object as
observed from a rotating frame of reference
![Page 35: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/35.jpg)
Gyroscope
![Page 36: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/36.jpg)
Gyroscope
• Sensor.TYPE_GYROSCOPE• Measure the angular velocity of a device• Detect all rotations, but few phones have it– iPhone4, iPad2, Samsung Galaxy S, Nexus S– Values[] – iPhone 4 gives radians/sec, and makes it
possible to get the rotation matrix
![Page 37: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/37.jpg)
Accelerometer vs. Gyroscope
• Accelerometer– Senses linear movement, but worse rotations, good for tilt
detection, – Does not know difference between gravity and linear
movement• Shaking, jitter can be filtered out, but the delay is added
• Gyroscope– Measure all types of rotation– Not movement– Does not amplify hand jitter
• A+G = both rotation and movement tracking possible
![Page 38: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/38.jpg)
How to Use the Data – Example
float[] matrixR = new float[9];float[] matrixI = new float[9];
SensorManager.getRotationMatrix(matrixR, matrixI,matrixAccelerometer, matrixMagnetic);
float[] lookingDir = MyMath3D.matrixMultiply(matrixR,new float[] {0.0f, 0.0f, -1.0f}, 3);float[] topDir = MyMath3D.matrixMultiply(matrixR,new float[] {1.0f, 0.0f, 0.0f}, 3);GLU.gluLookAt(gl,0.4f * lookingDir[0], 0.4f * lookingDir[1], 0.4f * lookingDir[2],lookingDir[0], lookingDir[1], lookingDir[2],topDir[0], topDir[1], topDir[2]);
![Page 39: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/39.jpg)
Accelerometer Noise - Simple
• Exponential moving average
const float kFilteringFactor = 0.1f; //play with this value until satisfied
float accel[3]; // previous iteration
//acceleration.x,.y,.z is the input from the sensoraccel[0] = acceleration.x * kFilteringFactor + accel[0] * (1.0f - kFilteringFactor);accel[1] = acceleration.y * kFilteringFactor + accel[1] * (1.0f - kFilteringFactor);accel[2] = acceleration.z * kFilteringFactor + accel[2] * (1.0f - kFilteringFactor);
result.x = acceleration.x - accel[0];result.y = acceleration.y - accel[1];result.z = acceleration.z - accel[2];
Return result;
![Page 40: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/40.jpg)
Accelerometer Noise – Notes
• If it is too slow to adapt to sudden change in position, do more rapid changes when angle(accel, acceleration) is bigger
• You can throw away single values that are way out of average.
• |acc| does not have to be equal to |g| !• Kalaman filters – too complicated?
![Page 41: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/41.jpg)
Other sensors
• Light sensor• Proximity sensor• Pressure sensor• Temperature/Humidity/IR Gesture (Galaxy S4)
![Page 42: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/42.jpg)
Light sensor
• Sensor.TYPE_LIGHT• values[0] = ambient light level in SI lux units• SensorManager’s constants
– LIGHT_CLOUDY: 100– LIGHT_FULLMOON: 0.25– LIGHT_NO_MOON: 0.001– LIGHT_OVERCAST: 10000.0 (cloudy)– LIGHT_SHADE: 20000.0– LIGHT_SUNLIGHT: 110000.0– LIGHT_SUNLIGHT_MAX: 120000.0– LIGHT_SUNRISE: 400.0
![Page 43: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/43.jpg)
Proximity sensor
• Sensor.TYPE_PROXIMITY• values[0]: Proximity sensor distance measured
in centimeters (sometimes binary near-far)
![Page 44: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/44.jpg)
Temperature sensor
• Sensor.TYPE_TEMPERATURE• values[0] = temperature
![Page 45: Introduction to Mobile Sensing with Smartphones Uichin Lee April 22, 2013.](https://reader036.fdocuments.us/reader036/viewer/2022062304/56649dd45503460f94acc045/html5/thumbnails/45.jpg)
Pressure sensor
• Sensor.TYPE_PRESSURE• values[0] = pressure• no constants