EIS: A Wearable Device for Epidermal American Sign

Language RecognitionAuthors: Zijie Zhu, Xuewei Wang, Aakaash Kapoor, Zhichao Zhang,

Tingrui Pan, Zhou Yu

Presentation: Joe Sirrianni

American Sign Language

• American Sign Langaugs (ASL) is a communication language often used by the hearing impaired.

• ASL has its own grammer rules different from English. • There are around 1 million ASL users. • Most words have their own gestures, however, finger spelling (used

for names and technical terms) is also used and contains the familiar 26 letter alphabet.

• These finger spelling gestures involve both finger and hand movement.

Contribution

• The authors present a epidermal-iontroinic sensing (EIS) based wearable devices that wears on finger joints to for 35 fingerspelling ASL recogntions.

Other ASL recognition Device Sensors

Optical Signal Sensing:• Uses video/motion

capture to detect hand movements/recognize letters/words

• Ex: Microsoft Kinect, Leap Motion Devices

• They don’t work well in low lighting situations

Electrical Signal Sensing:• EMG: Uses muscle

contractions to monitor muscle activity

• IMU: Uses accelerometer, gyroscope, and magnetometer to model hand movement.

• Devices achieve good accuracy but incur a lot of noise when used by users.

Mechanical Signal Sensing• Uses Strain Sensors to

capture skin deformation to recognize ASL.

• Sensors can be put in a glove which has very high accuracy.

• Devices tend to not be long-term wearable and are bulky. They are also sensitive to outdoor and body temperature.

Iontronic Capacitive Sensing

• A liquid electrolytes have been used in semiconductors to control the electron transport in a device.

• These electrolytes are ionic-conducting and electronically insulating, which creates an electrical double layer (EDL).

• EDL is model of an interface holding a charge separation between ions and electrons at the nanometer scale distance. Their behavior is analogous with capacitors.

• Using this type of layer, enough capacitive signal can be generated, which can avoid noise from the human body, the air, and other environmental variables.

EIS Sensor

• An EIS Sensor uses iontronic capacitive sensing and is applied to the skin and used to measure pressure.

• The Sensor is imperceptible and wearable.• EIS Sensors can be easily attached to the body and have been used to

measure blood pressure, respiration rates, and muscle activities. • For this application, the sensor areas for their EIS sensor was 4mm x 4

mm.

EIS Sensor

EIS Sensor Device

Complete Device• In addition to the EIS sensors (1 per finger), they used an IMU

(internal measuring unit) to capture movement gestures.• The IMU is a 6-axis IMU comprised of an accelerometer (for

acceleration in 3D space) and a gyroscope (for orientation and angular velocity).

• Outputs from the sensors where sent to a LabVIEW program on a computer via USB.

Data processing Pipeline

Feature Extraction

• Time-Domain Features: 56 total features including:• Max & Min value for each sensor• Mean value for signals. Used to alleviate influence of hand shaking. • Standard Dev of signals to distinguish steady movement• Distance between adjacent sensors on adjacent fingers.

• Frequency Domain Features: • Power Spectral Density (PSD) feature on acceleration and gyroscope data.• Divide signal into separate frequency bands, calulcate mean on each band.

Classifiers

• They tried three types of classifiers:• Support Vector Machine (SVM)• Random Forest (RF)• Neural Network (NN) – Cosine normalization, 4 layers, ReLU activation

function.

Data collection

• Used 8 participants• Each of the 35 gestures were done by every participant 10 times. • Collected 315,000 data points

Results

Results - Classification

• Results on all of the data

Results - Classification

• Results on a per user basis (8-training samples, 2-testing samples)

Results - Classification

• Cross-user recognition (Different users for training and testing)

Results

User Study

My Analysis

Positives• Interesting use of EIS sensors. • It’s good that they tried different

combinations of features. • I like that they used a variety of

different classification techniques and compared them.

Criticisms• The researchers had to assist the

users in putting on the device. It does not seem easy to put on/take off as a glove or camera.

• They did not try any words, which were used in the Mechanical Sensing studies.

• EIS sensors have a short life-span because of wearing (addressed in future work).