Google Glass, the GDK, and HTML5-Based Mobile Apps
SVCC (10/11/2014)
Oswald Campesato
Topics in This Session
Google Glass and Glassware Glass Features (condensed summary) What is the Mirror API Some HTML5 Technologies What is the GDK PhoneGap and Google Glass What about Android Wear
Some Glass Features (briefly)
Directions (“ok glass…get directions”) Video (“ok glass…take a video”) Camera (“ok glass…take a picture”) Translate (scan text and translate) Answer gmail (voice input) Voice-activated Android apps Pictures/videos are backed up on G+ Touch pad (gesture detection in GDK):
Swipe left/right/down; tap; scroll
How to Create Glassware
Use the Glass Mirror API (lightweight/server-side)
Use Eclipse (or Android Studio) with the GDK
Use PhoneGap and add Glass-related plug-ins
Use the GDK + Mirror API to create “hybrid” apps:
https://github.com/mimming/kitty-compass
Note: Glass hybrid apps refer to GDK + Mirror API, which is different from HTML5 hybrid apps
Glass Pattern Building Blocks
UI elements:
+ Static cards (Card class is deprecated)
+ Live cards (Android service): low & high frequency
+ Immersions
Voice commands:
+ Focus on the action, not the agent
+ Decrease time from intent to action
+ Is colloquial and easy to say
+ Is general enough to apply to multiple Glassware
https://developers.google.com/glass/design/patterns
Internet Access for Glass
Go to https://glass.google.com/myglass
Enter the name of a WiFi and the password
Scan the QR code with Glass
“Okay Glass…get directions” (or whatever)
Note: GG remains tethered to your Android device even when you switch it off
Screencasting GG via MyGlass
1) Launch MyGlass on your Android phone
2) Tap “MyGlass” (top of device screen)
3) Tap “Screencast” (bottom of drop-down list)
4) “Swipe” left or right on your Android device
+ Swipe gestures update GG (and vice versa)
+ swipe left/forward: displays the present/future
+ swipe right/backward: displays the past
NB: Screencasting Glass-to-iOS-device works as well, but not from an iOS device to Glass
The Glass Mirror API (1)
“lightweight”: processes only on server-side Reduces battery consumption (versus GDK) Provides access to server-side Glass APIs Easy to “install” Supports static cards and timeline mgmt Support for various languages:
Python/JavaScript/Node/Java A ‘sandbox’ is available
Access requires a registered Glass device
The Glass Mirror API (2)
Static cards can contain:
+ Text, HTML, media (images and videos)
+ Standard and custom menu items
+ multiple “pages” (articles)
+ auto or manual pagination
Timeline management:
+ Subscribe to timeline notifications
+ LBS (location-based services)
The Glass Mirror API (3)
Disadvantages of using the Mirror API:
1) Internet access is required:
+ apps only work when in WiFi range
2) limited number of API calls (1000/day)
3) no voice trigger
4) No real-time (no animation)
5) Time delay receiving server response(s)
6) No device access:
+ Camera/Compass/GPS
NB: infrequent location updates
Glass Mirror API: what happens?
Step #1: user authenticates Step #2: Glass stores credentials (server-
side) Step #3: Glass inserts static card in
timeline Step #4: Glass synchronizes timeline Step #5: user sees static card on Glass
Glass Mirror API Quick Start Samples:
https://developers.google.com/glass/develop/mirror/quickstart/index
The GDK: Glass Development Kit
An “add-on” to the Android SDK The GDK provides Android-based APIs provides Glass-specific APIs GDK-based Glassware runs on Glass itself Use standard Android development tools ICS (4.0.3) or higher Simple installation: adb install abc.apk
Supports Timeline, cards, and immersions Support for sensors, Menu, and UI Support for touch pad and gestures Support for media (sound/camera/voice)
https://developers.google.com/glass
The +/- of the Glass GDK Advantages:
+ GDK-based apps work offline (Internet not required)
+ Unlimited access to GDK-based apps
+ Support for voice trigger
+ Real-time support (smoother experience)
+ support for animation effects
+ No delay (no server access required)
+ Device access (camera/compass/etc)
Disadvantages:
+ apps can drain the battery
+ more difficult to install GDK-based apps
Glass GDK Set-up
Option #1 software installation: Eclipse (3.5 or higher?) JDK 1.6 (or higher) ADT installation
Option #2 software installation: Android Studio (complete zip file)
Enable Debug Mode on Glass
+ swipe to "Settings" -> "Device info”
+ tap to open device menu
+ select "Turn on debug"
The Android ‘adb’ Utility
Install apks on devices: adb install abc.apk
List available devices: adb devices
List apks on Glass: adb shell pm list packages –f
Launch the Glass home card:
adb shell am start -n com.google.glass.home/.timeline.MainTimelineActivity
Create/Deploy “Hello World” in Eclipse
CREATE the application:
1) Launch Eclipse or Android Studio
2) File > New > Android Application Application
3) specify “HelloWorld” for “Application Name”
4) API 15 (or API 19) for all SDK-related options
5) Click “Next” 4 times and click “Finish” once
DEPLOY the app to a device: Run > Run > Android Application > OK
ELAPSED TIME: 60 seconds
Android Project Structure
The top-level project directory contains the following:
./src: Java code (usually in a package)
./res/values: XML files with text strings and layout styles
./res/layout: XML files for layouts (such as activity classes)
./res/drawable: static images (*)
./res/assets/www: HTML-related assets
./libs: JAR files
./gen: generated code
./bin: “apk” file (and other stuff)
AndroidManifest.xml is the main “control” file
NB: assets are “mapped” to properties in R.java
(*) drawable-hdpi, drawable-mdpi, and so forth
Android App Components Activity: essentially a screen in an application
Intent: launch Activity (provides other “stuff”)
Service: background operations (no UI), such as downloads
Broadcast receiver (listens to system-wide events)
Content provider: manages shared app data, such as contacts
Also defined in AndroidManifest.xml
The PhoneGap Way (HTML5)
Install node and npm
Install PhoneGap: npm install –g phonegap
Create an app: cordova create test1
Go to root directory: cd test1
Add Android: cordova platform add android
cordova plugin add https://github.com/aphex/cordova-glass-core
cordova plugin add https://github.com/aphex/cordova-glass-touch
cordova plugin add https://github.com/aphex/cordova-glass-gesture
Deploy Android apk: cordova run
What is CardBuilder?
Replaces deprecated Card class with new layouts TEXT and TEXT_FIXED: text content and a footer COLUMNS and COLUMNS_FIXED: display a mosaic
or icon on the left side and text on the right side AUTHOR: an avatar, primary+secondary headings,
and body text TITLE: an image or mosaic with a name and icon ALERT and MENU: for dialogs with warning
messages and action confirmation flows improvements to GestureDetector’s scroll handling GDK samples fully support Android Studio
What is a Glass Card?
A Card is “a unit of work” (one screen)
A Card contains:
+ Main body text
+ Left-aligned footer
+ Right-aligned footer for a timestamp
+ One image for the background
Creating a (deprecated) Card (native code):
+ set the properties of the card
+ invoke Card.toView() to convert to an Android view
Glass Live Cards Native Glassware “sits” on Timeline (left-
side)
display real-time info (weather/news/etc)
frequent updates (running in background)
Similar to Android Widgets (or Service)
Display text and images (and 3D graphics)
a 'stop' option is required for live cards
Live Card Project (1) Create Android Project in Eclipse (or Android
Studio)
Specify version 15 or higher
Rename MainActivity to MenuActivity
Remove launcher/main intent filters from manifest
Add Service as the main component, which handles Live Card creation/management
Live Card Project (2) Implement Live Card Service (override 2
methods)
onStartCommand() method: called when service is started Create a live card and publish it If live card already exists, then display it
onDestroy() method: Called when service is destroyed Unpublish the live card
Glass Immersions
They run outside the Timeline
Typically consist of Android Activities
fully-customizable screens
you can design your own UI
Access to all user input
Better for interactive experiences
Touch Gestures on Glass
GDK provides a gesture detector for the touchpad
Gestures include: tapping/swiping/scrolling
Working with gestures in the GDK:
+ create a GestureDetector
+ implement one or more listeners
+ override onGenericMotionEvent()
+ pass motion events to onMotionEvent()
Glass and Voice Activation (1)
OLD contents of <intent-filter> in AndroidManifest.xml:
<intent-filter>
<action android:name="android.intent.action.MAIN" />
<category android:name="android.intent.category.LAUNCHER" />
</intent-filter>
NEW contents of <intent-filter> in AndroidManifest.xml:
<intent-filter>
<action android:name="com.google.android.glass.action.VOICE_TRIGGER" />
<meta-data
android:name="com.google.android.glass.VoiceTrigger"
android:resource="@xml/voice_trigger" />
</intent-filter>
Glass and Voice Activation (2)
Contents of $APP_TOP/res/xml/voice_trigger.xml:
<?xml version="1.0" encoding="utf-8"?>
<trigger keyword="@string/glass_voice_trigger">
<input prompt="@string/glass_voice_prompt" />
</trigger>
Contents of $APP_TOP/res/values/strings.xml:
<?xml version="1.0" encoding="utf-8"?>
<resources>
<string name="glass_voice_trigger">take note</string>
<string name="glass_voice_prompt">tell me something</string>
</resources>
</intent-filter>
Sensors on Google Glass Supported sensors:
TYPE_ACCELEROMETER
TYPE_GRAVITY
TYPE_GYROSCOPE
TYPE_LIGHT
TYPE_LINEAR_ACCELERATION
TYPE_MAGNETIC_FIELD
TYPE_ROTATION_VECTOR
Unsupported sensors:
TYPE_AMBIENT_TEMPERATURE
TYPE_PRESSURE
TYPE_PROXIMITY
TYPE_RELATIVE_HUMIDITY
TYPE_TEMPERATURE
Glass Mirror API versus GDK Use Mirror API for:
+ platform independence
+ common infrastructure
+ built-in functionality
Use GDK for:
+ real-time interaction
+ access to hardware
+ offline functionality
Use the GDK and the Mirror API
What is HTML5?
“Anything that you’re working on right now that’s cool…that’s HTML5” (PPK)
HTML5 means different things to different people
Multiple Technologies (see W3C)
W3C REC status: MathML, SVG, and 2 others
Technologies “in” HTML5 Canvas 2D
CSS3
Drag-and-Drop (DnD)
File API
Geolocation
Microdata
Offline Applications
Server Sent Events (SSE)
SVG
Web Intents
Web Messaging
Web Storage
Web Sockets
Web Workers
HTML5 WAI ARIA
Web Accessibility Initiative: Accessible Rich Internet Applications
W3C specification for increasing the accessibility of
web pages and user interface components developed with Ajax, HTML, JavaScript and related technologies.
WAI-ARIA predates HTML5
WAI ARIA specification defines support for accessible web applications, which involves markup extensions that are often attributes of HTML5 elements
CSS3 Features (condensed) (Perf) Gradients (linear and radial) (Perf) Rounded corners/drop shadows CSS3 filters (merged with SVG filters) Simple shapes (circle/ellipse/rectangle) (Perf) 2D and 3D animation Multi-column support Exclusions and regions Flexbox (like “grid bag” in Java AWT)
CSS3 2D/3D Transforms
CSS3 2D transforms:
rotate, scale, translate, matrix, and perspective, and skew (deprecated)
CSS3 3D transforms (invoke the GPU):
1) similar to 2D versions (but no ‘skew’)
2) also three axis-specific methods:
rotateX, rotateY, rotateZ (and so forth)
• CSS3 animation via a keyframes rule
What “works” with CSS3?
Combine these with CSS3 in a Web Page:
+ SVG and D3.js (=JavaScript+SVG)
+ HTML5 Canvas (bitmap and no DOM)
For Web applications: jQuery (cross-browser + cross-platform) BackboneJS (reduces spaghetti code) AngularJS (open source from Google) Other JavaScript-based toolkits
What is HTML5 Canvas?
HTML5 <canvas> element a rectangular bitmap in a web page Direct mode (SVG is “deferred” mode) Draw-and-forget (“Etch-a-Sketch”) JavaScript APIs for rendering shapes Support for 2D but not for 3D Available on iOS5+ and Android ICS+ Hardware accelerated (RIM Playbook)
Basic Shapes in HTML5 Canvas
Line segments Circles and arcs Ellipses (with the scale transform) Bezier curves (quadratic/cubic) Polygons Paths (for arbitrary shapes)
What You can Do with 2D Shapes
Linear and radial gradients
Rotate, scale, skew, translate methods
Perspective and matrix() method
2D Animation effects
“Template” for HTML5 Canvas (Part 1)
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Graphics with HTML5 Canvas</title>
<script type="text/javascript"><!--
window.addEventListener('load', function () {
// Step #1: Get the canvas element
var elem = document.getElementById('myCanvas');
if (!elem || !elem.getContext) {
return;
}
“Template” for HTML5 Canvas (Part 2)
// Step #2: Get the canvas 2d context
var context = elem.getContext('2d');
if (!context) {
return;
}
// Step #3: INSERT YOUR CANVAS CODE HERE
});
// --></script>
</head>
<body>
<canvas id="myCanvas" width=“800" height=“500">
</canvas>
</body>
</html>
What is D3.js?
D3.js: an open source project (circa 2010) Created by Mike Bostock based on JavaScript a layer of "abstraction" over SVG also support for HTML5 Canvas github.com/mbostock/d3 https://github.com/mbostock/d3/wiki/Gallery
Why/When use D3.js?
For data visualization If you need “extreme versatility” leverage JavaScript skills leverage SVG HTML5 Web pages with D3.js and:
HTML5 Canvas, CSS3, SVG, jQuery, …
What Can D3.js Do?
All the stuff you can do in SVG graphics/animation (2D only) filters/gradients mouse/keyboard events custom charts/graphs Support for Ajax, JSON, XML, CSV files
Simple D3.js Example<html>
<head>
<script src="d3.min.js"></script>
<script>
d3.select("body")
.append("p")
.text("Hello From D3");
</script>
</head>
<body>
<p>Hello From D3</p>
</body>
</html>
D3 and Other Technologies
You can create HTML5 web pages with:
+ D3 & Bootstrap + Firebase (or Parse)
+ D3 & BackboneJS
+ D3 & AngularJS
+ D3 & HTML5 Web Sockets
+ D3 & NodeJS (or Meteor)
What about SVG?
XML-based vocabulary for 2D shapes Gradients/Bezier curves/polygons/animation DOM-based and declarative syntax Supported in iOS5+ and Android ICS+ Partial hardware acceleration No support for 3D effects
SVG versus Canvas:
http://msdn.microsoft.com/en-us/hh552482.aspx
Useful Features of SVG
render circles/ellipses/elliptic arcs squares/rectangles/parallelograms cubic/quadratic Bezier curves arbitrary polygons linear/radial gradients and filters mouse events and animation support (*) good for charts/graphs Works with CSS3 (reference SVG via “url()” (*) consider using D3.js
Other Aspects of SVG
SVG elements are inserted in the DOM so you can track/manage groups of SVG elements
no blurred/jagged edges when zooming in
Convenient format for import/export between tools
Can apply XSL stylesheets to SVG documents
Blog by Patrick Dengler: SVG versus Canvas
Disadvantages of SVG
• Verbose (what do you expect? It’s XML)
• Can be difficult/incomprehensible (SVG tiger)
Animation code can be cumbersome/tedious
Consider using D3 instead of “pure” SVG
Strengths of CSS3/SVG/Canvas
CSS3: a) 2D/3D graphics/animation, b) GPU support, c) embeddable in SVG <defs> element, d) “easing functions” for animation, e) animate HTML elements
Canvas: a) 2D graphics/animation, b) GPU support, c) good for updating many small objects (games), d) works with video (use ThreeJS/WebGL for 3D animation)
SVG: a) 2D graphics/animation, b) some GPU support, c) create arbitrary 2D shapes, d) custom <pattern>s, and e) grouping via the <g> element, and “easing functions in D3 (but SVG cannot animate HTML)
Combining CSS3 and SVG CSS3 can leverage the power of SVG:
+ reference SVG documents via “url()”
• SVG can leverage CSS3 by:
embedding CSS selectors in an SVG doc
use jQuery methods:
+ the css() method for updating properties
• Use jQuery Mobile features:
use bindings to handle mouse/touch events
CSS3+SVG or CSS3+D3.js?
CSS3+SVG: useful when 1) you already have SVG-based data and 2) you must support IE6 (can do with Raphael toolkit but not D3)
• CSS3+D3: for modern browsers, and also easier for defining event handlers and animation (probably also easier to maintain/enhance)
• Notes:• CSS3+SVG might not have GPU support• Consider D3 with BackboneJS/AngularJS/etc
What about CSS3 and HTML5 Canvas?
Insert a <canvas> element in a Web page• JavaScript APIs for 2D shapes (similar to Skia) All shapes are rendered in a <canvas> element Modern browsers support <canvas> element “write-and-forget” (no DOM-based “history”)• CSS3 selectors can match <canvas> element• Canvas often renders polygons faster than SVG • Tracing Canvas calls:
http://www.html5rocks.com/en/tutorials/canvas/inspection/
Some Useful CSS3 Resources
• caniuse.com
online forums (Yahoo group)
Various meetup groups in BA
stackoverflow.com
Impressive HTML5 Canvas Demos
• Various HTML5 Canvas demos:
http://www.craftymind.com/factory/html5video/CanvasVideo.html
http://codepen.io/stuffit/pen/KrAwx
http://davidwalsh.name/canvas-demos
PhoneGap and Glass
Perform the following steps:
+ Install PhoneGap: npm install –g phonegap
+ Create a PhoneGap mobile app (CLI)
+ Add HTML/CSS/JavaScript assets
+ Deploy to Google Glass via an IDE
+ command line installation:
adb install –r myapp.apk
CSS3 and Glass Demo
CSS3 2D skew (for the three cube faces) CSS3 3D linear/radial gradients CSS3 2D and 3D transforms CSS3 3D animation effects (keyframes)
Demos: Canvas and Glass
Demo #1 uses Android Canvas: touch-enabled bar chart uses the GDK with Android Canvas
Demo #2 uses HTML5 Canvas: PhoneGap-based application
What about Android Wear
Android Wear provides a separate SDK
Not compatible with the GDK(?)
Different paradigm from Glass
Google Glass Official Sites
Currently available Glassware: https://glass.google.com/glassware
Purchasing Google Glass:
https://glass.google.com/getglass/shop/glass
Google Glass community:
https://www.glass-community.com/?auth_token=y8WXiHyf_TbV-_0aDamI9Q1UFE5tTkEUyWCHoPPAy10&authuser=0&hl=en_US&ts=1405039468196
Bonus: NodeJS and Quadcopters var arDrone = require('ar-drone'); var client = arDrone.createClient();
client.takeoff();
client .after(5000, function() { this.clockwise(0.5); }) .after(3000, function() { this.animate('flipLeft', 15); }) .after(1000, function() { this.stop(); this.land(); });
Open Source Projects (graphics)
• Graphics Projects on http://code.google.com/p:
+ css3-graphics and html5-canvas-graphics
+ css3-jquery-graphics and d3-graphics
+ svg-graphics and svg-filters-graphics
+ easel-graphics, fabric-graphics, paper-graphics
+ ThreeJS, jQuery, Raphael, Google Go, Android
+ Dart, Dojo, JSF, Two.js, JavaFX 2.0
+ Lua, PHP, Perl, Python, Ruby, SWT graphics
Graphics Projects on github (ocampesato):
+ Famo.us, Polymer, AngularJS
NodeJS Course (10/19-10/21)
Adam CrabTree
Enerspace
2225 East Bayshore Road
Suite 100
Palo Alto, CA 94303
http://www.eventbrite.com/e/noderietytm-nodejs-bootcamp-october-tickets-12539314427
Recent/Upcoming Books
1) HTML5 Canvas and CSS3 Graphics (2012)
2) jQuery, CSS3, and HTML5 for Mobile (2013)
3) HTML5 Pocket Primer (2013)
4) jQuery Pocket Primer (2013)
5) HTML5 Mobile Pocket Primer (2014)
6) D3 Pocket Primer (2014)
7) Google Glass Pocket Primer (2015)
8) AngularJS Pocket Primer (2015)
• “WebGL: Up and Running” (Tony Parisi)
Co-creator of VRML and X3D
Top Related