Touch UI in Mobile Java™ Technology-Based Devices · M3G – Mobile 3D Graphics M2G – The SVG-T...

Touch UI in Mobile Java™ Technology-Based Devices

“Beware of the Thing”Aleksi Uotila, Chief EngineerGörkem Ercan, Chief Engineer

TS-5325

2008 JavaOneSM Conference | java.sun.com/javaone | 2

“Beware of the Thing”

Learn how the touch user interface capabilities on mobile devices impact your mobile Java application development


Agenda

IntroductionDevice and hardware aspectsTouch interaction meansLook on the mobile Java APIs for touchTips and tricks & gotchasConclusions


New interaction paradigm for Mobile Java applications

Touch allows more freedom on interaction style selection for application

• Menu selections• Form filling• Direct manipulation

Touch interaction must be separately designed for the application


Device concepts – pointing device

“Stylus touch” “Finger touch”


Device concepts

“Stylus assisted finger touch”


Device concepts – device keys

Keypad (ITU-T + navigation)

Full keyboards (QWERTY)

No keys, or limited keys


Device concepts – touch screen hardware

Resistive touch screens• Better suited for stylus touch

Capacitive touch screens• Better suited for finger touch

Multi-touch support?• Depends on HW and low-level

drivers 3 . Conductive transparent metal coating 4 . Lower resistive circuit layer5. Glass / acrylic substrate

1. Polyester film 2 . Upper resistive circuit layer 3 . Conductive transparent metal coating

4. Insulating dots


Touch interactions

Simple pressDouble tapping (not to be mixed with double click)DraggingLong press

Single and multi-touch gesturesHover / pressureTactile feedback

Touch device’s native UI style uses different combinations of the above!


Touch interactions: text input

Handwriting recognition (HWR)

Virtual keyboards

Finger keyboards


Agenda



Look on the mobile Java APIs for touch

MIDP LCDUI – The UI toolkit of MIDletsM3G – Mobile 3D GraphicsM2G – The SVG-T APIMMAPI – MultimediaeSWT – Rich & cool open source UI toolkitNokia UI API – Providing UI extensions


Touch support in MIDP LCDUI: High-level APIs

LCDUI components should automatically support touch• TextBox, Form, List, Alert

Developer visible API features:• Commands• StringItem and ImageItem BUTTON and HYPERLINK appearance

modes


LCDUI Components Touch Demo


Touch support in MIDP LCDUI: Low-level APIs

Canvas• protected void pointerPressed(int x, int y) • protected void pointerReleased(int x, int y) • protected void pointerDragged(int x, int y) • public boolean hasPointerEvents()• public boolean hasPointerMotionEvents()

CustomItem• Same event methods as in Canvas• protected final int getInteractionModes() with bits

POINTER_PRESS, POINTER_RELEASE, and POINTER_DRAG bits


Touch support in MIDP LCDUI: Game actions

User must be able to trigger Game Actions: • FIRE, UP, DOWN, LEFT,

RIGHT, GAME_A, GAME_B, GAME_C, GAME_D

Issue how to remove the keypad?


Touch support in MIDP LCDUI: Game actions

User must be able to trigger Game Actions: • FIRE, UP, DOWN, LEFT,

RIGHT, GAME_A, GAME_B, GAME_C, GAME_D

Issue how to remove the keypad?• User settings available• Nokia-MIDlet-On-Screen-Keypad


LCDUI Custom Components Touch Demo


Touch support in M3G

User presses the touch screen

Device HW/ SW generates touch

press event

CanvaspointerPressed(int x,

int y)

Applicat ion modifies M3G scene graph according to the pointerPressed

event

Applicat ion redraws the newly modified scene graph to the

display


Touch support in M3G:pick

Group has method:

boolean pick(int scope, float x, float y, Camera camera, RayIntersection ri)

pick casts a pick ray into the scene, for example, from a pointer press.


Demo of M3G


Touch support in M2G

Device HW/ SW generates touch

press event

CanvaspointerPressed(int x,

int y)

Applicat ion forwards events to M2G via

dispatchMouseEvent

SVG Engine generates further

needed events (DOMActivate/DOMFocusIn)

Applicat ion does some act ion and modifies the SVG

model to give feedback

Applicat ion not ified which SVG Element

was clicked via org.w3c.dom.events

.EventListener

public void dispatchMouseEvent(java.lang.String type, int x, int y)

Pointer events are not automatically delivered to M2G contentEvents from UI toolkit need to be forwarded to M2G


M2G Code SampleSVGImage img = createImage(...);//SVGElement elem elem = img.getDocument().getElementById("menuItem_0");elem.addEventListener("click", this, false); animator = SVGAnimator.createAnimator(iImage); animator.setSVGEventListener(this);//from javax.microedition.m2g.SVGEventListenerpublic void pointerPressed(int x, int y) { img.dispatchMouseEvent("click", x, y); }//from org.w3c.dom.events.EventListenerpublic void handleEvent(Event e) { SVGElement elem = (SVGElement)e.getCurrentTarget(); // do something with elem}


Demo of M2G


MMAPI – agnostic of user interaction

Mobile Media API has no touch UI interaction dependencies

Completely agnostic of used interaction means

Applications use UI toolkit APIs (e.g. LCDUI) to create the user interaction

Some MMAPI implementations may implement end user visible touch support• Volume control of audio playback• On-screen video playback controls for UI embedded video


Touch support in eSWT

All eSWT implementations support touch• Due to its roots eSWT always supported touch

Menu interactions may differ in touch and non-touch• Indifferent to the developers

Some widgets provide functionality available only in touch• Table headers• Scrollbar selection events


eSWT Touch Demo


eSWT Touch support APIs

Query touch availability with Screen• public boolean isTouchScreen() • Query all screen instances, yes, there can be more than one

eSWT supports MouseDoubleClick, MouseDown, MouseUp, MouseMove• MouseListener, MouseMoveListener • MouseEvent

eSWT MouseEvent delivers • x,y coordinate the event occurred• widget that event occurred • stateMask of the keypad buttons• button the mouse button, possible to differentiate long press


eSWT Code Sample

Shell shell = new Shell();//Add a mouse listener to Shellshell.addMouseListener(new MouseListener() { public void mouseUp(MouseEvent e) { //coordinates are relative to the widget System.out.println("MouseUp:“ + e.x +" "+ e.y ); } public void mouseDown(MouseEvent e) { System.out.println("MouseDown:"+ e.x +" "+e.y ); } public void mouseDoubleClick(MouseEvent e) { //double click events are very rare on devices }});


Touch support in Nokia UI API

Tactile feedback extensions provided in Nokia UI API• works both for LCDUI and eSWT

Eliminates the passive feeling of touch screens

Allows applications to provide tactile feedback in custom components or games

Tactile feedback is often automatic in high-level components and native applications


Nokia UI API Code Sample

//com.nokia.mid.ui.TactileFeedbackTactileFeedback tactileFeedback = new TactileFeedback();//do direct feedback; verify latencytactileFeedback.directFeedback( TactileFeedback.FEEDBACK_STYLE_BASIC);tactileFeedback.directFeedback( TactileFeedback.FEEDBACK_STYLE_SENSITIVE);

MyCustomItem myItem = MyCustomItem(...);tactileFeedback.registerFeedbackArea(myItem, 0, //index

10, 10, //x & y coordsmyItem.getWidth()-20,myItem.getHeight()-20,TactileFeedback.FEEDBACK_STYLE_BASIC);


Agenda



UI design

Touch is a totally different interaction paradigmDesign touch UI for the application independently from scroll and select interactionStrive for direct manipulation where possibleReserve BIG enough touchable areasSingle hand interaction in a touch deviceText input needs to be optimized in touch as well


Tips for touch/non-touchCommand types

Pay real attention to Command types

Context specific commands

Nearly always define • select command for list• default command for a form item


Select Command Code Sample

List list = new List("Fruit", Choice.IMPLICIT);list.append("Gac", null);list.append("Bael", null);list.append("Cherimoya", null);list.append("Oroblanco", null);list.append("Longan", null);Command backCmd = new Command("Back", Command.BACK, 0);Command selectCmd = new Command(“Select", Command.OK, 0);list.addCommand(backCmd);list.addCommand(selectCmd);list.setCommandListener(this);list.setSelectCommand(selectCmd);


Pointer event gotchas

protected void pointerPressed(int x, int y)

Expect drag events outside the Canvas area

Too small areas, impossible to touch


Do not forget non-touch devices

It is more likely that your application will run on a keypad device• 700+ million devices with Java support in market

Avoid the need for long scrolling• Long forms• Provide commands as well as Buttons on screen context


Summary

Touch support is very diversely implemented in touch devices

More and more touch devices are emerging so design specifically for touch

UI components implement interactions for you

Custom component development is very hard: testing required in all devices you expect to deliver

Most likely keypad device users will be still majority of your customers


For More Information

Forum Nokia site at http://www.forum.nokia.com• Java SDKs and emulators, documents, discussion boards, wiki

Nokia Java™ ME Developer’s Library:• http://www.forum.nokia.com/main/resources/technologies/java/doc

umentation/

Technical sessions & a BOF from Nokia still come:• TS-6244: Regional and Cultural Accessibility for the Java™ ME:

Africa• TS-5140: Mobile Service Architecture 2: Latest News on JSR 248

and 249• TS-5631: Get on the Map with the Java™ ME Technology-Based

Location API • BOF-5856: Best Practices for Efficient MIDP Programming


Aleksi Uotila, Chief EngineerGörkem Ercan, Chief EngineerTS-5325

Touch UI in Mobile Java™ Technology-Based Devices · M3G – Mobile 3D Graphics M2G – The SVG-T...

Documents

Transcript of Touch UI in Mobile Java™ Technology-Based Devices · M3G – Mobile 3D Graphics M2G – The SVG-T...