Developing Multi-touch Applications

The future is here

Brian Blanchard

Chief Architect

Essence Technology

Agenda

• Introduction– Past, Future, & Present

• Touch scenarios, “Good, Better, Best” model, and developer roadmap

• Platform details:– Native Win32 gesture and touch support– Manipulations and Inertia– WPF support

• UX guidelines for touch applications• Where to get started - SDK, drivers, etc• Wrap up

What is Multi-Touch?

• Multi-touch * denotes a set of interaction techniques which allow computer users to control graphical applications with several fingers.

• The concept of Multi-touch technology was created to incorporate collaboration into the computing experience.

• * Wikipedia definition

Multi-touch Past

• 26 years in the making• Started in 82 by the university of Toronto• First commercial attempt in 84 by bell labs• Adopted by MS in 2001

Multi-touch Future

• LucidTouch: Transparent handheld inputs• Obscura Digital:

– HP - D5: Multi-Touch wall– VisionAire: Holographic Multi-Touch

Multi-touch Is Here!Hardware

– Multi-touch capable machines in market today, multi-touch appears in a broad set of form factors

– Desktops: $1200 - $2000

– Laptops: $1800 - $3000

Software

– Windows 7, PDC partner demonstrations & Touch platform announcement, .Net 4.0

Consumers

– New scenarios, next wave of user experience, high “WOW” factor

Touch Scenarios And Windows 7

We’ve identified several key scenarios for multi-touch:•Navigating and consuming the web•Reading & sorting email•Viewing photos•Playing casual games•Consuming music and video•Navigating files & arranging windows•Using Office applications

All focused on consumption

That led to four areas of investment:

•Developer Platform: At the root is the touch developer platform that exposes touch APIs for any application

•UI Enhancements: Focusing on the core scenarios, many parts of the core UI have been optimized for touch experiences.

•Gestures: Multi-touch gestures have been added to enable consistent panning and zooming in most applications.

•Applications: A set of multi-touch focused applications that demonstrate the power of touch. These will ship separate from Win7.

A Structured Approach To Adding Touch Support To Your Application

• Which application scenarios get the biggest benefit from touch?– Content consumption, mobile, media

and entertainment

• Decide which tier of investment is appropriate:– Good – Free support, UX tweaks– Better – Gesture support, touch-friendly UX– Best – Touch-optimized experience

Touch Development Roadmap

Windows 7 ReleaseNET 4.0 / Surface 2.0 Release

Multi-Touch ControlsMulti-Touch Controls

Multi-Touch APIMulti-Touch API

Surface

Multi-Touch Controls &

API

Surface

Multi-Touch Controls &

API

Multi-Touch APIMulti-Touch API

Multi-Touch API and ControlsMulti-Touch API and Controls

Touch Platform OverviewGood Better Best

APIs For Free!• Panning/zoom gestures• Right click gesture

• Gesture notifications• Pan/zoom/rotate/etc

• Raw touch data• Manipulation and Inertia

processors

Native Win32

• Controls with standard scrollbars

• WM_GESTURE message • WM_TOUCH• COM based Manipulation

and Inertia Processors

WPF • WPF 4.0 pan support in ScrollViewer

• Gesture events• Inertia configuration

• Touch events• Manipulation and Inertia

Processors

WinForms • Controls with standard scrollbars

• WM_GESTURE message• P/Invoke

• Manipulation and Inertia Processors in Microsoft.Ink.DLL

• Real-time Stylus or Ink Collector

Windows 7 Gestures & WM_GESTURECOMMAND

Brian BlanchardChief ArchitectEssence Technology

WM_GESTURECOMMAND• WM_GESTURECOMMAND

– Better: Good gesture support is key– Window message conveying gesture notifications

and information to applications– Why use it? Richer than legacy notifications,

smoother more responsive scrolling and zooming

• Compound gestures• Coalescing• Message bubbling and capture• Legacy handling & SFP

WM_GESTURECOMMAND• BOOL WINAPI GetGestureCommandInfo( • __in UINT uMsg, • __in WPARAM wParam, • __in LPARAM lParam, • __in LPARAM lExtraInfo, • __inout PGESTUREINFO pGestureInfo• );

• typedef struct _GESTURECOMMANDINFO {UINT cbSize;DWORD dwFlags;DWORD dwCommand;DWORD dwArguments;POINTS ptsLocation;

• } GESTURECOMMANDINFO, *PGESTURECOMMANDINFO;

WM_GESTURECOMMAND

Interpreting gestures:dwCommand dwArguments ptsLocation

Pan Distance between contacts

Current center of gesture

Zoom Distance between contacts

Current center of gesture

Rotate Absolute angle on rotate start, delta on updates

Current center of gesture

Two-finger tap NA Current center of gesture

Right click gesture NA First finger down

WM_GESTUREFuture Windows 7 Build

• Minor API surface changes:– Naming: WM_GESTURE,

GESTUREINFO, GetGestureInfo(), etc– Additional fields in GESTUREINFO

• Added gesture configuration support:– WM_GESTURENOTIFY– GESTURECONFIG struct– Set/GetGestureConfig()

Single-finger Panning Future Windows 7 Build

• Single-finger panning (SFP) provides an even easier way to pan where it is natural

• How to control SFP in your application:– SetGestureConfig() for GC_PAN– Specify

GC_PAN_WITH_SINGLE_FINGER_VERTICALLY and/or GC_PAN_WITH_SINGLE_FINGER_HORIZONTALLY

• If you aren’t getting the SFP default:– Consider standard scrollbars– SBM_GETSCROLLINFO (remarks here are helpful)– Implement WM_GESTURE support

WM_TOUCH

Brian BlanchardChief ArchitectEssence Technology

WM_TOUCH• WM_TOUCH* family of messages

– Best: WM_TOUCH enables touch-optimized experiences– WM_TOUCHDOWN, WM_TOUCHMOVE, WM_TOUCHUP– Semantically similar to mouse messages– Conveys raw touch data to Win32 apps– Scenario examples:

• Finger painting, custom gestures, feeding higher level controls, etc

• Capture• Coalescing

WM_TOUCH• BOOL WINAPI RegisterTouchWindow(• __in HWND hwnd,• __in ULONG ulFlags);

• BOOL WINAPI GetTouchInputInfo(• __in HANDLE hTouchInput, // input event handle• __in UINT cInputs, // number of elts in the array• __out_ecount(cInputs) PTOUCHINPUT pInputs, // array of touch inputs• __in int cbSize); // sizeof(TOUCHINPUT)

typedef struct tagTOUCHINPUT { LONG x; LONG y; HANDLE hSource; DWORD dwID; DWORD dwFlags; DWORD dwMask; DWORD dwTime; ULONG_PTR dwExtraInfo; DWORD cxContact; DWORD cyContact;} TOUCHINPUT, *PTOUCHINPUT;typedef TOUCHINPUT const * PCTOUCHINPUT;

• BOOL WINAPI CloseTouchInputHandle(• __in HANDLE hTouchInput); // input event handle

WM_TOUCH Future Windows 7 Build

• WM_TOUCH consolidated to a single message

Manipulations• Best: Manipulations are a great foundation for

touch-optimized experiences• Examples of higher level APIs/controls• Manipulations are:

– 2D affine transformations (translate, scale, rotate)– Superset of supported gestures– Supports multiple concurrent manipulations

• Need a source of raw data: WM_TOUCH or RTS• Similar to Surface APIs

• Interfaces:– IManipulationProcessor– IManipulationEvents

Inertia• Provides basic physics• Works hand in hand with Manipulations

• Interfaces:– IInertiaProcessor– IManipulationEvents – Same event interface

as Manipulations

Multi-touch In .Net Framework• For WPF:

– 3.5 update near Windows 7 launch • Multi-touch, gesture APIs

– 4.0 release• Multi-touch specific new controls

• For WinForms (Windows 7 launch):– Interop to native Win32 APIs – see sample– Use managed Manipulations/Inertia wrappers

Multi-Touch In WPF• UIElement and UIElement3D changes

– Gesture events (tracking)– Touch system gesture events (single)– Raw touch events– Touch and touch digitizer devices

• Multi-touch support in controls:– ScrollViewer update to accept pan gestures– Base controls updated to be multi-touch aware– Multi-capture support– New multi-touch specific controls

• Compatible with Surface SDK 2.0

Gestures In WPF• Gesture events on UIElement & UIElement3D

– TouchGestureStarted, TouchGestureChanged, TouchGestureEnded

– NOT routed, to enable multi-capture scenarios• Gesture in WPF based on Manipulations• Enable Inertia at TouchGestureStarted

– Inertial movements as TouchGestureChanged Events

• Touch system gestures

Touch Events In WPF• Touch Events on UIElement and UIElement3D

– TouchDown, TouchMove, TouchUp

– TouchEnter, TouchLeave

– Preview variants

• Promotion: Touch -> Stylus -> Mouse

• Multi-Capture

– CaptureTouchDevice(TouchDevice)

– ReleaseTouchDeviceCapture(TouchDevice)

– ReleaseAllTouchDeviceCaptures()

– TouchDeviceCollection TouchDevicesCaptured { get; }

– TouchDeviceCollection TouchDevicesCapturedWithin { get; }

– IsAnyTouchDeviceCaptured();

– IsAnyTouchDeviceCapturedWithin();

Tablet Platform Extensions• Real-time Stylus (RTS)

– RTS is an excellent raw touch data provider– IRealTimeStylus3::MultiTouchEnabled()

• Getting multi-touch data via InkCollector is a little trickier. E.g.:

ATOM m_atom = ::GlobalAddAtom(MICROSOFT_TABLETPENSERVICE_PROPERTY);

m_dwProperty = TABLET_ENABLE_MULTITOUCHDATA;

::SetProp(m_hwnd, (LPTSTR)m_atomPenService, (HANDLE)m_dwProperty);

UX Considerations

• ISVs are enthusiastic about multi-touch!• We are working with several ISVs today on multi-

touch applications, including:– AutoDesk– Avoco Secure– Corel– Cyberlink– Fuel Industries– Identity Mine– Sonic

Call To Action• Decide which application scenarios make

sense for touch• Decide which tier of investment

is appropriate• Build something amazing!

Getting Started

• What do you need to start coding?

– Hardware:

• Multi-touch beta drivers for in-market devices:

HP TouchSmart 2 Dell Latitude XT

– Software:

• Window 7 M3 build – On the drive

• Touch Platform SDK and samples – On the PDC drive

• The Multi-point touch drivers are not yet a part of any OS. Your choice in hardware will impact the software needed.

• WPF Multi-touch

– .NET 4.0 Preview Releases in 2009

– .NET 3.5 update to be announced

• LucidTouch:• http://research.microsoft.com/users/baudisch/projects/lucidtouch/index.html

• Obscura Digital: – HP - D5: Multi-Touch wall

• http://channel9.msdn.com/forums/Coffeehouse/255503-Surface-Thats-not-a-multi-touch-screen-/

– VisionAire: Holographic Multi-Touch• http://makeitv.blogspot.com/2008/08/minority-report-holo-display-is-now.html