ITCamp 2012 - Tim Huckaby - Keynote

itcampro @ itcamp12 # Premium conference on Microsoft technologies

The Future of Software Now: NUI - A New Genre of User

Experience

Tim Huckaby Microsoft MVP & RD

InterKnowlogy Chairman/Founder

Actus Interactive Software CEO

Mobile &

Development ITCamp 2012 sponsors

NUI: The Natural User Interface

• Common Gestures of Touch

• Physical Object Interaction

• Virtual Object Interaction

Multi-Touch Capable

• Motion Based Interface

• Tracking of full-body movement, facial expression, and voice with precision

Gesture Capable

• Multi-array Microphone

• Uniquely Distinguish Audio & Voices Voice Capable

• Conscious and non-conscience interfaces

• Non Invasive

Neural Capable

Tim Huckaby Founder / Chairman, InterKnowlogy Founder / CEO, Actus Software Microsoft RD & MVP

A little bit about me and why I am here…

About Tim, InterKnowlogy & Actus…

InterKnowlogy (www.InterKnowlogy.com) • Tim Huckaby, Founder/Chairman (TimHuck@InterKnowlogy.com)

– 13+ year old services firm – Custom Application Development

– focused on building NUI solutions

– Web, mobile, and desktop

– windows 7/8, Surface, Windows phone 7, azure, Kinect,

•Largest Client: Microsoft

Actus Software (www.Actus-Software.com) • Tim Huckaby, Founder/CEO (TimHuck@Actus-Software.com)

•Software Product Company

– Interactive Kiosk and Active Digital Signage

– Multi-Touch, Gesture, and Voice Driven Solutions

We have come a long way…

The “Green-Screen”

Image courtesy of Computer History Museum

Touch Capable Computing Devices Are Not New

Past Future (Now) Present

1972: PLATO IV Touch Screen Terminal

The Samsung SUR40 for Microsoft Surface

Windows 8 - Metro

Apple iPad

What is Relatively New is:

Inexpensive Touch Capable, Non

Proprietary Hardware

A Windows OS designed for Multi-

A Consistent Multi-Touch API for .NET

across all Windows Hardware/Software

Platforms

Developer Productivity in Touch

Capable Apps

The Forecast of Multi-Touch Hardware

Capability in Every New Computing

Device in the Short Term

• HP TouchSmart TM2

• Rates in Windows as 5.0

• Price in October-2010: Starting from: $1499.99*

• Price in March-2011: Starting from: $799.99*

• Price Today (HP TouchSmart 320m ): $579.99

About the Windows 7 Computer I am Using Today…

HTML 5 Touch Events Spec, Version 2

W3C Editor's Draft 14, November 2011

Defines a set of low-level events that represent

one or more points of contact with a touch-

sensitive surface

Changes of those points with respect to the

surface and any DOM elements displayed upon

it (e.g. for touch screens) or associated with it

(e.g. for drawing tablets without displays).

Addresses pen-tablet devices, such as drawing

tablets, with consideration toward stylus

capabilities.

Source: https://dvcs.w3.org/hg/webevents/raw-file/tip/touchevents.html

HTML 5 Touch Examples

What is also Relatively New: Multi-Touch User Interfaces Combined with Object

recognition of Physical Objects

The Microsoft Surface is expensive

because it has 5 cameras looking up

that do Object Recognition

Based on a “byte tag” (not a QR

Object Recognition in Windows

The Use Cases are limitless…

Surface Gaming - Craps

Windows 8 - Designed for Multi-Touch

Metro - Perfect for Gesture

Windows 8:

• Metro mode / Desktop Mode

• Actus Interactive Kiosk – Metro

• Semantic Zoom

• Learning made Fun / Entertaining

• Proven as a learning tool beyond just

Children

• Computer based Games are a significant part

of popular culture

• Educational Video Game Curriculums

– MIT, UC Irvine, Minnesota, Mich State, etc.

NUI: Games as a Tool for Learning

NASA: Mars Bound

World's Fastest-Selling Consumer Electronics Device

… a Future Beyond Gaming

Combination of RGB camera, depth sensor and multi-array microphone

RGB camera delivers three basic color components

Depth sensors “sees” in 3-D

Microphone locates voices by sound and excludes ambient noise

The sensor provides eyes and ears - but it needs a brain

Raw data from the Kinect is just “a whole bunch of noise that someone needs to

take and turn into signal”

Our software finds the signal

Software makes all the magic possible

Skeletal Tracking

Face, Gesture Recognition

Audio Echo cancellation

Audio Beam Forming

Speech Recognition

Machine Learning

What is Kinect?

Kinect for Windows

Combination of RGB camera, depth sensor and multi-array microphone

RGB camera delivers three basic color components

Depth sensors “sees” in 3-D

Microphone locates voices by sound and excludes ambient noise

Consumer Priced: $249.99

Tracks full-body movement, facial expression, and voice with precision

Kinect sensor bar scans and monitors 1 million points on the body

Kinect for Windows SDK

Skeletal Tracking

Face, Gesture Recognition

Audio Echo cancellation

Audio Beam Forming

Speech Recognition

Machine Learning

The Good News: • The Kinect SDK for Windows

The Not So Good News: • Windows Only Device; K4W does not work on your

Xbox; you cannot build apps for 18 million Xbox Kinect owners with it.

XBOX Kinect

The XBOX Kinect for Windows

Touch-less Operating Room

Neural Interfaces

Conscious and Non-Conscious Interfaces

• High fidelity EG acquisition device • No conductive gel, etc. • Non-invasive • Use Case Examples:

• Conscious: mind manipulation of objects • Non-Conscious: Ranking music

• First target market: gaming • $300!!!

Direct API access to all Emotiv detection suite inputs and controls,

including training, profile management and gyro outputs

Program in C++ and compatible languages, .NET (VB.NET, C#), Java

Developer plug-in available for Unity3D

Active developer community (exclusive Developer user forum)

C++ Header files and Import Libraries

…which means…

Wrap the SDK in managed C++ thereby exposing a the API to a C#

application

That is very powerful…

Emotiv Software Development Kit (SDK)

We are within a decade of building the Holodeck

“The Ultimate NUI Application”

• We are at the cusp of a revolution in software & hardware

• Hardware and software platforms will ultimately synchronize

• Using Natural User Interface (NUI) technologies improve the User Experience

• The hardware, tools, plumbing and platforms for NUI computing are powerful and available now.

– …and will just continue to get better, cheaper, powerful and more available

• Within a realistic time frame NUI computing will be the norm

• The concepts of NUI change the way we design and build software

Summary

HoloDesk - Direct 3D Interactions with a Situated See-Through Display

Tim Huckaby

• More info on InterKnowlogy:

www.InterKnowlogy.com Silverlight.InterKnowlogy.com www.vimeo.com/InterKnowlogy

• More info on Actus Software:

www.Actus-Software.com

• About Tim Huckaby… – Founder/Chairman, InterKnowlogy – Founder/Chairman, Actus Software – Microsoft® Regional Director – Southern

California – Microsoft® .NET Partner Advisory Council

Founder / Member – Microsoft® MVP - .NET – INETA Speaker – International .NET Users

Group Association – Board Member: Association for Competitive

Technology – Board Member: Dev Connections – Author / Speaker

• Contact me: Tim Huckaby

E-mail: TimHuck@InterKnowlogy.com

TimH@Actus-Software.com

• Phone:

Office: 760-444-8640 Mobile: 619 990 9200

Blog: http://team.interknowlogy.com/blogs/timhuckaby

FaceBook: http://www.facebook.com/Tim.Huckaby.Carlsbad

LinkedIn: http://www.linkedin.com/in/timhuckaby

Twitter: @TimHuckaby

Mobile &

Development ITCamp 2012 sponsors

Mobile &

Development

Agenda

Mobile &

Development

ITCamp 2012 - Tim Huckaby - Keynote

Technology

Transcript of ITCamp 2012 - Tim Huckaby - Keynote

ITCamp 2012 - Dan Nicola - Scrum v2

ITCamp 2012 - Martin Kulov - Using the cloud for load testing

ITCamp 2011 - Raul Andrisan - What’s new in Silverlight 5

ITCamp 2011 - Mihai Nadas - Windows Azure interop

Emerging Experiences - More Personal Computing (MPC) - Tim Huckaby

Lessons Learned from the Trenches of NUI (Tim Huckaby)

ITCamp 2012 - Cristi Lefter - SQL Server 2012 for developers

ITCamp 2011 - Tiberiu Covaci - Fast Faster ...Async ASPNet

ITCamp 2011 - Mihai Tataran - Migrating to Azure

Bonnie Huckaby 2030 Lesson Plan

ITCamp 2013 - Tim Huckaby - Kinect for Windows - Designing Software for Gesture & Voice Controlled User Interfaces

Sedrick Huckaby, Faith & Family

Visiting Artist: Sedrick Huckaby - Boston University · of American Art, the Minneapolis Institute of Art, and The Art Institute of Chicago, ... Texas Observer. Sedrick Huckaby, “America’s

ITCamp 2015. Apps 2015: a legal odissey

ITCamp 2011 - Paula Januszkiewicz - Password secrets revealed

ITCAMP Lync Online-Onpremise Comunidad Office 365

Presentation ITCamp 2014 - Intellectual Property in Software

ITCamp - Novidades do windows server 2012

ITCamp 2011 - Paul Roman - High Availability for Exchange 2010

ITCamp 2013 - Peter Leeson - Intelligent Evolution