September 22, 2019 Sam Siewert

CS415Human Computer Interaction

Lecture 5 – Post WIMPComputer Vision (AR, VR Intro)

Catalog / MCO Description

Emphasis on Analysis and DesignHCI Concepts (Human, Computer, Interaction)Usability Metrics, Standards, GuidelinesTools and methods for rapid prototyping to support Agile Test-Driven-Design and early user evaluations

Sam Siewert 2

Learning Objectives1D Interaction: CLI and Scripting - Useful in Data Centers, Automation (strings, best effort)

2D Interaction: Windowed GUIs, Mobile Touchscreen -Current State of Practic (X,Y pointer and bitmaps, best effort)

3D Interaction: Machine Vision, AR/VR, Rendering, Domain Specific (X,Y,Z or X,Y and time constrained)

4D & AI: Intelligent Systems - Computers Acting more like Humans

– 3D Computer Vision for HCI (X,Y,Z, real-time, intelligent)– Natural Language Processing (X,Y, real-time, intelligent)– Machine Learning and Rule-based inference (X,Y,Z. intelligent)

Deesign for Usability– Metrics, Evaluation, Design for Quality (non-functional)– Standards, Guidelines, Rules (Expert research and gurus)

Design & Build or Improve an HCI Prototype, Evaluate– Independent and Dependent Variables– Test Scenario– Significance of results Sam Siewert 3

HCI Complexity Dimensions

1…3 - Space, X, Y, Z

4 - Time• Best effort• Constrained (interactive)• Real-time

Intelligence• Regular (state machine)• Context Free• Context Sensitive• Recursively Enumerable

Useability• Cognitive load• Ability to complete tasks• Look and feel• Evaluation• Standards & Guidelines• History

Overall Learning ObjectiveQualities of a Software Application or System– E.g. Mobile App– Desktop App– VR/AR System– Game– Mission Critical HCI (Avionics,

Medical, etc.)Quality is a “non-functional” requirementWe want and need quality to succeed, but difficult to design in and hard to directly testMust involve users and design of experimentsCan leverage past experience and wisdom of UX Gurus

Sam Siewert 4

Quality of a Software Application or SystemFor the User - User Experience (UX)• Methods of Design and Prototyping to

Assess and Improve UX• Design of Experiments to Test UX

Assignment Strategy – Usability!Assignment #1 [1D] – Shells and Command Line Interfaces – Prototype Code

Assignment #2 [2D] – WIMP GUI Building for Interactive Apps – Prototype Code

Assignment #3 [3D] –Beyond WIMP with CV/MV & 3D – Play with Code

EXAM-1

Assignment #4 [4D] – Interactive and Real-Time Systems – Prototype System

Assignment #5 – Propose Proof-of-Concept HCI of Your Interest – Design Focus and Design of Experiments

EXAM-2

Assignment #6 – Build HCI Proof-of-Concept, Test

FINAL EXAM - Present Results from Assignment #6

[Harder Programming, Easier Programming, Medium Programming]

Sam Siewert 5

Notes on InteractionCLI – 1D interaction, best effort (time)

– History helps– File completion helps– Background jobs helps– Multiple shells (with WIMP or multiplexing CLI)– Scripting automates

WIMP – 2D interaction, best effort (time)– Position, select, “click” – Windows, Icons, Menus, Pointer– Increases visual interaction– Cost is graphics processing at human interactive rates– GPU Co-processors offload CPU

VR/AR - 2D, 3D, time constraints (interactive)– real-time (constraints and requirements - deadline and

function, such as anti-lock braking)– In use in specialized domains – CAD, Visualization, VR– Emerging interaction models – AR (Augmented Reality)– Do we need this? Can it provide benefit?– History, physics and evolution would say yes – we live in a

3D world driven by the arrow of time– Cost – 3D Rendering in real-time (Ray tracing? Polygon?

Other?)

Sam Siewert 6

Agile definitions• Constraint - Limitation• Requirement -

Testable feature or function

• Quality refers to whether a product works, and whether it fulfills the project stakeholders’ needs. [Test: functional and non-functional]

• Usability - quality metric, by evaluation in an experiment

General definitions• Interactive• Real-Time

• Function• Completed

before deadline

Interaction & Computing Rates

Sam Siewert 7

Rate Clock Measure Human Interaction

Thread Context

ISR Instructions FPGA / ASIC

1 Hz sec seconds X X

10 Hz 1/10th

secmillisec X X

100 Hz 1/100th

secmillisec X

1000 Hz (KHz)

millisec microsec X X

MHz microsec

nannosec X X X

GHz nanosec picosec X X

THz picosec femtosec sub-nanosecond

Humans Operate at Rates down to 10’s of milliseconds– Frame rates up to 60 Hz (16.67 milliseconds)– Pointer and keyboard input up to 1 to 8 milliseconds (125 Hz – 1000 Hz), 40-80 WPM, up to

400 characters/secCPU cores run at 100’s MHz to several GHz1000 Instructions in one microsecond on GHz CPU coreThread scheduled processing down to 1 millisecondISR processing down to 1 microsecondCo-processor, Instruction below 1 microsecond to sub-nanosecond

Post WIMPIntegration of Computer Vision, Natural Language Processing and 3D Projection –Augmented Reality

MagicLeap – 3D imagery– “Magic Leap's total of $2.3 billion raised since

2011 makes it the most well-funded startup in the AR space.” - Business Insider

– “The first thing that dropped my jaw about Magic Leap was the fact that Google Inc. dove head-first into the company by investing half a billion dollars.” - Capitalist Creations

MS HoloLens – Holography

Other Forms of 3D Display and Projection

Wearable Immersive VR [Issues with Proprioception and Vestibular Comfort]

Sam Siewert 8

http://www.magicleap.com

https://www.microsoft.com/en-US/hololens

Pranav Mistry:SixtySenseTechnology, nowwith Samsung

Post WIMP - Discussion

WIMPS are Ubiquitous– Desktop– Mobile– even Server (Windows Server 2019)– Cloud - e.g. Console, Portal, Dashboard

Build Better WIMP Applications and Systems– Tasking and Task Flow [Concurrency]– Integration of Applications, Data Sharing, and Embedding– Human Factors and Ergonomics– Improved Look and Feel– Haptic Feedback? Other?

Is Web (2D) [VRML] Holding us Back?Is Mobile [OpenGL ES] Holding us Back?Is Lack of a Convincing 3D Display Holding us Back?3D Interaction Difficult? - Proprioception, VR/ARCost? Sam Siewert 9

AR/VRPromise?

Minute Paper:

Ubiquitous AR?

• Does Pranav’s “demo” appear ubiquitous?

• Can it become?• Should it?• Why?

Advanced InteractionActive Sensing for 3D– RGB depth mappers project IR grid (e.g. indoor robotics - UW Project)– Time-of-Flight Cameras (Time-of-flight_camera)– LIDAR is an Array of TOF Sensors for Point Cloud (Lidar)– RADAR (e.g. K-band radio wave)

Passive Sensing for 3D– 3D visual cues for Computer Vision (human emulation) and Machine

Vision (cheating)– Acoustic cameras (OptiNav)– 360 Cameras - E.g. Garmin VIRB 360

AR - Augmented Reality (Pokemon Go most widely deployed)VR - MS Hololens, Occulus, etc. for gaming mostly, but also for CAD and scientific visualization

Sam Siewert 10

Machine Vision & Robotics

Sam Siewert 11

Machine Vision

Computer Vision

(Human Model)

Image Processing

“Photo Shop”

Physics(Electromagnetic

Spectrum)

AI(Machine Learning)

Robotics

https://www.starwars.com/

Transformational Example(s)Very Common Engineering ArchitectureOpenCV PoC – Hough Linear or Circular TransformC code PoC – Image Enhancement, Sharpen, Brighten/Contrast, Encrypt/Decrypt, Compress/DecompressBatch or Continuous Real-Time

Sam Siewert 12

Digital Media Security & Transport

Computer Vision

Canny Edge Finder – Threshold ControlConsider a Transformation – Continuous Image Processing with a Simple Control Added

Sam Siewert 13

License Plate or Sign OCRMATLAB Example – Reading a SignCanny is Fundamental Step – Transformational Capability in a Larger Application (OCR)Correct Threshold for Intensity Gradient to Segment Image

Sam Siewert 14

Interactive Example(s)OpenCV PoC – Interactive Edge Finder, OCRC/C++ Allegro Games - http://alleg.sourceforge.net/Also Transformational (Ok to be Hybrid)Display + Slider = Interactive

Sam Siewert 15

Interactive Control(s)

Visual Feedback (Interactive)

OpenCV ExamplesOpenCV overview - https://opencv.org/

Simple examples– Jetson, OpenCV 2.x - computer-vision– Windows, OpenCV 3.x - computer_vision_cv3_win– Linux, OpenCV 3.x - computer_vision_cv3_tested

Code walk-through (Windows) with demo

Code walk-through (R-Pi) with demo

Sam Siewert 16

Hearing from ExpertsFei Fei-Li – Computer Vision, Ted Talk

David Eagleman – Sensory Substitution and Umwelt

Chris Melk – Digital Cinema with VR Headsets

Tony DeRose – Pixar, Math Behind the Movies

John Lasseter – Pixar, On Early Days of 3D Animation [DVD]

Future of VR in the Workplace

Sam Siewert 17

Fei Fei-Li Discussion PointsCognitive Model for Scene Parsing and Semantics– Claims that Humans process images at 5 Hz (200 millisec)– We know full motion video needs to be 24 Hz or greater– Why is there a difference?– Visual processing vs. visual capture – Sensing vs. Perception

Why is the compilation of Millions of Quality Images important? [image-net.org]

Scene segmentation? Why is it hard?

What is Amazon Mechanical Turk?

Comparison of Video ANN to Human Brain? [100K to millions of nodes] – her project uses 24 million nodes, 140 million parameters, 15 billion connections)– Human Eye [6-7 million cones, 10x number of rods, about 50 million]– Human Brain [10 billion neurons, 1 trillion synapses]– Human capability is 400x nodes, 1000x connections– Does this explain human superiority in scene description?

Sam Siewert 18