Michael Naimark Visiting Associate Arts Professor

Interactive Media Arts NYU Shanghai

VR / AR FundamentalsINTM-SHU  280C - 1   Topics in New Media & Entertainment Fall 2017

a practical guide to the big ideas

Jiayi Wang

Week 5: Input & Interactivity

Using our effectors and intentions as inputs and how they

shape interactive experiences.

Week 1 - Introduction to VR and AR

Week 2 - FUNDAMENTALS 1: Audiovisual Resolution and Fidelity Fooling a single static eye and single static ear experiencing a framed representation.

Week 3 - FUNDAMENTALS 2: Audiovisual Spatiality and Immersion Fooling two static eyes and ears, then two moving eyes and ears, then removing the frame.

Week 4 - FUNDAMENTALS 3: Other Senses (Haptic, Smell, Taste, Mind) Fooling the other senses.

Week 5 - FUNDAMENTALS 4: Input and Interactivity Using our effectors and intentions as inputs and how they shape interactive experiences.

Week 6 - FUNDAMENTALS 5: Social and Liveness Being immersed with other people and live events.

Week 7 - Current Dilemmas This is a lead-in, “pre-production,” to the second half-semester.

First half semesterClass Structure

Michael Naimark Visiting Associate Arts Professor

Interactive Media Arts NYU Shanghai

VR / AR FundamentalsINTM-SHU  280C - 1   Topics in New Media & Entertainment Fall 2017

a practical guide to the big ideas

Jiayi Wang

Week 5: Input & Interactivity

Using our effectors and intentions as inputs and how they

shape interactive experiences.

VR / AR Fundamentals

Input & Interactivity Using our effectors and intentions as inputs and how they shape interactive experiences.

Jiayi Wang

Input

I/O Computation Sensors (Non-AV)

• Mechanical • Position & Movement (IMUs) • Biometric & Environment

Vision/Cameras • Body • Hands • Eyes

Sound/Microphones • Voice Control • Smart Mics

Mind • Electrodes & Big Machines • Optical Imaging & Holography • Psi & Consciousness Research

Interactivity

UI Navigation v Manipulation Symmetry Responsiveness Inference Control v Illusion of Control

Mixing Board Analogy

Human sensors sense signals from a system output.

Human sensors sense signals from a system output. Human effectors effect to a system input.

Human sensors sense signals from a system output. Human effectors effect to a system input.

(Not all intentional and explicit. System may infer intent.)

VR / AR Fundamentals

Input & Interactivity Using our effectors and intentions as inputs and how they shape interactive experiences.

Jiayi Wang

Input

I/O Computation Sensors (Non-AV)

• Mechanical • Position & Movement (IMUs) • Biometric & Environment

Vision/Cameras • Body • Hands • Eyes

Sound/Microphones • Voice Control • Smart Mics

Mind • Electrodes & Big Machines • Optical Imaging & Holography • Psi & Consciousness Research

Interactivity

UI Navigation v Manipulation Symmetry Responsiveness Inference Control v Illusion of Control

Smart Sensors Smart Cameras

Smart Mics

Smart Sensors Smart Cameras

Smart Mics

and much of the computation is now tiny and local.

VR / AR Fundamentals

Input & Interactivity Using our effectors and intentions as inputs and how they shape interactive experiences.

Jiayi Wang

Input

I/O Computation Sensors (Non-AV)

• Mechanical • Position & Movement (IMUs) • Biometric & Environment

Vision/Cameras • Body • Hands • Eyes

Sound/Microphones • Voice Control • Smart Mics

Mind • Electrodes & Big Machines • Optical Imaging & Holography • Psi & Consciousness Research

Interactivity

UI Navigation v Manipulation Symmetry Responsiveness Inference Control v Illusion of Control

Sparkfun Sensors

Buttons & Switches Encoders

Position & Movement (IMUs) (Intertial Measurement Unit)

Environment Biometric

Heartmath HRV sensor

Heart Rate Variability

Empatica E4 wristband

How would you measure breath?

How would you measure breath?

some attempts

VR / AR Fundamentals

Input & Interactivity Using our effectors and intentions as inputs and how they shape interactive experiences.

Jiayi Wang

Input

I/O Computation Sensors (Non-AV)

• Mechanical • Position & Movement (IMUs) • Biometric & Environment

Vision/Cameras • Body • Hands • Eyes

Sound/Microphones • Voice Control • Smart Mics

Mind • Electrodes & Big Machines • Optical Imaging & Holography • Psi & Consciousness Research

Interactivity

UI Navigation v Manipulation Symmetry Responsiveness Inference Control v Illusion of Control

Motion Capture

Vicon IR cameras with retroreflective markers

Scatter / Depthkit Kinect (IR emitter) + DSLR

8i “Holograms” (NOT) (lots of cameras)

VR / AR Fundamentals

Input & Interactivity Using our effectors and intentions as inputs and how they shape interactive experiences.

Jiayi Wang

Input

I/O Computation Sensors (Non-AV)

• Mechanical • Position & Movement (IMUs) • Biometric & Environment

Vision/Cameras • Body • Hands • Eyes

Sound/Microphones • Voice Control • Smart Mics

Mind • Electrodes & Big Machines • Optical Imaging & Holography • Psi & Consciousness Research

Interactivity

UI Navigation v Manipulation Symmetry Responsiveness Inference Control v Illusion of Control

Leap Motion

technology

VR / AR Fundamentals

Input & Interactivity Using our effectors and intentions as inputs and how they shape interactive experiences.

Jiayi Wang

Input

I/O Computation Sensors (Non-AV)

• Mechanical • Position & Movement (IMUs) • Biometric & Environment

Vision/Cameras • Body • Hands • Eyes

Sound/Microphones • Voice Control • Smart Mics

Mind • Electrodes & Big Machines • Optical Imaging & Holography • Psi & Consciousness Research

Interactivity

UI Navigation v Manipulation Symmetry Responsiveness Inference Control v Illusion of Control

Gaze Tracking Head tracking + head position

Eye Tracking For control, for foveate imaging, and for gaze tracking

Fove

Eye Tribe video

How does (long range) eye contact work?

VR / AR Fundamentals

Input & Interactivity Using our effectors and intentions as inputs and how they shape interactive experiences.

Jiayi Wang

Input

I/O Computation Sensors (Non-AV)

• Mechanical • Position & Movement (IMUs) • Biometric & Environment

Vision/Cameras • Body • Hands • Eyes

Sound/Microphones • Voice Control • Smart Mics

Mind • Electrodes & Big Machines • Optical Imaging & Holography • Psi & Consciousness Research

Interactivity

UI Navigation v Manipulation Symmetry Responsiveness Inference Control v Illusion of Control

Voice control is everywhere.

Smart Mics

VR / AR Fundamentals

Input & Interactivity Using our effectors and intentions as inputs and how they shape interactive experiences.

Jiayi Wang

Input

I/O Computation Sensors (Non-AV)

• Mechanical • Position & Movement (IMUs) • Biometric & Environment

Vision/Cameras • Body • Hands • Eyes

Sound/Microphones • Voice Control • Smart Mics

Mind • Electrodes & Big Machines • Optical Imaging & Holography • Psi & Consciousness Research

Interactivity

UI Navigation v Manipulation Symmetry Responsiveness Inference Control v Illusion of Control

EEG electrodes fMRI scanner

“Implanted electrodes simply won’t scale,”

“optical imaging”

Facebook “Typing with Thoughts” Goal is 100 words per minute

Openwater

“optical imaging”

simple wearable with fMRI resolution

Selected Psi Research Publications Dean Radin

Double Slit Experiment

Global Consciousness Project

Typing with your mind at 100 words per minute?

Moving things with your mind?

VR / AR Fundamentals

Input & Interactivity Using our effectors and intentions as inputs and how they shape interactive experiences.

Jiayi Wang

Input

I/O Computation Sensors (Non-AV)

• Mechanical • Position & Movement (IMUs) • Biometric & Environment

Vision/Cameras • Body • Hands • Eyes

Sound/Microphones • Voice Control • Smart Mics

Mind • Electrodes & Big Machines • Optical Imaging & Holography • Psi & Consciousness Research

Interactivity

UI Navigation v Manipulation Symmetry Responsiveness Inference Control v Illusion of Control

UI = User Interface

CHI = Computer User Interface SIGCHI conference since 1982

UX = User eXperience

“X” is the new “I”

VR / AR Fundamentals

Input & Interactivity Using our effectors and intentions as inputs and how they shape interactive experiences.

Jiayi Wang

Input

I/O Computation Sensors (Non-AV)

• Mechanical • Position & Movement (IMUs) • Biometric & Environment

Vision/Cameras • Body • Hands • Eyes

Sound/Microphones • Voice Control • Smart Mics

Mind • Electrodes & Big Machines • Optical Imaging & Holography • Psi & Consciousness Research

Interactivity

UI Navigation v Manipulation Symmetry Responsiveness Inference Control v Illusion of Control

Navigation = interactively changing user’s orientation and position “You are a ghost.”

(Different for orientation and position in cinematic VR.)

Manipulation = interactively changing the state of the database “Moving things.” “Blowing things up.”

(Requires either 3D computer model or branching.)

VR Interactivity

Some Useful Distinctions

https://medium.com/@michaelnaimark/vr-interactivity-59cd87ef9b6c

VR / AR Fundamentals

Input & Interactivity Using our effectors and intentions as inputs and how they shape interactive experiences.

Jiayi Wang

Input

I/O Computation Sensors (Non-AV)

• Mechanical • Position & Movement (IMUs) • Biometric & Environment

Vision/Cameras • Body • Hands • Eyes

Sound/Microphones • Voice Control • Smart Mics

Mind • Electrodes & Big Machines • Optical Imaging & Holography • Psi & Consciousness Research

Interactivity

UI Navigation v Manipulation Symmetry Responsiveness Inference Control v Illusion of Control

Conviviality: when means of production equals means of distribution

Ivan Illich, Tools for Conviviality (1972) (a driving force for the personal computer)

VR / AR Fundamentals

Input & Interactivity Using our effectors and intentions as inputs and how they shape interactive experiences.

Jiayi Wang

Input

I/O Computation Sensors (Non-AV)

• Mechanical • Position & Movement (IMUs) • Biometric & Environment

Vision/Cameras • Body • Hands • Eyes

Sound/Microphones • Voice Control • Smart Mics

Mind • Electrodes & Big Machines • Optical Imaging & Holography • Psi & Consciousness Research

Interactivity

UI Navigation v Manipulation Symmetry Responsiveness Inference Control v Illusion of Control

When do you think an interactive anything is not working?

When do you think an interactive anything is not working?

Latency

“Dishonest” interface (bad design)

VR / AR Fundamentals

Input & Interactivity Using our effectors and intentions as inputs and how they shape interactive experiences.

Jiayi Wang

Input

I/O Computation Sensors (Non-AV)

• Mechanical • Position & Movement (IMUs) • Biometric & Environment

Vision/Cameras • Body • Hands • Eyes

Sound/Microphones • Voice Control • Smart Mics

Mind • Electrodes & Big Machines • Optical Imaging & Holography • Psi & Consciousness Research

Interactivity

UI Navigation v Manipulation Symmetry Responsiveness Inference Control v Illusion of Control

The AI controversy

(why does “smart typing” suck?)

VR / AR Fundamentals

Input & Interactivity Using our effectors and intentions as inputs and how they shape interactive experiences.

Jiayi Wang

Input

I/O Computation Sensors (Non-AV)

• Mechanical • Position & Movement (IMUs) • Biometric & Environment

Vision/Cameras • Body • Hands • Eyes

Sound/Microphones • Voice Control • Smart Mics

Mind • Electrodes & Big Machines • Optical Imaging & Holography • Psi & Consciousness Research

Interactivity

UI Navigation v Manipulation Symmetry Responsiveness Inference Control v Illusion of Control

What if you (the user/player/experiencer)

did everything as I (the author/maker) expected?

Would you know? Would you care?

“Kino-Automat”, Raduz Cincera, Czech pavilion, Expo '67 Montreal

“Cinelabyrinth” Raduz Cincera, Gas pavilion, Expo '90 Osaka

“Cinelabyrinth” Raduz Cincera, Gas pavilion, Expo '90 Osaka

VR / AR Fundamentals

For the next 2 Wednesdays, you are to experience, critique, post, and present your selected VR titles. This means:

1) Make sure your titles are here and running. (Work with platform point person.)

2) Write a brief review for each on the Documentation site.

3) Tag and rank your titles on the Catalog page. (Work with Ada and Dave.)

4) Present to class.

Week 3: Assignment for Wednesday

VR / AR News of the Week Mon 9 Oct

VR / AR Fundamentals

Bad Content Oculus will start refunding users for bad VR content

Good Content Discovery, Google Launch Multi-Part Virtual Reality Trip Around the World ‘Everest VR’ Hits PlayStation VR Today TRIPP Raises $4 Million to Develop ‘Mood Altering’ VR Experiences

Hardware Wars Sony Isn’t ‘Entirely Comfortable’ With PSVR Market Lead By Current Margin Google, Facebook and Microsoft Line Up For VR Standalone Push How to get the new Microsoft and Samsung ‘mixed reality’ headset Snap CEO's Focus on Spectacles Underscores Hardware Ambitions

Hardware Innovations A Finnish VR startup is developing a ‘human eye-resolution' headset that is '10 years ahead of its time' Jaunt VR’s 6DOF Interactive Experience Free The Night Debuting Exclusively For Windows Mixed Reality The Basslet: a wearable subwoofer for your body

Surprises (or maybe not) Artist virtually vandalises Jeff Koons’ Snapchat artwork to protest “AR corporate invasion” Go ahead, try to guess what Magic Leap does based on this video