Post on 27-Jun-2020
Michael Naimark Visiting Associate Arts Professor
Interactive Media Arts NYU Shanghai
VR / AR FundamentalsINTM-SHU 280C - 1 Topics in New Media & Entertainment Fall 2017
a practical guide to the big ideas
Jiayi Wang
Week 5: Input & Interactivity
Using our effectors and intentions as inputs and how they
shape interactive experiences.
Week 1 - Introduction to VR and AR
Week 2 - FUNDAMENTALS 1: Audiovisual Resolution and Fidelity Fooling a single static eye and single static ear experiencing a framed representation.
Week 3 - FUNDAMENTALS 2: Audiovisual Spatiality and Immersion Fooling two static eyes and ears, then two moving eyes and ears, then removing the frame.
Week 4 - FUNDAMENTALS 3: Other Senses (Haptic, Smell, Taste, Mind) Fooling the other senses.
Week 5 - FUNDAMENTALS 4: Input and Interactivity Using our effectors and intentions as inputs and how they shape interactive experiences.
Week 6 - FUNDAMENTALS 5: Social and Liveness Being immersed with other people and live events.
Week 7 - Current Dilemmas This is a lead-in, “pre-production,” to the second half-semester.
First half semesterClass Structure
Michael Naimark Visiting Associate Arts Professor
Interactive Media Arts NYU Shanghai
VR / AR FundamentalsINTM-SHU 280C - 1 Topics in New Media & Entertainment Fall 2017
a practical guide to the big ideas
Jiayi Wang
Week 5: Input & Interactivity
Using our effectors and intentions as inputs and how they
shape interactive experiences.
VR / AR Fundamentals
Input & Interactivity Using our effectors and intentions as inputs and how they shape interactive experiences.
Jiayi Wang
Input
I/O Computation Sensors (Non-AV)
• Mechanical • Position & Movement (IMUs) • Biometric & Environment
Vision/Cameras • Body • Hands • Eyes
Sound/Microphones • Voice Control • Smart Mics
Mind • Electrodes & Big Machines • Optical Imaging & Holography • Psi & Consciousness Research
Interactivity
UI Navigation v Manipulation Symmetry Responsiveness Inference Control v Illusion of Control
Mixing Board Analogy
Human sensors sense signals from a system output. Human effectors effect to a system input.
Human sensors sense signals from a system output. Human effectors effect to a system input.
(Not all intentional and explicit. System may infer intent.)
VR / AR Fundamentals
Input & Interactivity Using our effectors and intentions as inputs and how they shape interactive experiences.
Jiayi Wang
Input
I/O Computation Sensors (Non-AV)
• Mechanical • Position & Movement (IMUs) • Biometric & Environment
Vision/Cameras • Body • Hands • Eyes
Sound/Microphones • Voice Control • Smart Mics
Mind • Electrodes & Big Machines • Optical Imaging & Holography • Psi & Consciousness Research
Interactivity
UI Navigation v Manipulation Symmetry Responsiveness Inference Control v Illusion of Control
Smart Sensors Smart Cameras
Smart Mics
and much of the computation is now tiny and local.
VR / AR Fundamentals
Input & Interactivity Using our effectors and intentions as inputs and how they shape interactive experiences.
Jiayi Wang
Input
I/O Computation Sensors (Non-AV)
• Mechanical • Position & Movement (IMUs) • Biometric & Environment
Vision/Cameras • Body • Hands • Eyes
Sound/Microphones • Voice Control • Smart Mics
Mind • Electrodes & Big Machines • Optical Imaging & Holography • Psi & Consciousness Research
Interactivity
UI Navigation v Manipulation Symmetry Responsiveness Inference Control v Illusion of Control
Sparkfun Sensors
Buttons & Switches Encoders
Position & Movement (IMUs) (Intertial Measurement Unit)
Environment Biometric
Heartmath HRV sensor
Heart Rate Variability
Empatica E4 wristband
How would you measure breath?
some attempts
VR / AR Fundamentals
Input & Interactivity Using our effectors and intentions as inputs and how they shape interactive experiences.
Jiayi Wang
Input
I/O Computation Sensors (Non-AV)
• Mechanical • Position & Movement (IMUs) • Biometric & Environment
Vision/Cameras • Body • Hands • Eyes
Sound/Microphones • Voice Control • Smart Mics
Mind • Electrodes & Big Machines • Optical Imaging & Holography • Psi & Consciousness Research
Interactivity
UI Navigation v Manipulation Symmetry Responsiveness Inference Control v Illusion of Control
Motion Capture
Vicon IR cameras with retroreflective markers
Scatter / Depthkit Kinect (IR emitter) + DSLR
8i “Holograms” (NOT) (lots of cameras)
VR / AR Fundamentals
Input & Interactivity Using our effectors and intentions as inputs and how they shape interactive experiences.
Jiayi Wang
Input
I/O Computation Sensors (Non-AV)
• Mechanical • Position & Movement (IMUs) • Biometric & Environment
Vision/Cameras • Body • Hands • Eyes
Sound/Microphones • Voice Control • Smart Mics
Mind • Electrodes & Big Machines • Optical Imaging & Holography • Psi & Consciousness Research
Interactivity
UI Navigation v Manipulation Symmetry Responsiveness Inference Control v Illusion of Control
Leap Motion
technology
VR / AR Fundamentals
Input & Interactivity Using our effectors and intentions as inputs and how they shape interactive experiences.
Jiayi Wang
Input
I/O Computation Sensors (Non-AV)
• Mechanical • Position & Movement (IMUs) • Biometric & Environment
Vision/Cameras • Body • Hands • Eyes
Sound/Microphones • Voice Control • Smart Mics
Mind • Electrodes & Big Machines • Optical Imaging & Holography • Psi & Consciousness Research
Interactivity
UI Navigation v Manipulation Symmetry Responsiveness Inference Control v Illusion of Control
Gaze Tracking Head tracking + head position
Eye Tracking For control, for foveate imaging, and for gaze tracking
Fove
Eye Tribe video
VR / AR Fundamentals
Input & Interactivity Using our effectors and intentions as inputs and how they shape interactive experiences.
Jiayi Wang
Input
I/O Computation Sensors (Non-AV)
• Mechanical • Position & Movement (IMUs) • Biometric & Environment
Vision/Cameras • Body • Hands • Eyes
Sound/Microphones • Voice Control • Smart Mics
Mind • Electrodes & Big Machines • Optical Imaging & Holography • Psi & Consciousness Research
Interactivity
UI Navigation v Manipulation Symmetry Responsiveness Inference Control v Illusion of Control
Smart Mics
VR / AR Fundamentals
Input & Interactivity Using our effectors and intentions as inputs and how they shape interactive experiences.
Jiayi Wang
Input
I/O Computation Sensors (Non-AV)
• Mechanical • Position & Movement (IMUs) • Biometric & Environment
Vision/Cameras • Body • Hands • Eyes
Sound/Microphones • Voice Control • Smart Mics
Mind • Electrodes & Big Machines • Optical Imaging & Holography • Psi & Consciousness Research
Interactivity
UI Navigation v Manipulation Symmetry Responsiveness Inference Control v Illusion of Control
“Implanted electrodes simply won’t scale,”
“optical imaging”
Facebook “Typing with Thoughts” Goal is 100 words per minute
Openwater
“optical imaging”
simple wearable with fMRI resolution
Selected Psi Research Publications Dean Radin
Double Slit Experiment
Global Consciousness Project
Typing with your mind at 100 words per minute?
Moving things with your mind?
VR / AR Fundamentals
Input & Interactivity Using our effectors and intentions as inputs and how they shape interactive experiences.
Jiayi Wang
Input
I/O Computation Sensors (Non-AV)
• Mechanical • Position & Movement (IMUs) • Biometric & Environment
Vision/Cameras • Body • Hands • Eyes
Sound/Microphones • Voice Control • Smart Mics
Mind • Electrodes & Big Machines • Optical Imaging & Holography • Psi & Consciousness Research
Interactivity
UI Navigation v Manipulation Symmetry Responsiveness Inference Control v Illusion of Control
UI = User Interface
CHI = Computer User Interface SIGCHI conference since 1982
UX = User eXperience
“X” is the new “I”
VR / AR Fundamentals
Input & Interactivity Using our effectors and intentions as inputs and how they shape interactive experiences.
Jiayi Wang
Input
I/O Computation Sensors (Non-AV)
• Mechanical • Position & Movement (IMUs) • Biometric & Environment
Vision/Cameras • Body • Hands • Eyes
Sound/Microphones • Voice Control • Smart Mics
Mind • Electrodes & Big Machines • Optical Imaging & Holography • Psi & Consciousness Research
Interactivity
UI Navigation v Manipulation Symmetry Responsiveness Inference Control v Illusion of Control
Navigation = interactively changing user’s orientation and position “You are a ghost.”
(Different for orientation and position in cinematic VR.)
Manipulation = interactively changing the state of the database “Moving things.” “Blowing things up.”
(Requires either 3D computer model or branching.)
VR Interactivity
Some Useful Distinctions
https://medium.com/@michaelnaimark/vr-interactivity-59cd87ef9b6c
VR / AR Fundamentals
Input & Interactivity Using our effectors and intentions as inputs and how they shape interactive experiences.
Jiayi Wang
Input
I/O Computation Sensors (Non-AV)
• Mechanical • Position & Movement (IMUs) • Biometric & Environment
Vision/Cameras • Body • Hands • Eyes
Sound/Microphones • Voice Control • Smart Mics
Mind • Electrodes & Big Machines • Optical Imaging & Holography • Psi & Consciousness Research
Interactivity
UI Navigation v Manipulation Symmetry Responsiveness Inference Control v Illusion of Control
Conviviality: when means of production equals means of distribution
Ivan Illich, Tools for Conviviality (1972) (a driving force for the personal computer)
VR / AR Fundamentals
Input & Interactivity Using our effectors and intentions as inputs and how they shape interactive experiences.
Jiayi Wang
Input
I/O Computation Sensors (Non-AV)
• Mechanical • Position & Movement (IMUs) • Biometric & Environment
Vision/Cameras • Body • Hands • Eyes
Sound/Microphones • Voice Control • Smart Mics
Mind • Electrodes & Big Machines • Optical Imaging & Holography • Psi & Consciousness Research
Interactivity
UI Navigation v Manipulation Symmetry Responsiveness Inference Control v Illusion of Control
When do you think an interactive anything is not working?
When do you think an interactive anything is not working?
Latency
“Dishonest” interface (bad design)
VR / AR Fundamentals
Input & Interactivity Using our effectors and intentions as inputs and how they shape interactive experiences.
Jiayi Wang
Input
I/O Computation Sensors (Non-AV)
• Mechanical • Position & Movement (IMUs) • Biometric & Environment
Vision/Cameras • Body • Hands • Eyes
Sound/Microphones • Voice Control • Smart Mics
Mind • Electrodes & Big Machines • Optical Imaging & Holography • Psi & Consciousness Research
Interactivity
UI Navigation v Manipulation Symmetry Responsiveness Inference Control v Illusion of Control
VR / AR Fundamentals
Input & Interactivity Using our effectors and intentions as inputs and how they shape interactive experiences.
Jiayi Wang
Input
I/O Computation Sensors (Non-AV)
• Mechanical • Position & Movement (IMUs) • Biometric & Environment
Vision/Cameras • Body • Hands • Eyes
Sound/Microphones • Voice Control • Smart Mics
Mind • Electrodes & Big Machines • Optical Imaging & Holography • Psi & Consciousness Research
Interactivity
UI Navigation v Manipulation Symmetry Responsiveness Inference Control v Illusion of Control
What if you (the user/player/experiencer)
did everything as I (the author/maker) expected?
Would you know? Would you care?
“Kino-Automat”, Raduz Cincera, Czech pavilion, Expo '67 Montreal
“Cinelabyrinth” Raduz Cincera, Gas pavilion, Expo '90 Osaka
“Cinelabyrinth” Raduz Cincera, Gas pavilion, Expo '90 Osaka
VR / AR Fundamentals
For the next 2 Wednesdays, you are to experience, critique, post, and present your selected VR titles. This means:
1) Make sure your titles are here and running. (Work with platform point person.)
2) Write a brief review for each on the Documentation site.
3) Tag and rank your titles on the Catalog page. (Work with Ada and Dave.)
4) Present to class.
Week 3: Assignment for Wednesday
VR / AR News of the Week Mon 9 Oct
VR / AR Fundamentals
Bad Content Oculus will start refunding users for bad VR content
Good Content Discovery, Google Launch Multi-Part Virtual Reality Trip Around the World ‘Everest VR’ Hits PlayStation VR Today TRIPP Raises $4 Million to Develop ‘Mood Altering’ VR Experiences
Hardware Wars Sony Isn’t ‘Entirely Comfortable’ With PSVR Market Lead By Current Margin Google, Facebook and Microsoft Line Up For VR Standalone Push How to get the new Microsoft and Samsung ‘mixed reality’ headset Snap CEO's Focus on Spectacles Underscores Hardware Ambitions
Hardware Innovations A Finnish VR startup is developing a ‘human eye-resolution' headset that is '10 years ahead of its time' Jaunt VR’s 6DOF Interactive Experience Free The Night Debuting Exclusively For Windows Mixed Reality The Basslet: a wearable subwoofer for your body
Surprises (or maybe not) Artist virtually vandalises Jeff Koons’ Snapchat artwork to protest “AR corporate invasion” Go ahead, try to guess what Magic Leap does based on this video