HUMAN-SYSTEMS INTERACTIONS

V2.1c

www.ensiie.fr/~bouyer/

guillaume.bouyer@ensiie.fr

JIN 2019 – HCI

Human-Computer Interactions

2

Design

Human

Technical

Interfaces, software,

implementation…

UX, UI, evaluation…

Psychology, physiology, ergonomics…

HCI

JIN 2019 – HCI

Objectives

•Knowledge on how to design, program and evaluate HCIs for interactive projects

•Practice •development (Unity)

•search for information/inspiration

•taking into account the needs/context of the user and the ideas/constraints of the designer

•Concepts applicable for games, VR&AR and PFE

3

bouyer@ensiie.fr

http://www.ensiie.fr/~bouyer/JIN

Office 111 @ ENSIIE

JIN 2019 – HCI

Methodology and planning

•1. Introductory course on interfaces and associated functionalities

•2. Gamepad handling under Unity (individual, graded)

•3. Analysis of existing 2D platform games (paired, graded)

•4. Searching for information on platform game development: everyone

•5. Controller development

•6. Search for information (and videos) on "game feel".

•7. Course on UX and evaluations

•8. Feedback design and development

•9. Testing and project finalization

4

PART 1

HUMAN-GAMES INTERFACING

JIN 2019 – HCI

Main Reference

•Game Engine Architecture, Jason Gregory, A K Peters/CRC Press, 2009 (http://www.gameenginebook.com/)

6

JIN 2019 – HCI

Types of devices

•Keyboard & mouse

•Joypad controller

•Gesture/3D controller

•Wiimote, Kinect, PS Move, touch surface

•Hybrid controller

•Wii U, PS Vita, 3DS, VR controller

•Built-in controllers

•Arcade machines

•Specialized input devices and adapters

•Music devices

•Steering wheels

•Dance pad

7

Various form factors and layouts

Common input types

Some kinds of outputs

LOW-LEVEL DEVICES INPUTS

8

JIN 2019 – HCI

Digital Buttons

•2 states: •pressed = down

•not pressed = up

•(cf. closed or open electrical flow)

•1 button usually represented by a bit•0 = up

•1 = down

•→ states of all of the buttons on a device packed into a single unsigned integer value

9

JIN 2019 – HCI

Digital Buttons: Microsoft ’s XInput API

•typedef struct _XINPUT_GAMEPAD {// 16-bit unsigned integerWORD wButtons;// 8-bit unsigned integerBYTE bLeftTrigger;BYTE bRightTrigger;// 16-bit signed integerSHORT sThumbLX;SHORT sThumbLY;SHORT sThumbRX;SHORT sThumbRY;

} XINPUT_GAMEPAD;

10

#define XINPUT_GAMEPAD_DPAD_UP 0x0001 // bit 0#define XINPUT_GAMEPAD_DPAD_DOWN 0x0002 // bit 1#define XINPUT_GAMEPAD_DPAD_LEFT 0x0004 // bit 2#define XINPUT_GAMEPAD_DPAD_RIGHT 0x0008 // bit 3#define XINPUT_GAMEPAD_START 0x0010 // bit 4#define XINPUT_GAMEPAD_BACK 0x0020 // bit 5#define XINPUT_GAMEPAD_LEFT_THUMB 0x0040 // bit 6#define XINPUT_GAMEPAD_RIGHT_THUMB 0x0080 // bit 7#define XINPUT_GAMEPAD_LEFT_SHOULDER 0x0100 // bit 8#define XINPUT_GAMEPAD_RIGHT_SHOULDER 0x0200 // bit 9#define XINPUT_GAMEPAD_A 0x1000 // bit 12#define XINPUT_GAMEPAD_B 0x2000 // bit 13#define XINPUT_GAMEPAD_X 0x4000 // bit 14#define XINPUT_GAMEPAD_Y 0x8000 // bit 15

Struct contains a variable wButtons that holds the state of all buttons

Bit mask defines which physical button corresponds to each bit in the word

An individual button’s state can be read by masking the wButtons word with the

appropriate bit mask

bool IsButtonADown(const XINPUT_GAMEPAD& pad){// Mask off all bits but bit 12 (the A button).return ((pad.wButtons & XINPUT_GAMEPAD_A) != 0);

}

JIN 2019 – HCI

Analog Axes and Buttons

•Axes•Range of values

•Used to represent the degree to which a trigger is pressed, orthe 2D position of a joystick (two analog inputs x and y)

•Buttons•Ex : Metal Gear Solid 2

•Signals usually too noisy to be usable

•Input signal usually digitized using integer or float

11

JIN 2019 – HCI

Analog Axes and Buttons: Microsoft ’s XInput API

•16-bit signed integers for left and right thumb sticks•[-32768, 32767]

•8-bit unsigned integers for left and right shoulder triggers•[0 , 255]

12

typedef struct _XINPUT_GAMEPAD {// 16-bit unsigned integerWORD wButtons;// 8-bit unsigned integerBYTE bLeftTrigger;BYTE bRightTrigger;// 16-bit signed integerSHORT sThumbLX;SHORT sThumbLY;SHORT sThumbRX;SHORT sThumbRY;

} XINPUT_GAMEPAD;

JIN 2019 – HCI

Relative Axes

•The position of an analog button, trigger, joystick, or thumb stick is absolute

•Clear zero value

•For relative devices•No clear location at which the input value should be zero

•Zero input value = the position of the device has not changed

•Non-zero input values = delta value from last time

•Ex: mice, mouse wheels, track balls…

13

JIN 2019 – HCI

3D Orientation

•3 accelerometers to estimate the orientation of the controller

•Measure the acceleration along each axis based on constant downward gravity

•Quantized into three signed 8-bit integers (x, y, and z)

•Ex: PS3 Sixaxis and DualShock 3, Nintendo WiiMote

14

JIN 2019 – HCI

Other Inputs

•3D Position•IR Camera (Wiimote)

•Location, size and distance of 2 fixed IR LEDs

•Camera (EyeToy)

•Gestures (Kinect)•Computer vision-based techniques (ex. structured light) -> depth map

•Infer body position: skeleton, joints

•Bodily movements, poses

•Touch•Single or multiple contact points

•Movements

15

LOW-LEVEL DEVICES OUTPUTS

16

JIN 2019 – HCI

Rumble & Force-Feedback

•Vibrations•One or more motors rotating a slightly unbalanced weight at various speeds

•Game can control:

•motors on/off

•speeds -> different tactile effects

•Ex. PS2, PS3, Xbox 360, Wii controllers

•Force•Actuator(s) driven by a motor

•Resist the motion of the player

•Game can control:•motors on/off

•strength and direction of the forces

•Ex. arcade driving games: steering wheel resists the player’s attempt to turn it, simulating difficult driving conditions or tight turns

17

JIN 2019 – HCI

Other Outputs

•Audio•Small speaker

•Embedded USB audio I/O device

•Memory card slot on the pad (Dreamcast)

•LEDs

•Specific outputs for specific controller (music instruments…)

18

COMMUNICATION WITH A DEVICE

19

JIN 2019 – HCI

Interfacing

•Game software reads devices inputs and writes outputs in various ways, depending on the specific design of the device

•Polling

•Interrupts

•Wireless

20

JIN 2019 – HCI

Polling

•Input = explicitly and periodically querying the state of the device

•Usually once per iteration of the main game loop

•By reading hardware registers directly

•By reading a memory-mapped I/O port

•Via a higher-level software interface

•Outputs = writing

21

JIN 2019 – HCI

Polling: Microsoft ’s XInput API

•XInputGetState() every frame

•Communicates with the hardware and/or drivers

•Reads the data in the appropriate way

•Packages an XINPUT_STATE struct•contains an embedded instance of an XINPUT_GAMEPAD struct

•Returns pointer

22

JIN 2019 – HCI

Interrupts

•Controller sends data to the game engine when changing state

via hardware interrupts

•Electronic signal to CPU

•CPU suspends execution of the main program, runs a routine to read

and store the state of the device

•CPU back to the main program

•Game can pick up the data at any time

•Ex: mouse when it moves, or button pressed/released

23

JIN 2019 – HCI

Wireless Devices

•I/O can't be read and written by simply accessing registers or memory-mapped I/O ports

•Ex. bluetooth protocol:

•Request the device to send input data back to the host

•Send output data to the device

•Handled by a thread separate from main loop

•Encapsulated behind an interface called from the main loop

•Ex: WiiMote, DualShock 3, Xbox 360 wireless controller

24

HIGH-LEVEL INTERFACE SYSTEMS

25

JIN 2019 – HCI

Game Engine Interface Systems

•Inputs processing•From raw data to smooth, pleasing, intuitive behaviors in-game

•Input management features

•Level of abstraction•Decouple raw inputs & logical game actions

•Ex. button-mapping table

26

JIN 2019 – HCI

Dead Zone

•Analog axis => input values between predefined range 𝐼𝑚𝑖𝑛 and 𝐼𝑚𝑎𝑥

•Ideal = control not touched => steady and clear “undisturbed” value 𝐼0•𝐼0 = 0

•𝐼0 = (𝐼𝑚𝑖𝑛 + 𝐼𝑚𝑎𝑥)/2 or 𝐼0 = 𝐼𝑚𝑖𝑛

•In practice: •Noisy voltage produced by device

•No input fluctuates around 𝐼0

•Definition of a dead zone•Any input values within the dead zone clamped to 𝐼0

•[𝐼0 – 𝛿 , 𝐼0 + 𝛿] for a joystick

•[𝐼0, 𝐼0 + 𝛿] for a trigger

•Wide enough to deal with the most noisy inputs generated by a normal device

•Small enough not to interfere with the responsiveness expected by the player

27

JIN 2019 – HCI

Analog Signal Filtering

•A noise signal is usually of a high-frequency, relative to the signal produced by the player

•Examples•Discrete 1st order low-pass filter

•Combine the current unfiltered input value with last frame’s filtered input

•𝑓 𝑡 = 1 − 𝑎 𝑓 𝑡 − ∆𝑡 + 𝑎. 𝑢 𝑡

•𝑤𝑖𝑡ℎ 𝑎 =∆𝑡

𝑅𝐶+∆𝑡𝑎𝑛𝑑 RC constant

•Average on 𝑛 frames•Store the input data in a n-element circular buffer

28

JIN 2019 – HCI

Detecting Button Up and Down

•Bit-wise operators to compare buttons’ state bits between

frames

•Previous XOR Current => 1 for changed buttons

•Result AND Current => button-down event

•Result AND NOT Current => button-up event

29

JIN 2019 – HCI

Detecting Chords

•Group of buttons pressed at the same time•Watch the states of the buttons

•Perform the requested operation when all of them are down

•Problem 1: Ambiguities if chord includes a button assigned to an action

•Do to perform both

•Check that the other chord buttons are not down when detecting the individual button-presses

30

JIN 2019 – HCI

Detecting Chords

•Problem 2: Humans rarely press buttons in the same frame•Game design such that a chord does all the actions of the individual buttons + additional action

•Delay between individual button-down event detection and valid game event

•During the delay period, a detected chord prevails over the individual button-down events

•Detect the chord when buttons pressed, but trigger effect when released

•Begin the single-button move immediately and preempt it by the chord

31

JIN 2019 – HCI

“Gesture”

•Sequence of actions performed within maximum time-frame

•Implemented with a history of actions•1st detected component

•Add to buffer, with a time stamp

•Each subsequent detected component•Time difference -> if within the time-frame, add to buffer

•If entire sequence completed within the time-frame •Generate event

•If any non-valid intervening inputs detected, or if any component outside valid time window

•Reset buffer

32

JIN 2019 – HCI

Robustness

•Managing multiple devices for multiple players

•One-to-one mapping between controller index and player index

•Assigning controllers to players when start

•Detecting low-battery conditions

•Game or OS

•Unobtrusive on-screen message and/or a sound effect

•Lost connection

•Ex: controller being unplugged or running out of batteries

•Usually pause gameplay, display a message, and wait for the controller to be reconnected

•Multiplayer games

•Suspend or temporarily remove the avatar corresponding to a removed controller

•Allow the other players to continue playing the game

•The removed/suspended avatar might reactivate when the controller is reconnected

33

JIN 2019 – HCI

Cross-Platform

•Conditional compilation directives in the code, wherever device interactions

•Hardware abstraction layer•Abstract button and axis ids -> enum AbstractControlIndex

•Translate between raw control ids on the current target hardware and abstract control ids

•Ex.: bit-swizzling operation to rearrange the bits of the buttons 32-bits word, separate the only trigger axis into two distinct L&R axes on the Xbox, scaling the values appropriately so the range of valid values is the same on all platforms…

•Functional approach•Naming the abstract controls according to their function in the game, rather than their physical locations on the joypad

•Introduce higher-level functions that detect abstract gestures, with custom detection code on each platform

•Platform-specific versions of all of the game code that requires device I/O

34

JIN 2019 – HCI

Input Re-Mapping

•Many games allow to choose the controls•Ex: sense of the vertical axis for camera, predefined button mappings, full control of individual keys

•Level of indirection•Table to maps each control index to a logical function via a unique id

•Different controls produce different kinds of input data: only permit logic control mappings

•Normalize all the inputs and group into classes•Digital buttons: states packed into a 32-bit word, one bit per button

•Unidirectional absolute axes: produce floating-point input values in [0, 1]

•Bidirectional absolute axes: produce floating-point input values in [–1, 1]

•Relative axes: produce floating-point input values in [–1, 1]

35

JIN 2019 – HCI

Context-Sensitive Controls

•A single physical control can have different functions depending on context

•“Use” button -> open, pick up…

•Modal control -> navigate and control camera, steer a vehicle…

•Problem: how to decide the state given the context •Ex: equidistance between 2 items

•State machine, priority system…

•Lots of trial-and-error

36

JIN 2019 – HCI

Control Ownership

•Controls might be “owned” by different parts of the game

•Player control

•Camera control

•Menu system (pausing...)

•“Logical devices”

•Subsets of the inputs on the physical device

•Each assigned to a system (camera, player…)

37

JIN 2019 – HCI

Disabling Inputs

•Ex.:

•Disable all player controls during cinematic

•Disable free camera rotation when walking through a doorway

•Use a bit mask to disable individual controls on the input device

•When needed, neutral or zero value returned instead of the actual value read

•Put the logic in the player or camera code itself

38

JIN 2019 – HCI

Interfacing in practice

•Interactions are the basis of the player mechanics: correct and

smooth handling is an important part of any good game

•Deal with

•Variations between different input devices

•Filtering

•Command mappings

•Achieving the right “feel”

•Limitations from manufacturers (technical requirements checklists TRCs)

•=> Devote significant time and engineering to a careful and

complete implementation of the interface system

39

JIN 2018 – HUMAN INTERFACE DEVICES

Unity

•Input•http://docs.unity3d.com/ScriptReference/Input.html

•Input Manager•Custom axis and buttons, dead zone, gravity, sensitivity, key binding…

•Time

40

PART 2

NOTIONS OF HCI DESIGN & EVALUATION

SOME DEFINITIONS & MODELS

42

JIN 2019 – HCI

User-Centered Design

•Norman & Draper, 80’s

•Iterative design process in which designers focus early on the users, their needs, their tasks and their environment in each phase of the design process

•Active participation of users

•Iteration of solutions, until the needs and requirements expressed by users are fulfilled

43

Source: interaction-design.org

JIN 2019 – HCI

User Experience (UX)

•“User experience encompasses all aspects of the end-user's interaction with the company, its services, and its products.”

•Don Norman and Jakob Nielsen

•Introduced by Norman, Miller & Henderson (95)•Beyond “user interface” and “usability”

•Covers all aspects of a person's experience with a system, including industrial design, graphic elements, interface, physical interaction and instructions for use

•Popularized by Merhloz (98) et Garrett (02), esp. for web design

44

JIN 2019 – HCI

User Experience (UX)

•Requirements1. Meet the exact needs of the client

2. Create products that are a pleasure to own/use (through simplicity and elegance)

•=> True user experience is more than just giving customers what they say they want, or providing checklist features

•=> Seamless merging of services from multiple disciplines: engineering, marketing, graphical, industrial and interface design.

JIN 2019 – HCI

UX vs User Interface

•UI = anything a user may interact with to use a product or service

•Screens, touchscreens, keyboards, sounds, lights...

•UX focuses on the user’s journey through the product

to solve a problem

•UI focuses on how a product’s surfaces look and function, a series of snapshots in time

JIN 2019 – HCI

Usability

•How easy & pleasant the features are to use

•Learnability•How easy to accomplish basic tasks the first time?

•Efficiency•Once users have learned, how quickly can they perform tasks?

•Memorability•When users return after a period of non-use, how easily can they restore their skills?

•Errors•How many errors, how severe, how easily can they recover?

•Satisfaction•How pleasant?

JIN 2019 – HCI

Designer vs. User Perspectives

48

Source: Marc Hassenzhal, The thing and I: Understanding the relationship between user and product (2003)

JIN 2019 – HCI

UX Honeycomb (Web)

•Useful•Fill a need. Overwise no real purpose for the product itself.

•Usable•Easy to use, familiar to understand, short learning curve.

•Desirable•Aesthetics attractive and easy to translate.

•Findable•Information easy to navigate. Able to quickly find a solution to a problem.

•Accessible•Users with disabilities can have the same user experience as others.

•Credible•The company and its products or services need to be trustworthy.

•Valuable•Value to the business which creates it and to the user who buys or uses it,

Peter Morville Peter Morville

Peter Morville

Source: Peter Morville

JIN 2019 – HCI

Affordance (Gibson 86)

•An action a user perceives as possible

•The quality of an object that communicates a way to use it

•Hyperlink, drawer handle…

•In game design will mean that the players know how to interact with items without significant instruction

Be careful of misleading

affordance…

JIN 2019 – HCI

Dark patterns

•Elements of product design created to make users do things they might not want to do—actions that benefit the business, not users

SOME EVALUATION METHODS

JIN 2019 – HCI

User Experience Research

•Practice of analysing a user’s interaction with the product to find insights and identify weaknesses•Qualitative methods

•Focus on observation rather than gathering numerical data

•Suitable for answering questions about the reasons that caused a problem

•Interview, diary studies, usability testing…

•Quantitative methods•Suitable for measuring success or discovering shortcomings

•Best to answer questions like “how many,” “how much,” “how often,”...

•Clickstream analytics, A/B testing, survey…

JIN 2019 – HCI

Usability testing

•Iterative method of testing few functionalities of a digital product by observing real users as they attempt to complete tasks on it

•Goals•Get user reactions and feedback

•Check if the user can perform the tasks proposed

•See if product meet user’s expectations

•Check if the design is matching business decision to real world use

JIN 2019 – HCI

Usability testing in practice

•Ask users to externalize thoughts and feelings

•Keep test environment as realistic as possible

•Do not lead the users•It's not about what you think of a good user experience, but how the user perceives the solution. so shut up, relax and listen.

•Takes notes & record/log the session•Focus on what users do

•Quantitative information: time on tasks, success and failure rates, effort (#clicks, perception of progress)

•Qualitative information: stress responses, subjective satisfaction, perceived effort or difficulty

•Do not jump into any conclusions during the session

JIN 2019 – HCI

5 seconds test

•Measure what information users take away and what impression they get within the first 5s after viewing a design

•Commonly used to check whether web pages effectively communicate their intended message

•Ex: What is the purpose of the page? What are the main elements you recall? Who do you think the intended audience is? Did the design/brand appear trustworthy?

•Not suited to measure comprehension of complex information•A page that requires lots of reading (prefer a design survey)

•Predicting user behavior (prefer a click test or navigation test)

•Asking complex questions (prefer a design survey)

JIN 2019 – HCI

AttrakDiff

•Hassenzahl, Burmester & Koller, 2003

•Questionnaire•Pragmatic quality

•Hedonic quality (identity - stimulation)

•Attractiveness

•Helps to understand how users personally rate the usability and design of the product

•Single Evaluation

•A/B Comparison

•Before-After Comparison

JIN 2019 – HCI

Heuristic evaluation

•Small set of evaluators examine the interface and judge its compliance with recognized usability principles (the "heuristics")

•Advantages•Low cost, quick and easy to apply

•Can obtain feedback early in the design process

•Heuristics can be used both as a design and evaluation support

JIN 2019 – HCI

Heuristic evaluation

•1: Visibility of system status

•2: Match between system and the real world

•3: User control and freedom

•4: Consistency and standards

•5: Error prevention

•6: Recognition rather than recall

•7: Flexibility and efficiency of use

•8: Aesthetic and minimalist design

•9: Help users recognize, diagnose, and recover from errors

•10: Help and documentation

•NB : Applicable to video games

10 Usability Heuristics for User Interface Design (Jakob Nielsen 1994)

59

JIN 2019 – HCI

Heuristic evaluation

•Derived from the flow theory (Csíkszentmihályi, 1975)

•1. Clear Goals

•2. Appropriate Feedback

•3. Focused Concentration

•4. Ergonomical Transparency

•5. Technology Appropriation

•6. Challenges/Skills Balance

•7. Potential control

•8. Follow the Rhythm

•9. Know Thy User’s Motivations

•10. Conservative Innovation

10 Heuristics for an optimal User Experience (Colombo & Pasch, 2012)

60

JIN 2019 – HCI

Heuristic evaluation

•Ergonomic criteria for the evaluation of human-computer interfaces (Bastien & Scapin 92)

•Ergonomic criteria for Human-Virtual Environments Interations (Bach & Scapin 2005)

•Playability heuristics for mobile games

•…

Others

61

JIN 2019 – HCI

Heuristic evaluation

•Requires knowledge and experience to apply the heuristics effectively

•Judgment often based on expertise rather than heuristics

•Trained usability experts are sometimes hard to find and expensive

•May identify more minor issues and fewer major issues, or even false issues

•Non-exhaustiveness of the dimensions covered by the heuristics

•The heuristics are often vague, no precise recommendations or evaluation grid/criteria

•=> Limited validity and reliability, recommended to use it in combination with other user-centered methods

Drawbacks

62

JIN 2019 – HCI

Expert review

•Less formal evaluation

•Experts base their report not only on heuristics, but rather on their knowledge of user tasks, HCI guidelines and standards, and personal experience

63

JIN 2019 – HCI

Further readings

•https://uxdesign.cc/

•https://www.smashingmagazine.com/usability-and-user-experience/

•https://www.nngroup.com/articles/

•https://medium.com/topic/ux

•https://uxplanet.org

•http://www.allaboutux.org/

64