Lecture 12

implementation support

Today’s Lecture programming tools

levels of services for programmers windowing systems

core support for separate and simultaneous user-system activity programming the application and control of dialogue interaction toolkits

bring programming closer to level of user perception user interface management systems

controls relationship between presentation and functionality

Introduction We have spent much effort up to this point

considering design and analysis of interactive systems from a relatively abstract perspective.

Now we will focus the task of coding the interactive application and how it is structured.

Introduction The detailed specification gives the programmer

instructions as to what the interactive application must do. The programmer must translate that into machine executable

instructions to say how that will be achieved on the available hardware devices.

The objective of the programmer then is to translate down to the level of the software that runs the hardware devices.

Quest of this lectureHow does HCI affect of the programmer?

Advances in coding have elevated programminghardware specific

interaction-technique specific

Layers of development tools windowing systems interaction toolkits user interface management systems

Elements of windowing systems The first important feature of a windowing

system is its ability to provide programmer independence from the specifics of the hardware devices. A typical workstation will involve some visual

display screen, a keyboard and some pointing device, such as a mouse.

Windowing Systems.. It is imperative to be able to program an

application that will run on a wide range of devices.

The programmer wants to direct commands to an abstract terminal, which understands a more generic language and can be translated to the language of many other specific devices.

Windowing Systems.. Besides making the programming task easier,

the abstract terminal makes portability of application programs possible. Only one translation program – or device driver –

needs to be written for a particular hardware device and then any application program can access it.

Windowing Systems.. A given windowing system will have a fixed generic language

for the abstract terminal which is called its imaging model. The imaging models are sufficient to describe very arbitrary

images. For efficiency reasons, specific primitives are used to handle

text images, either as specific pixel images or as more generic font definitions.

Examples of Imaging Model Pixels

The display screen is represented as a series of columns and rows of points – or pixels – which can be explicitly turned on or off, or given a color. This is a common imaging model for personal computers and is also used by the X windowing system.

Examples of Imaging Model Graphical kernel system (GKS)

An international standard which models the screen as a collection of connected segments, each of which is a macro of elementary graphics commands.

Examples of Imaging Model Programmer’s hierarchical interface to

graphics (PHIGS)Another international standard, based on GKS

but with an extension to model the screen aseditable segments.

Examples of Imaging Model PostScript

A programming language developed by Adobe Corporation which models the screen as a collection of paths which serve as infinitely thin boundaries or stencils which can be filled in with various colors or textured patterns and images.

Imaging Models Though these imaging models were initially

defined to provide abstract languages for output only, they can serve at least a limited role for input as well. For example, the pixel model can be used to

interpret input from a mouse in terms of thepixel coordinate system.

Elements of Windowing System It would then be the job of the application to process

the input event further once it knows where in the image it occurred.

The other models above can provide even more expressiveness for the input language. They can relate the input events to structures that are

identifiable by the application program.

Sharing Windowing support several separate user

tasks simultaneously. Sharing the resources of a single hardware

configuration with several copies of an abstract terminal.

Each abstract terminal will behave as an independent process and the windowing system will coordinate the control of the concurrent processes.

Sharing To ease the programming task again, this

coordination of simultaneously active processes can be factored out of the individual applications, so that they can be programmed as if they were to operate in isolation.

Sharing The window system must also provide a means of

displaying the separate applications. accomplished by dedicating a region of the display screen to

each active abstract terminal. The coordination task then involves

resolving display conflicts when the visible screen regions of two abstract terminalsoverlap.

roles of a windowing system

Architectures of windowing systems

three possible software architectures all assume device driver is separate differ in how multiple application management is implemented

1. each application manages all processes everyone worries about synchronization reduces portability of applications

2. management role within kernel of operating system applications tied to operating system

3. management role as separate applicationmaximum portability

The client-server architecture

X Window System A classic example of a window system based on the client–

server architecture is the industry-standard X Window System (Release 11), developed at the Massachusetts Institute of Technology (MIT) in the mid-1980s. It is based on a pixel-based

imaging model and assumes that there is some pointing mechanism available.

X Windows X adoption is due to a standard.

X is based on a network protocol which clearly defines the server–client communication.

The X Protocol can be implemented on different computers and operating systems, making X more device independent.

X Windows It also means that client and server need

not even be on the same system in order to communicate to the server.

X Windows The X server performs the following tasks:

allows (or denies) access to the display from multiple client applications;

interprets requests from clients to perform screen operations or provide otherinformation;

De-multiplexes the stream of physical input events from the user and passes themto the appropriate client;

Tasks by X server minimizes the traffic along the network by relieving the

clients from having to keep track of certain display information, like fonts, in complex data structuresthat the clients can access by ID numbers.

X Windows architecture

X Windows architecture (ctd) pixel imaging model with some pointing mechanism

X protocol defines server-client communication

separate window manager client enforces policies for input/output: how to change input focus tiled vs. overlapping windows inter-client data transfer

Programming the Application We now concentrate our attention on programming

the actual interactive application, which would correspond to a client in the client–server architecture.

Interactive applications are generally user driven in the sense that the action theapplication takes is determined by the input received from the user.

Programming paradigms Two paradigms

read–evaluation loopNotification based

Read Evaluation loop The server sends user inputs as structured events to

the client application. As far as the server is concerned, the only

importance of the event is the client to which it must be directed.

The client application is programmed to read any event passed to it and determine all of the application-specific behavior that results as a response to it

Programming the application - 1

read-evaluation loop

repeatread-event(myevent)case myevent.type

type_1:do type_1 processing

type_2:do type_2 processing

...type_n:

do type_n processingend case

end repeat

Notification based The main control loop for the event

processing does not reside within the application.A centralized notifier receives events from the

window system and filters them to the application program in a way declared by the program.

Programming the application - 1

notification-basedvoid main(String[] args) {

Menu menu = new Menu();menu.setOption(“Save”);menu.setOption(“Quit”);menu.setAction(“Save”,mySave)menu.setAction(“Quit”,myQuit)

...}

int mySave(Event e) {// save the current file

}

int myQuit(Event e) {// close down

}

Going with the grain– Design Focus

system style affects the interfaces modal dialogue box

easy with event-loop (just have extra read-event loop)

hard with notification (need lots of mode flags)

non-modal dialogue box hard with event-loop (very complicated main loop) easy with notification (just add extra handler)

beware!

if you don’t explicitly design it will just happen implementation should not drive design

Using Toolkits A key feature of WIMP interfaces from the user’s

perspective is that input and output behaviors are intrinsically linked to independent entities on the display screen. This creates the illusion that the entities on the screen are

the objects of interest –interaction objects we have called them – and that is necessary for the action world of a direct manipulation interface.

Using ToolkitsA classic example is the mouse as a pointing

device. The input coming from the hardware device is

separate from the output of the mouse cursor on the display screen.

However, since the visual movement of the screen cursor is linked with the physical movement of the mouse device, the user feels as if he is actually moving the visual cursor.

Using toolkitsInteraction objects

input and outputintrinsically linked

Toolkits provide this level of abstraction programming with interaction objects (or techniques, widgets, gadgets) promote consistency and generalizability through similar look and feel amenable to object-oriented programming

move press release move

Interfaces in Java – design Focus

Java toolkit – AWT (abstract windowing toolkit)

Java classes for buttons, menus, etc.

Notification based; AWT 1.0 – need to subclass basic widgets AWT 1.1 and beyond -– callback objects

Swing toolkit built on top of AWT – higher level features uses MVC architecture (see later)

User Interface Management Systems (UIMS)

UIMS add another level above toolkits toolkits too difficult for non-programmers

concerns of UIMS conceptual architecture implementation techniques support infrastructure

non-UIMS terms: UI development system (UIDS) UI development environment (UIDE)

e.g. Visual Basic

UIMS as conceptual architecture

separation between application semantics and presentation

improves: portability – runs on different systems reusability – components reused cutting costs multiple interfaces – accessing same functionality customizability – by designer and user

UIMS tradition – interface layers / logical components

linguistic: lexical/syntactic/semantic

Seeheim:

Arch/Slinky

presentation dialogue application

dialogue

lexical

physicalfunctional

core

func. core adaptor

Seeheim model

PresentationDialogueControl

Functionality(applicationinterface)

USERUSER APPLICATION

switch

lexical syntactic semantic

conceptual vs. implementationSeeheim

arose out of implementation experience but principal contribution is conceptual concepts part of ‘normal’ UI language

… because of Seeheim …… we think differently!

e.g. the lower box, the switch needed for implementation but not conceptual

presentation dialogue application

Semantic feedback different kinds of feedback:

lexical – movement of mouse syntactic – menu highlights semantic – sum of numbers changes

semantic feedback often slower use rapid lexical/syntactic feedback

but may need rapid semantic feedback freehand drawing highlight trash can or folder when file dragged

What’s this?

USER

Lexical Syntactic Semantic

APPLICATIONApplicationInterfaceModel

DialogueControl

Presentation

the bypass/switch

USER

Lexical Syntactic Semantic

APPLICATIONApplicationInterfaceModel

DialogueControl

Presentation

rapid semantic feedback

direct communicationbetween application

and presentation

but regulated bydialogue control

more layers!

dialogue

lexical

physicalfunctional

core

func. core adaptor

Arch/Slinky more layers! – distinguishes lexical/physical like a ‘slinky’ spring different layers may be thicker (more

important) in different systems or in different components

dialogue

lexical

physicalfunctional

core

func. core adaptor

monolithic vs. components Seeheim has big components

often easier to use smaller ones esp. if using object-oriented toolkits

Smalltalk used MVC – model–view–controller model – internal logical state of component view – how it is rendered on screen controller – processes user input

MVCmodel - view - controller

model

view

controller

MVC issues MVC is largely pipeline model:

input control model view output

but in graphical interface input only has meaning in relation to output

e.g. mouse click need to know what was clicked controller has to decide what to do with click but view knows what is shown where!

in practice controller ‘talks’ to view separation not complete

PAC model PAC model closer to Seeheim

presentation – manages input and output abstraction – logical state of component control – mediates between them

manages hierarchy and multiple views control part of PAC objects communicate

PAC cleaner in many ways …but MVC used more in practice

(e.g. Java Swing)

PACpresentation - abstraction - control

abstraction presentation

control

A PC

A PC

A PC A P

C

Implementation of UIMS Techniques for dialogue controller

• menu networks • state transition diagrams

• grammar notations • event languages

• declarative languages • constraints

• graphical specification

for most of these see chapter 16

N.B. constraints instead of what happens say what should be true used in groupware as well as single user interfaces

(ALV - abstraction–link–view)

see chapter 16 for more details on several of these

Graphical Specification what it is

draw components on screen set actions with script or links to program

in use with raw programming most popular technique e.g. Visual Basic, Dreamweaver, Flash

local vs. global hard to ‘see’ the paths through system focus on what can be seen on one screen

The drift of dialogue control internal control

(e.g., read-evaluation loop)

external control(independent of application semantics or presentation)

presentation control(e.g., graphical specification)

Implementation Consideration How components in a conceptual

architecture can be realized? Implementations based on the Seeheim

model must determine how the separate components of presentation, dialog controller and application interface are realized.

Implementation Considerations Window systems and toolkits provide the separation

between application and presentation. The use of callback proceduresin notification-based programming is one way to implement the application interface as a notifier.

In the standard X toolkit, these callbacks are directional as it is the duty of the application to register itself with the notifier.

Implementation considerations Myers has outlined the various implementation techniques used to specify the dialog controller

separately. Which are, Menu networks Grammar notations State transition diagrams Event languages Declarative languages Constraints and Graphical specification

Implementation Techniques Menu networks: The communication

between application and presentation ismodeled as a network of menus and submenus. The menu is used to embody all possible user

inputs at any one point in time.

Implementation Techniques Grammar notations: The dialog between

application and presentation can be treated as a grammar of actions and responses, and, therefore, described by means of a formal context-free grammar notation, such as BNF (Backus–Naur form)

Implementation Techniques State transition diagrams: State transition

diagrams can be used as a graphical means of expressing dialog.

Event languages: Event languages are similar to grammar notations, except that they can be modified to express directionality and support some semantic feedback.

Implementation Techniques Declarative Languages: A declarative approach

concentrates more on describing how presentation and application are related.

Constraints: Constraints embody dependencies between different values that must always be maintained.

Implementation Techniques Graphical specification: These

techniques allow the dialog specification to be programmed graphically in terms of the presentation language itself.

SummaryLevels of programming support tools Windowing systems

device independence multiple tasks

Paradigms for programming the application read-evaluation loop notification-based

Toolkits programming interaction objects

UIMS conceptual architectures for separation techniques for expressing dialogue