1a8 parrallel ui rendering

1/16Kris Van Hees & Jan Engelen

Parallel User Interface Rendering~

Accessibility for Custom Widgets

Kris Van Hees & Jan EngelenKatholieke Universiteit Leuven

Department of Electrical EngineeringESAT - SCD - DocArch


● Introduction● Parallel User Interface Rendering● Widgets● Cosmetic widgets● Conclusions● Future...

Agenda


● GUIs in all aspects of life: computers, MP3 players, home appliances, mobile phones, ...

● Inherent visual interaction model poses a complication for users with disabilities (especially blindness)

● Mixing graphical toolkits within a single environment complicates matters even more (Linux: Qt, GTK, Athena, …)

● Custom widgets: often not supported in assistive technology without fancy scripting (if available)

Introduction


● Abstracting the user interface offers advantages:– Unified user interaction semantics across toolkits and

modalities– Retaining flexibility in rendering for multiple modalities– Assistive technology as 1st generation rendering

● Parallel User Interface Rendering (PUIR) provides for equivalent concurrent renderings of the UI

Introduction (cont...)


● User interface defined in abstract form (XML file)– Effectively defines data and how to present data

● Abstract UI description interpreted by an AUI engine– Handles user interaction semantics

● UI presentation rendered by modality-specific agent(s)● Rendering agent depends on system specific toolkits● Rendering agents provide presentations concurrently

– Provide both visual and non-visual rendering

Parallel User Interface Rendering


● Interprets an abstract UI description (XML file)● Implements user interaction semantics● Encapsulates data in the UI● Provides focus management● Dispatches events to the application and rendering agents● Can be called from application and rendering agents● Has no knowledge of implementation details of

application and rendering agents

PUIR: The AUI engine


● Java Swing based in-process rendering agent● Complex changes to bypass user interaction in widgets● Reduced to provide merely the presentation of widgets● Some widgets operate slightly differently● Pointer device UI interaction interpreted in the rendering

agent:– Necessary because position is tied to GUI context– Semantic operations mapped to AUI operations

PUIR: GUI rendering agent


● Remote rendering agent● Basic speech output presentation as proof-of-concept● Proxy objects for transparency and caching● Equivalent with GUI rendering agent:

– Receives the same events– Synchronized with other rendering agents

PUIR: Non-visual rendering agent


● Keyboard:– Textual input– Navigation, text selection, editing, …– Processed directly by AUI (for proof-of-concept)

● Mouse:– Processed by rendering agent– Position to be interpreted within graphical context– Semantic user interaction delegated to AUI engine

PUIR: Input processing


● Braille keyboard:– Coded textual input translated into text– Text processed as if entered on regular keyboard– Limited cursor addressable positional input– Positional input interpreted in context– Semantic user interaction delegated to AUI engine

PUIR: Input processing (cont...)


● Two entities per widget:– Abstract widget:

● Operates entirely within the AUI engine● Implements user interaction semantics● Encapsulates data

– Widget presentation:● Usually one per rendering agent● Implements rendering in output modality

Widgets


● Design:– Define UI in abstract UI description– Typically by means of design tool– Rendering specific information can be included

Widgets (cont...)


● Runtime construction:– Instantiation of UI based on AUI description– For every widget:

● Find built-in abstract widget class● If none, search in alternate locations● If none, UI cannot be instantiated

Widgets (cont...)


● Rendering:– One-to-one correspondence between abstract widget

class and presentation class– For every widget:

● Find built-in widget presentation class● If none, search in alternate locations● If none and cosmetic widget, use core alternate

(more on this 2 slides from now...)

● If none, use default place-holder widget

Widgets (cont...)


● Rendering (cont...):– Rendering agent knows what widgets are supported– Rendering agent can alert user appropriately– Rendering agent can query information from AUI

about the unsupported widget– Accidental user interaction is guarded against

Widgets (cont...)


● Widgets with a very specific presentation

● Typically derivative of an existing core widget

● User interaction semantics remain the same

● AUI description specifies equivalence between cosmetic widget and core widget

● Core widget can act as default presentation when no custom presentation class is found

Cosmetic widgets


● Left-to-right:

– Four regular buttons

– VU-meter style widget: equivalent with numeric label

– Four knobs: equivalent with discrete slider

– Four indicator lights: equivalent with label

– Four regular buttons

Cosmetic widgets (cont...)


● Parallel User Interface Rendering– Powerful technique for Design-for-All– Coherence between concurrent equivalent renderings– Generic: not just for non-visual renderings– Robust handling of custom widgets– Support for custom widgets independent from

applications and specific uses– Can be used in automated application testing as well

Conclusions


● Completion of proof-of-concept● More testing● Public release

● Default presentation of custom widgets merely based on abstract widget information

● Remote operation of applications

At all stages, feedback from real users is be crucial!

Future...


Kris Van Hees & Jan EngelenKatholieke Universiteit Leuven

Department of Electrical EngineeringESAT – SCD – DocArchKasteelpark Arenberg 10

B-3001 HeverleeBelgium

[email protected], [email protected]

1a8 parrallel ui rendering

Education

Transcript of 1a8 parrallel ui rendering