How to Measure Information: - from Action to Cognition - Matthias Rauterberg Department of...

How to Measure Information:- from Action to Cognition -

Matthias RauterbergDepartment of Industrial DesignTechnical University Eindhoven

2004

© M. Rauterberg, 2004

2/35

Possible interpretations of ‘information'

1.) ‘information' as a message (syntax)

2.) ‘information' as the meaning of a message (semantic)

3.) ‘information' as the effect of a message (pragmatic)

4.) ‘information' as a process

5.) ‘information' as knowledge (e.g. database)

6.) ‘information' as an entity of the world (next to energy and matter)

Ref: Folberth, O. & Hackl, C. (1986, eds.) Der Informationsbegriff in Technik und Wissenschaft. München: Oldenbourg.


3/35

“Information” for learning systems

before reception after reception author

DoF of the decision content of the decision HARTLEY 1928

uncertainty certainty SHANNON 1949 uncertainty information BRILLOUIN 1964 potential information actual information ZUCKER 1974 entropy amount of information TOPSØE 1974


4/35

context human

mental model

situation-1

situation-2

complexity

positive incongruity

negative incongruity

Incongruity and learning

incongruity = complexitycontext – complexityhuman

learning

Ref: Rauterberg, M. (1995). About a framework for information and information processing of learning systems. In: E. Falkenberg, W. Hesse & A. Olive (eds.), Information System Concepts--Towards a consolidation of views (IFIP Working Group 8.1, pp. 54-69). London: Chapman&Hall.


5/35

physical operation

feedback control of action

goal-, sub-goal-setting

mental operation

task(s)

planning of execution selection of means

The complete human action cycle

direct observable

not direct observable


6/35

s0

d

s1

h

s0

a

s2

F3

s3

CR

s3

F9

s1

_

s3

TAB

s3

F2

s3

_

s3

TAB

s3

_

s3

The basic idea

Any human task solving process can be described in a finite state-transition chain, if the task can be described in an ‘action space’, specified by a finite set of States and Transitions.

State description: s0 : main menu s1 : modul "data" s2 : routine "browse" s3 : "wrong input" state

Action description: _ : ascii key "BLANK" a : ascii key "a" d : ascii key "d" h : ascii key "h" CR: carriage return F2: function key "2" F9: function key "9" TAB:tabulator key


7/35

s0 d s1

s1 h s0

s1 a s2

s2 F3 s3

s3 CR s3

s3 F9 s1

s3 _ s3

s3 TAB s3

s3 F2 s3

elementary processes Petri-Net

s0

d h

s1

a

s2s3

F3

F9

CR

_

TAB

F2

folding

The folding operation in Petri nets


8/35

Task description

In the experiment all 12 users had to play the role of a camping place manager. This manager uses a database system with a data base consisting of three data files: PLACE, GROUP, and ADDRESS. All users had to solve the following four different tasks operating the database system:

Task 1: "How many data records are in the file ADDRESS, in the file PLACE, and in the file GROUP? Find out, please."

The user has to activate a specific menu option ("Datafile" in module "Info" of the menu interface) and to read the file size (solutions: PLACE = 17 data records, GROUP = 27 data records, ADDRESS = 280 data records).

Task 2: "Delete only the last data record of the file ADDRESS, the file PLACE, and the file GROUP (sorted by the attribute 'namekey')."

The user has to open (sorted according to the given attribute), select and delete the last data record (file: PLACE, GROUP, ADDRESS).

Task 3: "Search and select the data record with the namekey 'D..8000C O M' in the file ADDRESS, and show the content of all attributes of this data record on the screen. Correct this data record for the following attributes: State: Germany, Place number: 07. Remarks: Database system dealer can give a demonstration."

The user must select a certain data record (file: ADDRESS), update the data record with regard to the three attributes: State, Place number, Remarks.

Task 4: "Define a filter for the file PLACE with the following condition: all holidaymakers arrived on date 02/07/87. Apply this filter to the file PLACE, and show the content of all selected data records in the mask browsing mode on the screen."

The user must define a filter for the attribute "arrival date", apply the filter to the data file PLACE, and display the content of each data record found on the screen.


9/35

System description

The dialog system was the relational data base system (ADIMENS version 2.21) with a character oriented user interface (CUI) running on standard IBM PC's with standard keyboard.

The whole dialog structure is strictly hierarchical organized with three levels:

(1) the main menu has 7 dialog operations (ordinary ASCII characters chosen from a menu) to go down to 7 different modules, and 5 function keys with specific semantics;

(2) at the module level each module has exactly 4 different dialog operations to change to routines and on average 4.1 (±1.7; range: 0-5) function keys with specific semantics;

(3) at the routine level the user has only on average 3.7 (±2.9; range: 0-10) different function keys to control the dialog (additionally all ASCII keys and the 4 cursor keys are usable).

The number of all ordinary dialog contexts (main menu, modules, routines) is 1+7*4=29.

But to describe the complete dialog structure with all help, error and additional dialog states we needed at least 144 different system states.

To change from one state to the other the system offers overall 358 different dialog operations (= transitions).


10/35

physical execution

evaluation and control

goal-, sub-goal setting

mental execution

task description

action planning selection of means

goal

system state

selected action

result

Observable data of human behavior

transitionstate


11/35

G_2

Start menu

M_3

Main menu Main menu. F file

F_3

User key press

1

User key press

Main menu

i

User key press

Info

Info.file Info.screen1 Info.screen2

d

M_22 M_22 M_22

DB content display

User key press

Info.screen3 Info

Info.screen1 stopped

Info Main menu Start menu Start menu Start menu

M_11 d

Automatic transition

User key press

M_22 M_22 M_22 M_11 d

User key press

M_22 BL F_10 M_22 M_22

M_11 h h

User key press

User key press

G_2 F_10

User key press

... continues

... continues

... continues

... continues






Main menu

marked

Info.file Info.screen1 Info.screen2 Info.screen3 Info

Info.screen1Info.file

Info.screen3

Info.screen2Info.screen1

DB content display

DB content display

DB content display

DB content display

DB content display

User key press

User key press

DB content display

DB content display

DB content display

Example of a task solving process


12/35

s0

d

s1

h

s0

b

s2

F3

s3

CR

s3

F9

s1

_

s3

TAB

s3

F2

s3

_

s3

TAB

s3

_

s3

structure as a

Petri net

s0

d h

s1

b

s2s3

F3

F9 CR

_

TAB

F2

FOLDING

observable process

unknown structure (e.g., mental model)

?

main menu level

module level

routine level

How to extract the user’s mental model?

1 2


13/35

How to measure complexity?

In Computer Science... • algorithmic information (e.g., Solomonoff-Kolmogorov-Chaitin) • computational universality • computational time/space • according McCabe in graph theory

In Physics... • thermodynamics potentials • long-range order • long-range mutual information • self-similar structures • thermodynamic depth • logical depth

In Psychology... • properties of objects (e.g. valence) • properties of attributes (e.g. cardinality) • properties of cognitive structure (e.g. centrality)


14/35

Net complexity metrics

McCabe (1976):

Kornwachs (1987):

Stevens, Myers and Constantine (1974):

[with P=1]

Validation study:Ccycle from McCabe outperforms all other metrics!

State-1

State-2

Transition-2Transition-1

simple Petri net:

Ref: Rauterberg, M. (1992). A method of a quantitative measurement of cognitive complexity. In: G. van der Veer, M. Tauber, S. Bagnara & M. Antalovits (eds.), Human-Computer Interaction: Tasks and Organisation--ECCE'92 (pp. 295-307). Roma: CUD.

T = number of transitionsS = number of states


15/35

state transition net

interactive dialog system

automatic recorded process

system description

adjacency matrixfrequency matrix

the analyzing program AMME

ascii text outputfile with

quantitative measures

graphic outputfile in PostScript

format

0 1 2 3 0 0 1 0 0 1 3 0 1 0 2 0 0 0 1 3 0 2 0 7

0 1 2 3 0 0 1 0 0 1 1 0 1 0 2 0 0 0 1 3 0 1 0 1

• simulation • task-subtask

• similarity • learning • MDS

• distances • personal styles • MDS

• complexity • routine

• interface design • deadlocks

USER

"defaultp.ps"

Petri net simulator PACE

"*.str""*.log"

Path finder KNOT

Markov analyzer SEQUENZ

transformation to a syntactical correct logfile

v2132 13 S'initial_state' 15@-12

390@330 S 0 0 Tnil nil 480@420

S 1 2 nil 0 cS CSCS cSsS SSRS rS 5 tftt 10 ft

"*.net" "*.ptf" "*.mkv" "*.pro""*.ps"

any Postscript interpreter

any text processor

The program structure of AMME

In an overview Ivory and Hearst (2001) compared 132 usability evaluation and modeling methods worldwide; 19 different modeling methods are based on logfile analysis: “AMME is the only surveyed approach that constructs a WIMP simulation model (Petri net) directly from usage data” (Ivory and Hearst 2001, p. 499). Therefore they conclude, “AMME appears to be the most effective method, since it is based on actual usage” (2001, p. 502).

Automatic Mental Model Evaluation


16/35

BCcycle = T – S + 1

BCcycle

Box Plot Box Plot

1 20

10

20

30

40

*

*

*

beginners - experts1 2 3 4

0

10

20

30

40

*

*

task no.

SOURCE SUM-OF-SQUARES DF MEAN-SQUARE F-RATIO P

experience 275.521 1 275.521 10.337 0.003

tasks 259.563 3 86.521 3.246 0.032

exp. x tasks 25.729 3 8.576 0.322 0.810

ERROR 1066.167 40 26.654

Behavioral Complexity (BC) according McCabe (1976)

Experiment:

N=6 novices; N=6 experts4 tasks with a databaseMetric BC=Ccycle

Ref: Rauterberg, M. (1993). AMME: an Automatic Mental Model Evaluation to analyze user behaviour traced in a finite, discrete state space. Ergonomics, vol. 36(11), pp. 1369-1380.


17/35

Validation of the functional equivalence, computed by the similarity ratio (SR)

Adding goal setting structure

Reconstructed mental task modelHuman mental model

?

Observation of human behaviour

Folding

Adding sequential and temporal information Model

execution

original behavioural sequence simulated behavioural sequence

Device model

2

G_2

Start menu

M_3

Main menu

F_3

User key press



MsDOS

G_2

Main menu

F_3

Start menu

M_3



User key press

MsDOS

Main menu Main menu

F-fileStart menu

F_10 M_3 h F_3 1 i hG_2

MsDOS

Main menu Main menu F-file

Start menu

F_10 M_3 h F_3 1 i hG_2

MsDOS

1

3

4

5

6

i

From analyzing to modeling


18/35

Two different mental models

Model-1 Model-4

Ref: Rauterberg, M. (1995). From novice to expert decision behaviour: a qualitative modelling approach with Petri nets. In: Y. Anzai, K. Ogawa & H. Mori (eds.), Symbiosis of Human and Artifact: Human and Social Aspects of Human-Computer Interaction--HCI'95 (Vol. 20B, pp. 449-454). Elsevier.


19/35

SR 1 org ,tR sim, tR max orgR

sim1N

orgN

t1

simN

org

2N

* 100%

The Similarity Ratio: SR

Legend: R is the absolute rank position in the original or simulated process

Ref: Rauterberg, M. (1995). From novice to expert decision behaviour: a qualitative modelling approach with Petri nets. In: Y. Anzai, K. Ogawa & H. Mori (eds.), Symbiosis of Human and Artifact: Human and Social Aspects of Human-Computer Interaction--HCI'95 (Advances in Human Factors/Ergonomics, Vol. 20B, pp. 449-454). Amsterdam: Elsevier.


20/35

d

a

F3

space

space

TAB

F2

TAB

CR

space

F9

h

original

d

a

F3

TAB

F9

h

d

a

F3

TAB

space

F2

F2

CR

F9

h

d

a

F3

space

CR

space

F9

a

F3

TAB

F9

a

...

d

h

d

h

d

h

d

h

d

a

F3

TAB

CR

CR

TAB

space

F2

F9

h

d

a

F3

space

space

CR

space

F9

h

d

a

F3

F9

a

F3

TAB

CR

space

space

F9

h

40% 77% 76% 10% 10% 10%67%79% 10% 83%

Simulated logfiles with Model-1

43%

d

h

10%

Simulation Results: Model-1

SR


21/35

d

a

F3

space

space

TAB

F2

TAB

CR

space

F9

h

original

d

a

F3

space

TAB

space

TAB

space

F2

CR

F9

h

d

a

F3

space

space

TAB

TAB

space

F2

CR

F9

h

d

a

F3

space

space

TAB

space

TAB

F2

CR

F9

h

d

a

F3

space

TAB

space

TAB

space

F2

CR

F9

h

d

a

F3

space

space

space

TAB

TAB

F2

CR

F9

h

d

a

F3

space

TAB

space

space

TAB

F2

CR

F9

h

d

a

F3

space

TAB

space

TAB

space

F2

CR

F9

h

d

a

F3

space

space

TAB

space

TAB

F2

CR

F9

h

d

a

F3

space

space

TAB

TAB

space

F2

CR

F9

h

d

a

F3

space

TAB

space

TAB

space

F2

CR

F9

h

d

a

F3

space

space

space

TAB

TAB

F2

CR

F9

h

d

a

F3

space

TAB

space

space

TAB

F2

CR

F9

h

94% 94% 96% 96% 94% 94%94%96% 94% 94% 94% 96%

Simulated logfiles with Model-4

95%

Simulation Results: Model-4

SR


22/35

Model-2 (first part): Event-driven goal setting strategy

Main menuMain menu

F-fileStart menu

F_10 M_3 h F_3 1 i hG_2

MsDOS

Syste

m le

vel

Co

gn

itive le

vel

Go

al

insta

ncia

tion

le

vel

Actio

n le

vel

task description

observable action

goal instanciation

goal selection

Approach-2: event-driven goal setting

Ref: Rauterberg, M., Fjeld, M. & Schluep S. (1997). Parallel or event-driven goal setting mechanism in Petri net based models of expert decision behaviour. In: S. Bagnara, E. Hollnagel, M. Mariani & L. Norros (eds.), Time and Space in Process Control--CSAPC'97 (Sixth European Conference on Cognitive Science Approaches to Process Control, pp. 98-102). Roma: CNR.

[Remark: Approach-1 is called ‘pure’ action driven model without the ‘goal selection’ level and only feedback between ‘action’ and ‘goal instanciation’]


23/35

Approach-3: parallel goal setting without feedback

Syste

m le

vel

Co

gn

itive le

vel

Go

al

insta

ncia

tion

le

vel

Actio

n le

vel

Model-3 (first part): Parallel goal setting strategy

M_3 h F_3 1 i hG_2F_10

Main menuMain menu

F-fileStart menuMsDOS

cognitive processtask

description

goal instanciation

observable action


24/35

Go

al

insta

ncia

tion

le

vel

Actio

n le

vel

M_3 h F_3 1 i hG_2F_10

Main menuMain menu

F-fileStart menuMsDOS

Syste

m le

vel

Co

gn

itive le

vel

Fe

ed

ba

ck le

vel

Model-4 (first part): Parallel goal setting with feedback

task description

feedback

cognitive process

observable action

goal instanciation

Approach-4: parallel goal setting with feedback


25/35

The model complexity (Ccycle) and similarity ratio (SR) of the approaches-1, -2, -3 and -4 [std:=standard deviation]. approach

no. 1 approach no. 2

approach no. 3

approach no. 4

Ccycle: (mean ± std):

13 ± 5 43 ± 17 57 ± 25 101 ± 43

Ccycle: (min…max.):

6…18 22…68 30…97 55…170

SR (mean % ± std):

41 ± 28 66 ± 21 88 ± 11 100 ± 0

SR (min…max. %):

3…79 36…98 67…100 100…100

# simulated sequences

5*6=30 5*6=30 5*6=30 5*6=30

Results for the 4 different modeling approaches


26/35

PS = #TST / #DS

Measuring 'Personality Style‘ (PS)

R = #UT / #DT

Measuring 'Routine‘ (R)

Ccycle = #T – #S+P with #S =< #T and P=1

C'cycle = #F–(#T + #S)+P with #S > #T and P=1

Measuring ‘Complexity’ (C)

Overview over additional measures

T = number of transitionsS = number of states

F = number of connectors

TST = task solving time

UT = all used transitions

DT = all different transitions

DS = all different states


27/35

The common assumption of the AI community

observable behaviour

mental model

learning

Ref: Rauterberg, M. (1996). About faults, errors, and other dangerous things. In: C. Ntuen & E. Park (eds.), Human Interaction with Complex Systems: Conceptual Principles and Design Practice (pp. 291-305). Norwell: Kluwer.


28/35

complex system

operator

beginner

advanced

expert

learning time

interaction

SC BC CC

We found a negative correlation between Behavior-Complexity BC and [assumed] Cognitive-Complexity CC

Experiment:

6 novices and 6 real experts4 tasks with a databaseMetric BC=Ccycle

The reality: look, what we found!


29/35

observable behaviour

mental model

How to interpret this negative correlation?

learning


30/35

Mental knowledge structures: a metaphor

s0

d h

s1

b

s2 s3F3

F9

CR

_

TAB

F2

“hill“: knowledge about unsuccessful behavior

"dale“: knowledge about successful behavior

This new view would have major impact e.g. on training procedures of operators of complex systems!

3D picture


31/35

Learning: the traditional understanding

before learning phase after learning phase

Learning is seen as ‘digging dales’!


32/35

Mental decision making for concrete actions is like rolling a ball between hills,consisting of two kinds of knowledge: successful and unsuccessful!

Decision and action: a new view


33/35

Learning and experience

task complexity

time-1 time-2 time-3

time

task-1 task-1' task-1''

Ref: Rauterberg, M. & Aeppli, R. (1995). Learning in man-machine systems: the measurement of behavioural and cognitive complexity. In: Proceedings of IEEE International Conference on Systems, Man and Cybernetics--SMC'95 (Vol. 5, IEEE Catalog Number 95CH3576-7, pp. 4685-4690). Piscataway: Institute of Electrical and Electronics Engineers.


34/35

2

4

6

8

10

12

14

16

18

20

task 1

week-3week-2week-1

task 1' task 1 task 1' task 1 task 1'21

22

23

24

25

26

27

28

task 1

week-3week-2week-1

task 1' task 1 task 1' task 1 task 1'

The learning experiment

Time structure and knowledge structure are different!

Task solving time Behavioral complexity

N=6 beginners (all male, average age of 25 ± 3 years)


35/35

Conclusions

• A valid metric for task complexity based on task structure allows an objective comparison

• Automatic analysis for unconstrained task solving behavior allows analysis with applied statistics

• This new analysis and modeling approach with AMME leads to new insights…


36/35

Thank you for your attention.

How to Measure Information: - from Action to Cognition - Matthias Rauterberg Department of...

Documents

Transcript of How to Measure Information: - from Action to Cognition - Matthias Rauterberg Department of...