New Metrics & Measurement

for information search dynamics in decision making

Presenter: Xiaolei ZhouAdvisor: Dr. Joe Johnson

Miami UniversityJDM Lab

Participants make a decision among several options (Rows), described by several attributes (Columns). For example, they must predict which movie has the highest receipts based on several binary features (above). The values of each cell in the information table are occluded until an eye fixation occurs on the cell

Stars Budget Rating Original

Movie A + - - +

Movie B - + + +

Movie C + - - -

Stars Budget Rating Original

Movie A +

Movie B

Movie CParticipants make a decision among several options (Rows), described by several attributes (Columns). For example, they must predict which movie has the highest receipts based on several binary features (above). The values of each cell in the information table are occluded until an eye fixation occurs on the cell

Stars Budget Rating Original

Movie A

Movie B -

Movie CParticipants make a decision among several options (Rows), described by several attributes (Columns). For example, they must predict which movie has the highest receipts based on several binary features (above). The values of each cell in the information table are occluded until an eye fixation occurs on the cell

Stars Budget Rating Original

Movie A

Movie B

Movie C +Participants make a decision among several options (Rows), described by several attributes (Columns). For example, they must predict which movie has the highest receipts based on several binary features (above). The values of each cell in the information table are occluded until an eye fixation occurs on the cell

Stars Budget Rating Original

Movie A -

Movie B

Movie CParticipants make a decision among several options (Rows), described by several attributes (Columns). For example, they must predict which movie has the highest receipts based on several binary features (above). The values of each cell in the information table are occluded until an eye fixation occurs on the cell

Stars Budget Rating Original

Movie A

Movie B

Movie C -Participants make a decision among several options (Rows), described by several attributes (Columns). For example, they must predict which movie has the highest receipts based on several binary features (above). The values of each cell in the information table are occluded until an eye fixation occurs on the cell

Stars Budget Rating Original

Movie A -

Movie B

Movie CParticipants make a decision among several options (Rows), described by several attributes (Columns). For example, they must predict which movie has the highest receipts based on several binary features (above). The values of each cell in the information table are occluded until an eye fixation occurs on the cell

Stars Budget Rating Original

Movie A

Movie B

Movie C -Participants make a decision among several options (Rows), described by several attributes (Columns). For example, they must predict which movie has the highest receipts based on several binary features (above). The values of each cell in the information table are occluded until an eye fixation occurs on the cell

Stars Budget Rating Original

Movie A +

Movie B

Movie CParticipants make a decision among several options (Rows), described by several attributes (Columns). For example, they must predict which movie has the highest receipts based on several binary features (above). The values of each cell in the information table are occluded until an eye fixation occurs on the cell

Stars Budget Rating Original

Movie A

Movie B

Movie C -Participants make a decision among several options (Rows), described by several attributes (Columns). For example, they must predict which movie has the highest receipts based on several binary features (above). The values of each cell in the information table are occluded until an eye fixation occurs on the cell

Strategy Used: Lexicographic (LEX)

Stars Budget Rating Original

Movie A ＋ － － ＋Movie B －Movie C ＋ － － －

✔

Strategy Used: Weighted Additive ( WADD)

Stars Budget Rating Original

Movie A ＋ － － ＋Movie B － ＋ ＋ ＋Movie C ＋ － － －

✔

Existing Measurements

Transition Matrices1,2 ( Frequency of specific transition types)3. Number of AcquisitionsTime per AcquisitionProportion of Information acquiredFinal Choice Made

1. Payne, J.W., Bettman, J.R., & Johnson, E.J. (1993). The adaptive decision maker. Cambridge University Press.2. Böckenholt, U., & Hynan, L. S. (1994). Caveats on a process-tracing measure and a remedy. Journal of Behavioral Decision Making, 7, 103–117.3. Ball, C. T. (1997). A comparison of single-step and multiple-step transition analyses of multiattribute decision strategies. Organizational Behavior and Human Decision Processes, 69, 195-204.

A A A A E E E E I I I I I E E B B B B J J J C C A A E C J J J C C C C C C J J D D K K D D L L D LA A B B B C C C C C D D D C C D D E E E F F F E E G G G G H H H H I I I J J I J K K L L K L H H

LEX:WADD:

A1 B1 C1 D1 E1 F1 G1 H1 I1 J1 K1 L1

A0 2B0 1C0 1 2D0 1 2E0 1 1 1F0

G0

H0

I0 1J0 2 1K0 1L0 1

A1 B1 C1 D1 E1 F1 G1 H1 I1 J1 K1 L1

A0 1B0 1C0 2D0 1 1E0 1 1F0 1G0 1H0 1I0 2J0 1 1K0 2L0 1 1

LEX: WADD:

Stars Budget Rating Original

Movie A A + B - C - D +

Movie B E - F + G + H +

Movie C I + J - K - L -

What we missed ?Experimental paradigms such as eye-tracking collect high-resolution process data revealing the information acquired en route to making decisions. However, the metrics deployed in analyzing these data have not kept pace, focusing instead on summary statistics. Analysis of search dynamics has been severely limited to crude measures such as relative direction (row- vs. column-wise transitions, or search “pattern” in the task below)1,2, or at best counting the frequency of very specific transition types3. We import techniques from other fields to remedy this shortcoming.

1. Payne, J.W., Bettman, J.R., & Johnson, E.J. (1993). The adaptive decision maker. Cambridge University Press.2. Böckenholt, U., & Hynan, L. S. (1994). Caveats on a process-tracing measure and a remedy. Journal of Behavioral Decision Making, 7, 103–117.3. Ball, C. T. (1997). A comparison of single-step and multiple-step transition analyses of multiattribute decision strategies. Organizational Behavior and Human Decision Processes, 69, 195-204.

Ordering Effects ?

“Describing”:

WADD

LEX

Lag Sequential Analysis

Lag Sequential Analysis

Order: Does search location at time t depend on location at time (t-1), (t-2),...?

Stationarity:Is the nature of the search process consistent over the course of a trial?

Homogeneity: Is the nature of the search process consistent across trials or participants?

18

Lag Sequential Analysis

Observed Frequency tableExpected Frequency tables (first order, second order, ...)G2(LRX2) - likelihood ratio statistic

19

IPF (Iterative Proportional Fitting)[Deming-Stephan algorithm]

IPF is a computer algorithm used to calculate expected frequency for each cell by using margins of every order’s two-way contingency table. (especially designed for calculate expected frequency tables of sparse matrices).

20

Hierarchical log-linear ModelExample: lag 2

Three models: 1. Saturated Model: [012] Three-way associations

2. Reduced Model 1: [01][12][02] homogeneous associations

3.Reduced Model 2: [01][12]

21

Hierarchical log-linear Model Testing

Adjusted Residuals

Lag=1: two models [01] vs. [0][1]

First, use IPF to compute the expected frequency for different models.

Second, test the significance of adjusted residuals between two models:

Lag=2: hierarchically test three models

[012] vs. [01][12][02], if no ordering effect, then [01][12][02] vs.[01][12]

First, use IPF to compute the expected frequency for three models.

second, test the significance of the adjusted residuals between models in order:

22

Results“Decreasing”: The higher time pressure is, subjects’ behaviors

become more random.

“green”- 1st order effect“dark blue” - complete random

effect

“Increasing”: The lower time pressure is, subjects’ behaviors

become more organized.

“light blue” - 1st order effect“red” - complete random effect

23

Conclusion

First order effect found~!!! Yay~

24

What we can do more?

More than “lags” or homogeneity?How do experimental conditions affect the search process?Does a proposed model describe actual search behavior?

How to differentiate strategies

“Comparing”:

WADD

LEX

“Distance”:

Sting Edit Distance

26

String Edit Distance[Needleman-Wunch Algorithm]

Dynamic programming methods5,6 are used stepwise to determine whether to insert, delete, or substitute codes at each position to minimize cost.

Operations:

5. Day, R. F. (2010). Examining the validity of the Needleman–Wunsch algorithm in identifying decision strategy with eye-movement data. Decision Support Systems, 49, 396–403.6. Cristino, F., Mathôt, S., Theeuwes, J., and Gilchrist, I. D. (2010). ScanMatch: A novel method for comparing fixation sequences. Behavior Research Methods, 42, 692-700.

27

Simple exampleTwo sequences:

Substitution Matrix:

Gap Penalty (d): -5

Score calculation: (theoretical range [0,1]) S(A,C)+S(G,G)+S(A,A)+(3*d) + S(G,G)+S(T,A)+S(T,C)+S(A,G)+S(C,T)

= -3 + 7 + 10 + (3*-5) + 7 + -4 + 0 + -1 + 0 = 1

A G A C T A G T T A C: | | | : : : :C G A - - - G A C G T

A G C TA 10 -1 -3 -4G -1 7 -5 -3C -3 -5 9 0T -4 -3 0 8

28

Implementation (ScanMatch Toolbox/Matlab)

A A A A E E E E I I I I I E E B B B B J J J C C A A E C J J J C C C C C C J J D D K K D D L L D LA A B B B C C C C C D D D C C D D E E E F F F E E G G G G H H H H I I I J J I J K K L L K L H H

LEX:WADD:

A A A A E E E E B B B B J J J J C C C C K K K K D D D D L L L L D D D DA A A B B B C C C D D D E E E F F F G G G H H H I I I J J J K K K L L L H H H

LEX_Theoretical :WADD_Theoretical:

LEX vs. WADDLEX_Theory

vs. WADD_theory

LEX vs. LEX_Theory

WADD vs. WADD_theory

LEX vs. WADD_Theory

WADD vs. LEX_theory

Stars Budget Rating Original

Movie A A + B - C - D +

Movie B E - F + G + H +

Movie C I + J - K - L -

29

Results 1

30

Results 2

31

Future Works

32

Ackownledgement