TOWARD EFFECTIVE ALGORITHM VISUALIZATION ARTIFACTS:

DESIGNING FOR PARTICIPATION AND COMMUNICATION

IN AN UNDERGRADUATE ALGORITHMS COURSE

by

CHRISTOPHER DAVID HUNDHAUSEN

A DISSERTATION

Presented to the Department of Computer and Information Scienceand the Graduate School of the University of Oregon

in partial fulfillment of the requirementsfor the degree of

Doctor of Philosophy

June 1999

ii

ABOUT THIS MANUSCRIPT

The following manuscript is a reformatted version of my original dissertation, which I defended inMay of 1999 in the Department of Computer and Information Science at the University of Oregon. Itis published as technical report number CIS-TR-99-07 (June, 1999) by the Department of Computerand Information Science, University of Oregon, Eugene. If you wish to have a hard copy of thetechnical report mailed to you, please contact Jan Saunders (jan@cs.uoregon.edu).

I am making my dissertation available on-line in hopes of better disseminating the work. I hope topublish pieces of this work as conference papers and journal articles; look for these to appear overthe next few years. In the meantime, if you would like to cite this work, please use the designation“unpublished doctoral dissertation,” and reference the technical report cited above.

I am still actively developing the prototype versions of SALSA and ALVIS presented in Chapter 7,and hope eventually to release them into the public domain. If you’re interested in a status report onSALSA and ALVIS, of if you have any comments, questions, or feedback on any of this work, pleasecontact me!

Chris HundhausenPostdoctoral ResearcherLaboratory for Interactive Learning TechnologiesDepartment of Information and Computer SciencesUniversity of Hawai'i at Manoa1680 East-West RoadHonolulu, HI 96822hundhaus@hawaii.eduhttp://lilt.ics.hawaii.edu/~hundhaus

Happy reading!

C.D.H.Honolulu, HINovember, 1999

iii

“Toward Effective Algorithm Visualization Artifacts: Designing for Participation and

Communication in an Undergraduate Algorithms Course,” a dissertation prepared by

Christopher D. Hundhausen in partial fulfillment of the requirements for the Doctor of

Philosophy Degree in the Department of Computer and Information Science. This

dissertation has been approved and accepted by:

Dr. Sarah Douglas, Chair of the Examining Committee

Date

Committee in charge: Dr. Sarah Douglas, ChairDr. Christopher WilsonDr. Michal YoungDr. Harry WolcottDr. John Stasko

Accepted by:

Dean of the Graduate School

iv

ABSTRACT

Algorithm visualization (AV) software graphically illustrates how computer algorithms work. Whilethe software initially had much promise as a pedagogical aid, research studies designed tosubstantiate its pedagogical benefits have yielded markedly mixed results. I argue that to harnessthe pedagogical promise of AV software, we need to rethink the theory of effectiveness that hasguided its design and pedagogical use. My starting point is an alternative theoretical foundationthat views learning not at the level of the individual, but rather at the level of the community ofpractice. On this alternative view, learning is seen in terms of participating more centrally in thepractices of the community.

To tailor this theoretical perspective to the particulars of the community of practice in whichalgorithms learning takes place, I conducted an ethnographic study of an undergraduate algorithmscourse in which AV software was used to facilitate students’ more central participation in thecommunity. Specifically, students were asked to use AV software to construct and present their ownvisualizations—two activities commonly performed only by community experts (algorithmsinstructors). The key finding of the study is that requiring students to use conventional AV softwarein this way actually impedes learning within the community, because it requires students to putinordinate amounts of time into community-irrelevant activities, and because it discourages studentsand instructors from engaging in meaningful conversations about algorithms. On the other hand,asking students to construct and present homemade visualizations made out of simple art suppliesappears to avoid these problems.

To explore this finding further, this dissertation pursues two parallel research directions: (1) acontrolled experiment that tests the hypothesis that, on a test of procedural understanding andrecall, students who construct their own, homemade visualizations will outperform students whointeract with a visualization constructed by an expert; and (2) a prototype AV system that supportsthe construction and presentation of unpolished, pen-and-paper visualizations. This researchprovides the beginnings of an alternative theory of effectiveness, which emphasizes the importanceof students' constructing and discussing unpolished, pen-and-paper visualizations as a means ofparticipating in a community of practice.

v

ACKNOWLEDGEMENTS

This epic journey would not have been possible without the help of several individuals. To beginwith, I owe a big hug and kiss to my parents, Patricia and David Hundhausen, who have alwaysbeen remarkably supportive of my interests, despite the fact that those interests have not alwaysbeen understandable or interesting to them.

Lee Schmidt, my high school mentor, and Tom Naps, my undergraduate mentor, planted the initialseeds for my venture into computer science and research. Lee sparked my interest in computerprogramming in Pascal, while Tom sparked my interest in doing long-term research, and thenpatiently taught me how to do it during the four summers I helped him to build software forLawrence University’s algorithm visualization laboratory (NSF ILI grant #88-51781).

I am grateful to Walter Tichy for inviting me to pursue the preliminary stages of this research as aFulbright Scholar in his Institute for Programming Structures and Data Organization at theUniversity of Karlsruhe, Germany. During my “Traumjahr” there (1993-94), Arne Frick, StefanHänssgen, Rick Chamberlin, Delel Chaabouni, Christoph Jerger, Aida Chouchane, Karim Diab,Frederic Million, Fabien Fersing, and Richard Kupprion sustained me with much neededencouragement and friendship as I learned to live “auf die echte deutsche Weise” (in the trueGerman way). When I arrived back in Eugene, Dr. Allen Malony advised and funded me during myfirst year in the Ph.D. program, while Gunnar Sacher helped me to maintain balance by beckoningme away from my computer for hiking and camping trips.

The ethnographic fieldwork and experiment presented in this dissertation would not have beenpossible without the enthusiastic participation of numerous CIS 315 students and their CIS 315instructor; unfortunately, I cannot mention them by name. I am grateful to the University ofOregon’s College of Arts and Sciences and Graduate School for two grants (the Mary ChambersBrockelbank Scholarship, and a Graduate Student Research Award) that paid the participants in theexperiment presented in Chapter 6. The prototype language and system presented in thisdissertation would not have been possible without the help of (a) Hank Bennett, who programmedthe back-end animation engine and a significant portion of the user interface, and (b) fellow HCI labmember Ted Kirkpatrick, who furnished ingenious advice at times when the Microsoft® FoundationClasses had me really stumped. Ted also kept me honest during the statistical analysis of myexperiment’s data.

I am grateful to Harry Wolcott for getting me excited about ethnography, and for offering sage adviceon the ethnographic research presented in this dissertation. I am grateful to John Stasko, one of thepioneers of algorithm visualization, for being such a wonderful remote collaborator and discussantthroughout my graduate studies, and for serving on my committee. Last, but not least, I am deeplyindebted to my inspirational advisor, Sarah Douglas, who challenged me while remainingencouraging all the way through, especially in the final months, when I needed her support the most.

My greatest thanks, however, goes lovingly to my best friend, life partner, and fiancée, LauraGirardeau, who has inspired me to grow in ways I could never have imagined. Despite my tendencyto act as though algorithm visualization is the most important thing in life, she was able to seethrough to my true self, remaining passionate, patient, giving, and forgiving. Namaste!

vi

DEDICATION

To my old self, who thought that he had to achieve to be loved

vii

TABLE OF CONTENTS

Chapter Page

1. INTRODUCTION........................................................................................................................1

1.1 Problems...........................................................................................................................31.2 Paths to Solutions.............................................................................................................41.3 The Thesis ........................................................................................................................51.4 Brief Summary of the Dissertation ..................................................................................6

1.4.1 EF Theory, Its Stronghold on Past Research, and Its Inadequacy ..........................61.4.2 Choosing an Alternative Theory...............................................................................61.4.3 Empirically Exploring the Sociocultural Constructivist Approach ..........................71.4.4 A Framework of Cause and Effect............................................................................81.4.5 Two Research Directions ..........................................................................................9

1.5 Organization of the Dissertation....................................................................................12

2. EPISTEMIC FIDELITY THEORY ...........................................................................................14

2.1 The Allegory of Musica ...................................................................................................152.2 Key Assumptions............................................................................................................16

2.2.1 The Knowledge Representation Assumption..........................................................162.2.2 The Knowledge Flow Assumption ..........................................................................162.2.3 The Graphical Medium Effectiveness Assumption ................................................162.2.4 Overriding Assumption: Epistemic Fidelity Is Crucial ..........................................172.2.5 Practical Implications.............................................................................................17

2.3 Variations on EF Theory ................................................................................................182.3.1 Weak EF Theory (Learner Involvement)................................................................182.3.2 Weak EF Theory (Individual Differences)..............................................................182.3.3 Weak EF Theory (Dual-Coding) .............................................................................18

2.4 EF Theory’s Influence on Past Research........................................................................182.4.1 Influence on AV Technology Design .......................................................................19

2.4.1.1 Support for Knowledge Flow ..........................................................................192.4.1.2 Support for Algorithmic Fidelity ....................................................................222.4.1.3 Support for Cultural Fidelity..........................................................................23

2.4.2 Influence on Effectiveness Evaluation ...................................................................242.4.3 Influence on Pedagogy............................................................................................25

2.4.3.1 AV in Lectures ................................................................................................252.4.3.2 AV in Labs ......................................................................................................262.4.3.3 AV for individual study ..................................................................................26

2.5 A Critique of EF Theory .................................................................................................272.6 Summary ........................................................................................................................31

3. CONSTRUCTIVISM: AN ALTERNATIVE THEORETICAL FOUNDATION ........................33

3.1 Cognitive Constructivism...............................................................................................343.1.1 Key Assumptions....................................................................................................34

3.1.1.1 Learners Actively Construct Their Own Knowledge......................................343.1.1.2 Stages of Intellectual Development: Concrete to Abstract ............................34

3.1.2 Implications for Pedagogy ......................................................................................35

viii

Page

3.1.3 Implications for Technology Design .......................................................................353.1.4 Implications for Effectiveness Evaluation..............................................................36

3.2 Discussion: Algorithms as a Social Practice..................................................................373.3 Sociocultural Constructivism .........................................................................................38

3.3.1 The Allegory of Musiphonia....................................................................................393.3.2 Key Assumptions....................................................................................................40

3.3.2.1 Knowledge and Learning within a Community .............................................403.3.2.2 Learning Is Changing Participation and Identity..........................................403.3.2.3 Access Is Crucial.............................................................................................41

3.3.3 Implications for Pedagogy ......................................................................................423.3.4 Implications for Technology Design .......................................................................433.3.5 Implications for Effectiveness Evaluation..............................................................44

3.4 Summary and Research Questions ................................................................................45

4. ETHNOGRAPHIC STUDIES ...................................................................................................46

4.1 Background.....................................................................................................................474.1.1 Informants: Students and the Instructor ..............................................................484.1.2 Algorithm Visualization Assignments and Supporting Technology.......................48

4.1.2.1 Study I Visualization Assignments ................................................................484.1.2.2 Study II Visualization Assignments ...............................................................49

4.1.3 Field techniques......................................................................................................504.2 Observations...................................................................................................................50

4.2.1 Animation Construction .........................................................................................524.2.1.1 Storyboard Construction Activities ................................................................534.2.1.2 Samba Construction Activities .......................................................................54

4.2.2 Animation Presentation and Discussion ................................................................564.2.2.1 Storyboard Presentations ...............................................................................564.2.2.2 Samba Animation Presentations....................................................................57

4.3 Discussion.......................................................................................................................584.3.1 Activity Relevance ..................................................................................................594.3.2 Designing for Conversations about Algorithms......................................................604.3.3 AV Construction and Presentation as Expert Forms of COSA Participation ........61

4.4 Summary ........................................................................................................................62

5. A FRAMEWORK OF CAUSE AND EFFECT...........................................................................63

5.1 AV Effectiveness Factors................................................................................................645.1.1 Representation Factors...........................................................................................64

5.1.1.1 Input Generality and Typset Fidelity.............................................................645.1.1.2 Story Content..................................................................................................65

5.1.2 Graphics Creation Technique.................................................................................655.1.3 Self-Construction....................................................................................................665.1.4 Instructor Communication .....................................................................................66

5.2 AV Effectiveness Measures ............................................................................................675.2.1 Activity Relevance ..................................................................................................675.2.2 Understanding and Recall......................................................................................675.2.3 Effective Communication .......................................................................................695.2.4 Community-Building ..............................................................................................69

5.3 Hypotheses .....................................................................................................................705.3.1 The Activity Relevance Hypothesis ........................................................................715.3.2 The Communication Effectiveness Hypothesis ......................................................71

ix

Page

5.3.3 The Understanding and Recall Hypothesis............................................................725.3.4 The Community-Building Hypothesis ....................................................................73

5.4 Summary and Research Directions................................................................................75

6. EXPERIMENTAL STUDIES ....................................................................................................76

6.1 A Series of Experiments .................................................................................................766.2 Experiment 1: Active Viewing versus Self-Construction ...............................................78

6.2.1 Design.....................................................................................................................786.2.2 Equalizing the Treatments.....................................................................................796.2.3 Participants ............................................................................................................806.2.4 Materials.................................................................................................................80

6.2.4.1 Active Viewing Treatment Group...................................................................816.2.4.2 Self Construction Treatment Group ...............................................................84

6.2.5 Tasks.......................................................................................................................856.2.6 Procedure................................................................................................................876.2.7 Scoring the Traces and Programs...........................................................................876.2.8 Results ....................................................................................................................876.2.9 Discussion...............................................................................................................90

6.3 Summary ........................................................................................................................92

7. PROTOTYPE LANGUAGE AND SYSTEM..............................................................................94

7.1 Empirical Foundations ...................................................................................................957.1.1 AV Content .............................................................................................................967.1.2 Process of AV Construction ....................................................................................977.1.2 Process of AV Execution and Presentation ............................................................977.1.3 Design Requirements .............................................................................................98

7.2 SALSA ............................................................................................................................987.2.1 Data Types..............................................................................................................997.2.2 Spatial Relations ..................................................................................................1007.2.3 Commands ............................................................................................................101

7.3 ALVIS ...........................................................................................................................1017.3.1 Overview...............................................................................................................1027.3.1 Storyboard Construction Interface .......................................................................103

7.3.1.1 Creating and Placing Storyboard Elements .................................................1047.3.1.2 Animating Cutouts .......................................................................................105

7.3.2 Storyboard Presentation Interface .......................................................................1067.4 Example of the Prototype in Use..................................................................................107

7.4.1 Creating the Storyboard Elements.......................................................................1087.4.2 Placing the Storyboard Elements .........................................................................1097.4.3 Programming the Spatial Logic............................................................................1117.4.4 Presenting the Storyboard....................................................................................113

7.5 Related Work................................................................................................................1157.6 Summary ......................................................................................................................116

8. CONCLUSION........................................................................................................................118

8.1 Research Contributions................................................................................................1208.2 Directions for Future Research ....................................................................................121

8.2.1 The Understanding and Recall Hypothesis..........................................................1218.2.2 The Activity Relevance Hypothesis ......................................................................122

x

Page

8.2.3 The Communication Effectiveness Hypothesis ....................................................1228.2.4 The Community-Building Hypothesis ..................................................................1238.2.5 The Prototype Language and System...................................................................127

8.3 A Vision for the Future: The “Algorithms Studio” ......................................................127

Appendix

A. ETHNOGRAPHIC STUDY I ..................................................................................................129

A.1 Background..................................................................................................................130A.1.1 Getting in the door ...............................................................................................131A.1.2 Disclaimers ..........................................................................................................131A.1.3 Using Samba for Algorithm Animation ...............................................................132A.1.4 Field Techniques..................................................................................................132

A.1.4.1 “Experiencing” Techniques .........................................................................133A.1.4.2 “Enquiring” Techniques ...............................................................................134A.1.4.3 “Examining” Techniques..............................................................................135

A.1.5 Informants: Students and the Instructor............................................................135A.2 Study I Animation Assignments..................................................................................136A.3 How Students Went about the Animation Assignments .............................................136

A.3.1 Time Spent...........................................................................................................138A.3.2 Animation-Building Activities .............................................................................138

A.3.2.1 Deciding on a Project....................................................................................138A.3.2.2 Implementing the Algorithm .......................................................................139A.3.2.3 Programming the Animation .......................................................................139

A.4 The Animation Presentation Sessions.........................................................................140A.4.1 A Typical Presentation Session............................................................................141A.4.2 Assignment Improvement Discussions................................................................143

A.4.2.1 The Value of Programming the Algorithm...................................................143A.4.2.2 The Difficulties of Graphics Layout .............................................................143A.4.2.3 Input Generality a Formidable Challenge...................................................144

A.5 The Instructional Side of the AV Assignments...........................................................145A.6 Assessing the Costs and Benefits of AV Assignments.................................................146

A.6.1 Students’ Perspective...........................................................................................146A.6.2 The Professor’s Perspective..................................................................................147

A.6.2.1 Grading is Difficult and Takes Too Long .....................................................148A.6.2.2 Students’ Time Not Well-Spent on the Animation Assignments.................148

A.7 Why are AV Assignments Effective?............................................................................150A.8 Discussion ....................................................................................................................151

B. ETHNOGRAPHIC STUDY II.................................................................................................153

B.1 Study II Animation Assignments ................................................................................154B.2 Research questions ......................................................................................................154B.3 Field techniques...........................................................................................................155B.4 Students’ Work on the Revised Animation Assignment..............................................156

B.4.1 Time Spent...........................................................................................................157B.4.2 Animation-Building Activities .............................................................................158B.4.3 Approaches to the Assignment.............................................................................158

B.5 The Presentation Sessions...........................................................................................159B.5.1 Storyboard presentations.....................................................................................159

B.5.1.1 Clarifying Questions ....................................................................................160

xi

Page

B.5.1.2 Design Discussions.......................................................................................161B.5.1.3 Tailoring Scenarios to Algorithms, and Algorithms to Scenarios................163

B.5.2 Final Animation Presentations............................................................................167B.6 Was the Revised Animation Assignment Format an Improvement?...........................169

B.6.1 Did a Single Animation Assignment Work out Better? .......................................169B.6.2 Was the Absence of an Input Generality Requirement an Improvement? ..........169B.6.3 Assessing the Storyboard Phase of the Assignment ............................................171

B.7 Comparing the Storyboard Phase to the Final Animation Phase ...............................172B.7.1 Differences in What Students Learned................................................................172B.7.2 Relevance of What Students Learned..................................................................173B.7.3 Are Both Phases Necessary?................................................................................174

B.8 Discussion ....................................................................................................................175

C. ETHNOGRAPHIC STUDY MATERIALS..............................................................................176

C.1 CIS 315 Syllabus for Study I .......................................................................................177C.2 CIS 315 Syllabus for Study II ......................................................................................180C.3 Study I Animation Assignments..................................................................................181

C.3.1 Assignment 1........................................................................................................181C.3.2 Assignment 2........................................................................................................181C.3.3 Assignment 3........................................................................................................182

C.4 Calls for Participation..................................................................................................184C.5 Informed Consent Agreements ....................................................................................185C.6 Study I E-Mail Surveys................................................................................................190C.7 Study II Animation Assignment ..................................................................................192C.8 Study II: Guidelines for Developing a Storyboard Presentation ................................194C.9 Study II Diary Form ....................................................................................................195

D. EXPERIMENT MATERIALS.................................................................................................196

D.1 Preliminary Interest Questionnaire ............................................................................197D.2 Background Information Questionnaire......................................................................198D.3 Informed Consent Agreement......................................................................................199D.4 Instructions: Active Viewing Group ...........................................................................201D.5 Instructions: Self-Construction Group .......................................................................207D.6 Tracing Task Instructions ...........................................................................................212D.7 Tracing Tasks ..............................................................................................................214D.8 Programming Task Instructions..................................................................................218D.9 Java QuickSelect Skeleton ..........................................................................................220D.10 C++ QuickSelect Skeleton .........................................................................................223D.11 Instructions for Finishing Up ....................................................................................226D.12 Exit Questionnaire: Active Viewing Group ..............................................................227D.13 Exit Questionnaire: Self-Construction Group ..........................................................228

E. EXPERIMENT DATA ............................................................................................................229

F. SALSA LANGUAGE SUMMARY...........................................................................................231

F.1 Commands....................................................................................................................232F.1.1 Storyboard Element Creation, Deletion, and Attribute Modification..................232F.1.2 Conditionals and Iteration ...................................................................................233F.1.3 Animation.............................................................................................................234

xii

Page

F.2 Attributes of Storyboard Elements ..............................................................................236F.2.1 SALSA Object Attributes .....................................................................................236F.2.2 Cutout attributes..................................................................................................237F.2.3 Position Attributes ...............................................................................................237F.2.4 Grid Attributes.....................................................................................................237

G. SAMPLE QUESTIONNAIRE FOR ASSESSING STUDENT ATTITUDES TOWARD ANALGORITHMS COURSE................................................................................................239

BIBLIOGRAPHY .........................................................................................................................243

xiii

LIST OF TABLES

Table Page

1. Summary of Ten AV Effectiveness Experiments ................................................................... 28

2. Independent Variables Manipulated in Ten AV Effectiveness Experiments ........................ 29

3. Ten AV Effectiveness Experiments vis-à-vis the Version of EF Theory For Which TheyWere Designed to Provide Evidence.................................................................................. 29

4. Four Storyboard Stories That Stood Out For Their Creativity and Innovation .................... 58

5. Independent and Dependent Variables in the Understanding and Recall Hypothesis ......... 77

6. Series of Planned Experiments .............................................................................................. 77

7. Design of Experiment 1.......................................................................................................... 78

8. The Table Participants Filled in for Each Trace.................................................................... 86

9. Mean Trace Scores and Times of the Two Treatments .......................................................... 88

10. Mean Program Scores and Times of the Two Treatments ..................................................... 89

11. Implications of Framework Hypotheses for AV Technology Design...................................... 95

12. The Ten Spatial Relations Supported By SALSA................................................................ 101

13. Brief Summary of SALSA Commands ................................................................................. 101

14. A 2 × 2 Design that Explores Self Construction and Instructor Communication................ 122

15. Ethnographic Field Techniques vis-à-vis Wolcott’s Taxonomy............................................ 134

16. Excerpts from the Content Log of a Typical Presentation................................................... 142

17. Field Techniques Used to Address Study II’s Research Questions...................................... 156

18. The Six Kinds of Questions That Were Offered and Elicited During StoryboardPresentations................................................................................................................... 161

19. Five Storyboard Stories that Stood Out for Their Creativity and Innovation..................... 164

20. Curt and Jeff’s Final Animation Scenario ........................................................................... 168

xiv

LIST OF FIGURES

Figure Page

1. Knuth’s Illustration of the Deque as a Railway Switching Network.......................................1

2. A snapshot of the “Grand Race” in Sorting Out Sorting ..........................................................2

3. A Snapshot from An Interactive Session with BALSA ............................................................3

4. Graphical Summary of the Dissertation ..................................................................................7

5. Pseudocode Description of Insertion Sort Algorithm............................................................. 11

6. An AV of the Insertion Sort Algorithm .................................................................................. 12

7. The POLKA User Interface for Controlling Animation Execution ........................................ 12

8. The Musibeest ........................................................................................................................ 15

9. A Schematic Diagram of Knowledge Flow According to EF Theory ...................................... 17

10. Knowledge Flow in Brown’s (1988) User Model for AV Software.......................................... 20

11. Prototypical Design of an AV Effectiveness Experiment....................................................... 25

12. Summary of the Experimental Support for each Version of EF Theory................................ 30

13. The Historical Evolution of AV Effectiveness Experiments and Empirical Studies.............. 31

14. Layers of AV technology-mediated participation................................................................... 43

15. Sample Samba Script Fragment and Animation Snapshot ................................................... 49

16. The Polka Control Panel ........................................................................................................ 49

17. Algorithm Themes Animated by Student Groups.................................................................. 51

18. Number of Geometric and Story-based Animations .............................................................. 51

19. Time Spent on Storyboards and Samba Animations ............................................................. 53

20. Students’ Storyboard Construction Activities........................................................................ 53

21. Students’ Samba Animation Construction Activities ............................................................ 54

22. A Taxonomy of AV Effectiveness Factors............................................................................... 64

23. A Taxonomy of AV Effectiveness Measures........................................................................... 67

24. Graphical Summary of the Framework ................................................................................. 71

25. The Activity Relevance Hypothesis........................................................................................ 71

xv

Page

26. The Communication Effectiveness Hypothesis...................................................................... 72

27. The Understanding and Recall Hypothesis ........................................................................... 72

28. The Community-Building Hypothesis.................................................................................... 74

29. The Understanding and Recall Hypothesis ........................................................................... 77

30. The “Console” Window ........................................................................................................... 81

31. The Polka Control Panel ........................................................................................................ 82

32. The “Forest Rangers” Window ............................................................................................... 83

33. The “History” Window............................................................................................................ 84

34. The “History Legend” Window............................................................................................... 84

35. Box Plot of Trace Scores of Each Condition ........................................................................... 89

36. Box Plot of Trace Times of Each Condition............................................................................ 89

37. Box Plot of Programming Task Scores................................................................................... 90

38. Box Plot of Programming Task Time..................................................................................... 90

39. Predefined Cutout Refpoints.................................................................................................. 99

40. Cutouts in a Sample SALSA Storyboard ............................................................................. 100

41. Snapshot of the ALVIS Environment .................................................................................. 103

42. The ALVIS Menus................................................................................................................ 103

43. The Create Cutout Dialog Box ............................................................................................. 104

44. The Cutout Graphics Editor................................................................................................. 105

45. The Script Move Dialog Box................................................................................................. 106

46. The Execution Control Toolbar ............................................................................................ 107

47. The Present/Edit Toolbar ..................................................................................................... 107

48. Pseudocode Description of the Bubble Sort Algorithm ........................................................ 108

49. The Football Bubble Sort Storyboard .................................................................................. 108

50. The ALVIS environment After Creating First Player ......................................................... 109

51. Creating a Grid with the Create Grid Dialog Box ............................................................... 110

52. The ALVIS Environment After All Cutouts Have Been Placed........................................... 110

53. SALSA Script for Football Bubble Sort Storyboard............................................................. 113

xvi

Page

54. The ALVIS Environment Before the Players Are Swapped ................................................ 114

55. “Attempt to Edit Previously Executed Line” Dialog Box ..................................................... 114

56. A Taxonomy of AV Construction Techniques....................................................................... 115

57. Example of the Semantic-Level Analysis Technique ........................................................... 126

58. A Schematic of the Area in Which the Fieldwork Was Conducted ...................................... 133

59. The Assignment 2 Animation Projects................................................................................. 137

60. The Assignment 3 Animation Projects................................................................................. 137

61. A Taxonomy of How Students Spent Their Time on Animation Assignments .................... 138

62. The Animation Projects Chosen By the 28 Student Groups ................................................ 157

63. Comparison of Time Students Spent on the Storyboard and Animation ImplementationComponents of the Assignment....................................................................................... 158

64. The Number of Student Groups Adopting Five Alternative Animation ImplementationStrategies......................................................................................................................... 159