Assessment of Cognitive Load in Multimedia Learning Using...

Assessment of Cognitive Load inMultimedia Learning

Using Dual-Task Methodology

Roland Brünken1, Susan Steinbacher1, Jan L. Plass2, and Detlev Leutner1

1 Erfurt University, Germany, and 2 New York University, USA

Abstract. In two pilot experiments, a new approach for the direct assessment of cognitive load during multimedia learningwas tested that uses dual-task methodology. Using this approach, we obtained the same pattern of cognitive load as predictedby cognitive load theory when applied to multimedia learning: The audiovisual presentation of text-based and picture-basedlearning materials induced less cognitive load than the visual-only presentation of the same material. The findings confirmthe utility of dual-task methodology as a promising approach for the assessment of cognitive load induced by complexmultimedia learning systems.

Key words: cognitive load, learning with multimedia, dual task

Current research on learning with multimediaincreasingly pays attention to insights on knowledgeacquisition provided by cognitive theory (Brünken &Leutner, 2000; Mayer, 2001). Of central interest inthis line of research is how different presentation for-mats used in multimedia learning systems facilitateknowledge acquisition to different degrees, andwhich theoretical explanations can account for theseeffects. While in the past few years empirical studieswere primarily concerned with the extent to whichdifferent presentation modes of information, i.e., thepictorial or textual presentation of learning materials,lead to different degrees of knowledge acquisition(e.g., Brünken, Steinbacher, Schnotz, & Leutner,2001; Mayer, 1997; 2001; Schnotz & Bannert, 1999),a few groups of researchers have begun to studywhether and to what degree the sensory modality ofinformation, i.e., auditory and/or visual presentationof verbal learning materials, influences knowledge

The CBT programs, “Cardiovascular system” and “Awalk through Firenze” were developed by Dr. RolandBrünken and Dipl.-Ing. Tobias Gall; the WinRT programwas developed by Dr. Roland Brünken and Stefan Leifheit.

All product names used in this paper are trademarksof their respective manufacturers.

DOI: 10.1027//1618-3169.49.2.109” 2002 Hogrefe & Huber Publishers Experimental Psychology 2002; Vol. 49(2): 109Ð119

acquisition (Brünken & Leutner; 2001; 2002;Mayer & Moreno, 1998; Moreno & Mayer 2000;Mousavi, Low, & Sweller, 1995). These studies haveconsistently found two effects that explain howknowledge acquisition is facilitated by multimediamaterials. The multimedia dual-coding effect, focus-ing on the presentation mode of information, de-scribes how learning is facilitated through the simul-taneous presentation of picture-based informationand closely related text-based learning materials(Mayer, 1997). The modality effect, focusing on thesensory modality of information, describes howknowledge acquisition is facilitated by an audiovis-ual presentation format, i.e., by simultaneously usingthe auditory and the visual sensory modalities in-stead of only the visual modality (Mayer, 2001).

In order to theoretically explain these effects,Mayer and associates (Mayer, 2001; Mayer & Mor-eno, 1998) have proposed a model of knowledgeacquisition with multimedia learning systems that in-tegrates various aspects of cognitive theory, such asworking memory models, dual coding theory, cogni-tive load theory, and generative theory of learning.The fundamental assumption of multimedia dual-processing theory (Mayer & Moreno, 1998) is theexistence of two separate processing systems, one for

110 Roland Brünken, Susan Steinbacher, Jan L. Plass, & Detlev Leutner

visual information and one for auditory informationin working memory (see Miyake & Shah, 1999; Bad-deley, 1986; Baddeley & Logie, 1999). Each of thesesystems has a limited processing capacity. Mayermakes the additional assumption that different pro-cesses of mental representation occur during the pro-cessing of pictorial and of textual information (seePaivio, 1986): Learning occurs when the learner per-ceives and selects relevant information, organizes itinto a coherent mental representation, and builds ref-erential connections between individual pieces of in-formation and integrates it with prior knowledge(Mayer, 1997; Schnotz & Bannert, 1999; Schnotz,Böckheler, & Grzondziel, 1999; Wittrock, 1990). Thisprocess takes place in working memory (Mayer &Moreno, 1998). Knowledge acquisition is especiallyenhanced when textual and related pictorial informa-tion is presented concurrently (contiguity effect:Mayer, 1997). In line with cognitive load theory(Chandler & Sweller, 1991; Sweller, 1988), Mayer andhis colleagues also assume that performance duringknowledge acquisition is dependent on the cognitiveresources available for information processing.

The simultaneous visual presentation of text- andpicture-based learning materials (i.e., on-screen textand pictures) leads to an information processing thatis, at least initially, carried out in the visual subsystemof working memory. Thus, the cognitive resourcesavailable in visual working memory have to be dividedbetween textual and pictorial information, whereas theresources of the phonological working memory sub-system remain largely unused. By contrast, in the caseof an audiovisual presentation of text- and picture-based learning materials (i.e., narrated text and on-screen pictures), the auditory information (the narra-tion) is processed in the phonological subsystem,while the visual information (the picture) is processedin the visual subsystem. In this design, the combinedresources of both subsystems can be used to processthe information, so that the audiovisual presentationof information allows for the utilization of more cog-nitive resources for information processing than thevisual-only presentation of the same learning materi-als. Hence, knowledge acquisition is comparativelyhigher with audiovisual information, as empiricalstudies have confirmed (modality effect: Brünken &Leutner, 2001; 2002; Mayer & Moreno, 1998; Mayer,Moreno, Boire, & Vagge, 1999; Moreno & Mayer,1999; Mousavi et al. 1995).

Measuring cognitive loadAt the heart of the model proposed by Mayer (Mayer,2001; Mayer & Moreno, 1998) is the assumption thatthe processing of verbal and pictorial information pre-sented in audiovisual modalities versus visual-only

Experimental Psychology 2002; Vol. 49(2): 109Ð119 ” 2002 Hogrefe & Huber Publishers

modality demands different amounts of cognitive re-sources. This in turn leads to different amounts of ac-quired knowledge given an otherwise identicalcontent of the learning materials. Based on this differ-ence in learning outcome, experimental studies onmultimedia learning that are based on Mayer’s modeluse cognitive load to explain the differences in learn-ing efficacy of different types of multimedia learningmaterials. Unlike other basic psychological studies onmemory, these studies do not directly assess actualworking memory load in the form of a manipulationcheck. The indirect measures used in some of thesestudies, using e.g., posttreatment questionnaires(Paas & van Merrienboer, 1993) or measures of time-on-task (e.g., Brünken & Leutner, 2002), do not pro-vide conclusive evidence for the actual cognitive loadexperienced by the learners (Kirschner, 2001).

The goal of the present studies was therefore todevelop and evaluate a measurement tool for thecomputer-based direct assessment of cognitive loadduring the learning process in multimedia learningsystems. To this end, the dual-task methodology,which is well known in experimental psychology(Heuer, 1996; Verwey & Veltman, 1996; Wickens,1984), was applied and experimentally tested for thefirst time in the context of multimedia learning.

The dual-task methodology is based on the as-sumption that the processing capacity of workingmemory is limited but can be flexibly allocated (seeBaddeley, 1986; Miyake & Shah, 1999). If two taskshave to be processed at the same time (dual-task con-dition), and both require the same cognitive re-sources, then these resources have to be split betweenthe two tasks. This means that fewer resources areavailable for processing each individual task thanwould be available for processing a single task (sin-gle-task condition). If the processing of a task de-pends on available cognitive resources, then perfor-mance in processing a secondary task will be re-duced in relation to the amount of cognitive re-sources required by a primary task. Notably,Baddeley and colleagues (Baddeley & Hitch, 1974;Baddeley, 1986; Baddeley & Logie, 1999) have usedmany variations of the paradigm in testing their pro-posed model of working memory.

Two variations of the dual-task methodology arecommonly used in the analysis of cognitive load inexperimental research. In the first method, the dual-task methodology is used in order to induce cognitiveload. It is assumed that the processing of a secondarytask demands cognitive resources that are conse-quently no longer available for processing the pri-mary task. The extent to which the addition of a se-condary task causes a reduction in the performanceof the primary task (compared with the single-taskcondition) is measured. Typical examples for suchsecondary tasks are articulatory suppression, tap-

111Dual-Task and Multimedia Learning

ping, tracking, or verbal generation of random num-bers (Baddeley, 1986; Logie, Baddeley, Mane,Donchin, & Sheptak, 1989; Klauer, Oberauer, Ross-nagel, & Musch, 1996).

In the second method, the dual-task methodologyis used to assess cognitive load. Here it is assumed thatdifferent variants of primary tasks require differentamounts of cognitive resources. Thus, performance inprocessing a simultaneously presented secondary taskvaries according to the cognitive load induced by theprimary task. Typical examples of secondary tasksused are choice reaction tasks, which comprise simpledecision-making. For instance, simple equations arepresented (e.g., 4 + 3 = 7; 5 + 2 = 6), and the partici-pant is asked to decide as quickly as possible whetherthe proposed solutions are correct or incorrect. In ad-dition to numerical tasks, verbal and pictorial materi-als are also employed (Baddeley & Logie, 1999; Logieet al., 1989). The dependent variables in these tasksare reaction time (interval between task presentationand reaction) and error rate pertaining to the process-ing of the secondary task.

Measuring Cognitive Load inMultimedia Learning Using Dual-Task MethodologyThe use of secondary tasks for assessing the cogni-tive load of a primary task seems to be a particularlypromising approach for measuring cognitive loadduring the processing of learning materials presentedby multimedia systems. If audiovisual and visual-only presentations of the same verbal and pictoriallearning materials differ in the amount of cognitiveresources required for information processing in thevisual subsystem of working memory, then partici-pants working with an audiovisual or a visual-onlymultimedia system should also exhibit differences inperformance when processing a simultaneously pre-sented secondary visual task. Participants learningwith audiovisual information in a dual-task conditionshould have comparatively more resources at theirdisposal for processing a visual secondary task thanparticipants learning with visual-only information.Thus, participants in an audiovisual multimedialearning condition should outperform participants ina visual-only multimedia learning condition in theprocessing of the secondary task.

In order to test this hypothesis, two pilot experi-ments with different multimedia learning systemswere conducted with 10 participants each. A secondgoal of the experiments was to determine whetherand to what extent the dual-task methodology is ap-plicable and useful within the context of research onlearning with complex multimedia systems.

” 2002 Hogrefe & Huber Publishers Experimental Psychology 2002; Vol. 49(2): 109Ð119

Experiment 1Method

Participants and Design

Study participants were 10 female students enrolledin the education program at Erfurt University (Ger-many). The average age of the participants was 22.8years (SD = 2.5). Due to expected large individualdifferences in reaction times, an all-within-subjectdesign with repeated measures was applied. Depen-dent variables were reaction times for the secondarytask and knowledge acquisition for the primary task(learning with a complex multimedia system). A pre-test was performed to allow for the assessment ofprior knowledge effects for the primary task.

Primary task. The learning system was a multi-media computer-based training (CBT) program onhow the human cardiovascular system works. Theprogram comprised of 22 screen pages; each pagecontained verbal and closely related pictorial de-scriptions of the relevant organs (e.g., heart, lungs,and veins), the physiological processes of gas andsubstance transportation, as well as the electrochemi-cal processes of heart stimulation and muscle con-traction. The program was developed in Toolbook(Asymetrix, 1997) for the Windows 95 operatingsystem (Microsoft, 1996). It was available in twovariants that differed in the presentation format ofinformation (audiovisual vs. visual-only). In the vi-sual-only variant, the verbal information was pre-sented visually (i.e., as on-screen text) simulta-neously with the picture. In the audiovisual variant,the same verbal information was presented in audi-tory format (i.e., narrated by a female speaker),while the on-screen text was not visible.

The program has been successfully used in previ-ous studies (Brünken & Leutner, 2001; 2002). Inthese studies, we were able to show that a modalityeffect for knowledge acquisition did indeed occur:Participants working with the audiovisual variant ofthe learning system acquired more knowledge thanparticipants who were working with the visual-onlyvariant of the system.

In order to implement the all-within-subject de-sign of the present study, the CBT program was rede-signed in such a way that, for each participant, thevisual-only and the audiovisual presentation formatof each of the 22 screen pages was alternated pageby page, and the sequence was altered between parti-cipants (see Table 1). The software was programmedto automatically advance to the next screen page, andthe presentation time of each page was identical forboth variants, corresponding to the duration of theaudio narration of the audiovisual variant (see alsoBrünken & Leutner, 2001).


Table 1. Research Design Used in Experiments 1 and 2

Presentation format of the multimedia learning system pages (primary task)

Sequence N Pretest (ST) 1 2 3 . . . n-1 n (ST) Posttest

1 2 X X V AV V . . . V AV X2 3 X X AV V AV . . . AV V X3 3 X V AV V . . . V AV X X4 2 X AV V AV . . . AV V X X

Note. V = visual-only presentation of learning materials as primary task; AV = audiovisual presentation of learning materialsas primary task; ST = single-task condition (only secondary task; N = number of participants); “X” indicates that therespective test or task was completed by this group.

Two criterion referenced paper-and-pencil tests(pretest and posttest; Klauer, 1987) were used to as-sess knowledge acquisition concerning the informa-tion presented in the learning system. Each test con-sisted of 20 multiple-choice items with four answeralternatives, where each of the alternatives could beright or wrong. Right answers were presented in ran-dom order. The content of the test items referred tothe organs as well as to the physiological and electro-chemical processes described in the multimedialearning materials. The tests had previously beensuccessfully implemented in assessing knowledgeacquisition for these materials (Brünken & Leutner,2001; 2002). However, in the present experiment theassessment of knowledge acquisition served as acontrol measure. In the dual task condition, theknowledge acquisition test should show whether theparticipants worked, according to the instruction theyhad received, on the primary task (the learning sys-tem) and whether they indeed acquired knowledgefrom this task. Since previous studies had shown thatthe modality effect could be replicated with the mul-timedia materials used for this study (Brünken &Leutner, 2001), we chose an within-subjects designto test the usefulness of the dual-task methodologyfor the measurement of cognitive load in multimedialearning, in which each participant was presentedwith both the visual-only and the audiovisual pri-mary task. Thus, in this design, the knowledgeacquisition test did not serve as a test for the replica-tion of the modality effect.

Secondary Task. A simple, continuous visual ob-servation task was selected as the secondary task: Theblack-colored letter “A” was displayed in a separatewindow on the screen (see Figure 1). After a randomperiod of 5Ð10 seconds, the color of the letter changedto red. The participants were asked to press the spacebar of the keyboard as soon as possible after the letterhad changed to red. Once the key was pressed, the let-ter changed back to black, and the next countdownstarted. The software automatically recorded the time


lapse between the appearance of the letter in red andthe pressing of the space bar.

However, due to the multitasking capabilities of theWindows operating system, the measurement of reac-tion times in milliseconds is very problematic. Cur-rently there are no commercially available Windows-based software tools for such a reaction time measure-ment that could be used in the framework of our dual-task study. PC-based measurement tools such as ERTS(Experimental Run Time System) (ERTSLab, 1999)operate on MS-DOS and cannot run simultaneouslywith Windows-based multimedia programs. There-fore, to be able to implement the dual-task condition,we developed the Windows-based software toolWinRT, which is able to measure reaction times inWindows-based multimedia software.

Procedure

Individual test sessions were conducted for each par-ticipant in one of the computer laboratories at ErfurtUniversity. The apparatus consisted of an 800 MHzPentium III PC with a 17 inch monitor that was setto a screen resolution of 1152 ¥ 864 pixels. The ex-perimenter was present during the entire investiga-tion. Before starting the experiment, the experi-menter instructed the participant in the operation ofthe software programs.

The participant first completed the paper-and-pencil pretest, which assessed prior domain-specificknowledge. Then, the study phase with the computerprograms began, which involved the single process-ing of the secondary task (single-task condition) aswell as the concurrent processing of the secondarytask and of the materials in the learning system asprimary task (dual-task condition). The single-taskcondition, which was completed either before or af-ter the dual-task condition, was included to obtainan individual baseline measure for each participant’sreaction time. Following the last measure, the paper-


and-pencil posttest was administered to assessknowledge acquisition.

The entire procedure, including the two paper-and-pencil tests, lasted approximately 60 minutes. The du-ration of the work with the computer programs took 3minutes for the single task condition and 14 minutes15 seconds for the dual-task condition.

The computer screen presented to the participantscontained two frames: a primary task frame and asecondary task frame. The primary task frame ap-peared centered against a black background. The se-condary task frame was placed in the center, verti-cally above the primary task frame, (see Figure 1).The size of the primary task frame was 22 ¥ 16 cm(width ¥ height), and the size of the secondary taskframe was 3 ¥ 3.5 cm. The letter displayed in thesecondary task frame was 1.2 ¥ 1.3 cm in size. Thus,the single-task and the dual-task condition repre-sented identical technical implementations, as bothprograms (Toolbook-application and WinRT) wereactive at the same time.

In the single-task condition, the secondary-taskframe was presented above the blank page of the pri-mary task frame (black background; empty beige-col-ored frame). In the dual-task condition, the learningprogram was presented in the primary task frame. Thepages of the learning program were advanced auto-matically; no mouse or keyboard input was required towork with the multimedia learning system.

Results

Scoring and analysis

The points scored in the pretest and in the posttest, aswell as reaction times in milliseconds, were recorded.A paired sample t-test was calculated to assess knowl-edge acquisition. For the analysis of reaction times,the data from the secondary task were first synchro-nized with the learning program on the basis of time-stamped log file data. Then the measures werematched to their corresponding test condition: singletask, dual-task with visual-only primary task, or dual-task with audiovisual primary task. Finally, individualoutliers were eliminated. According to the statisticaldefinition, a reaction time measure was defined as anoutlier when its absolute value was more than threestandard deviations above the individual mean in thecorresponding test condition.

Because the secondary task was presented withina randomized time interval, and because of indivi-dual differences in reaction times and differentamounts of outliers for each individual, we obtaineddifferent numbers of reaction time measures for eachparticipant and test condition (between 23 and 47measures per person and condition). Therefore, thereaction time measures for each of the three test con-


ditions were averaged for each participant (see Table2), and these three mean reaction times were usedfor all further analyses. Following the conventionalapproach for the evaluation of reaction time mea-sures used in experimental psychology (see Donk &Sanders, 1989), we first computed a one-factor re-peated measures analysis of variance (RM-ANOVA)with repeated measures on the three test conditions(single task, dual-task: visual-only, and dual-task: au-diovisual). Subsequently, post hoc paired samples t-tests with α-adjustment (total α = 0.05) were calcu-lated for two test conditions at a time.

Primary-task performance

For knowledge acquisition, the mean number of pointswas 6.9 (SD = 5.9) at pretest and 26.8 (SD = 6.7) atposttest. The difference is highly statistically signifi-cant (t9 = 6.97; p � 0.001), providing empirical evi-dence that the participants completed the primary taskaccording to the instructions and indeed acquiredknowledge from the multimedia learning system.

Secondary-task performance

Table 2 displays the individual reaction times inmilliseconds (M, SD, number of measures) in thethree test conditions for all 10 participants.

As predicted, the reaction times were lowest forthe single-task condition (M = 355.10, SD = 54.13).For the two dual-task conditions, the reaction timeswere lower for the audiovisual primary-task presenta-tion (M = 631.60, SD = 183.09) than for the visual-only primary-task presentation (M = 803.60, SD =234.92). The RM-ANOVA confirmed these results,as did the post hoc t-tests. The RM-ANOVA revealeda significant main effect of the factor “test condi-tion” (F(2, 9) = 41.191, p � 0.001, MSE =12429.54). The post hoc t-tests showed that the dif-ferences in the reaction times between the single-taskcondition and the visual-only dual-task condition(t9 = 7.318, p � 0.001, d = 2.63), the single-taskcondition and the audiovisual dual-task condition(t9 = 6.005, p � 0.001, d = 2.05) as well as thedifference between the visual-only and the audiovi-sual dual-task conditions (t9 = 4.279, p = 0.002, d =0.82) were statistically significant.

These results are in full agreement with our pre-dictions: The reaction times in secondary task perfor-mance increase when the primary task includes vi-sual-only learning materials compared to a primarytask with audiovisual materials of the same content.

To verify the results on the basis of the individualmeasures of reaction times, the analysis was repeatedseparately for each of the 10 participants. The sameeffect patterns were found as in the analysis basedon the computed means for reaction times.


Figure 1. Screenshot from the materials used in Experiment 1 under the dual-task condition. In the middle isthe page of the multimedia learning system (visual-only variant); the frame of the secondary task is locatedabove center.

Experiment 2

To replicate the results found in Experiment 1, andto test for generalizability across different learningsystems, a second experiment was conducted. Thesecond study used the same design and the same se-condary task as in Experiment 1, but included a dif-ferent multimedia system as the primary task.


MethodParticipants and design

Experiment 2 was conducted with 10 female studentsenrolled in the Psychology program at Erfurt Univer-sity, Germany. The mean age of the participants was21.2 years (SD = 1.93). Again, an all-within-subjectdesign with repeated measures was used (single task;dual-task: visual-only; dual task: audiovisual). Thedependent variables were knowledge acquisition inthe primary task (learning with the multimedia sys-tem) and reaction time in the secondary task.

Primary task. The multimedia learning systemused in the second experiment, which was also used


Table 2. Individual Measures in Experiment 1 for the Secondary Task (reaction time in milliseconds; N, M, SD)Broken Down by the Three Test Conditions.

Subject No. Single Task Dual Task Dual Task(audio-visual) (visual only)

N M SD N M SD N M SD

1 23 320 70 37 456 111 24 564 1192 24 442 97 39 901 714 26 1194 9203 23 392 129 26 951 982 42 992 6964 23 268 47 43 441 309 44 564 3815 25 410 202 34 585 306 42 678 3916 23 354 128 43 508 163 40 909 6397 24 335 52 38 612 358 47 736 4198 23 332 68 37 621 264 31 709 3429 23 397 135 40 768 476 41 1119 1039

10 24 301 57 45 476 139 38 571 229

in prior studies (Brünken et al., 2001), contained ver-bal and pictorial information about the historic cityof Firenze (Florence), Italy. The system was pro-grammed in Toolbook Instructor II (Asymetrix,1997) and comprised of 16 screen pages. Each pageincluded verbal information (on-screen text or audionarration) and closely related pictorial information(graphics and pictures) about the location and theimportance of a historic building, place, or work ofart (e.g., a cathedral, or Michelangelo’s sculpture Da-vid). Similar to a tourist guide, each page of the sys-tem contained at least one picture of a specific ob-ject, a verbal description with some historical factsconcerning this object, and a route map of the cityof Firenze that showed where the object was located.Verbal and pictorial/graphical information were pre-sented separately in two frames.

As in Experiment 1, the presentation format ofeach screen page was either visual-only (on-screentext and graphic/picture) or audiovisual (audio narra-tion and graphic/picture, with the text frame hidden),while the content of both variants was identical. Thepresentation format alternated from screen page toscreen page, and the sequence of alternation itselfwas alternated among different participants (see Ta-ble 1). The screen pages were automatically ad-vanced by the system, and the presentation time ofeach page was the same for both formats, determinedby the duration of the corresponding narration foreach screen page in the audiovisual treatment.

Knowledge acquisition of the content of thelearning system was assessed using two criterion-ref-erenced paper-and-pencil tests, which were con-structed as psychometrically parallel tests and whichserved as pretest and posttest. Both tests have suc-cessfully been used in prior studies (Brünken et al.,2001). Each test contained 20 multiple-choice itemswith four answer alternatives. The test items elicited


responses regarding historical facts and the locationof a specific object. Again, the knowledge acquisi-tion test served as a control measure to allow verifi-cation that the participants had indeed worked withthe learning system and had acquired knowledgefrom the information presented.

Secondary task. The secondary task used in Ex-periment 2 was the same visual observation task asin Experiment 1 (see Figure 2).

Procedure

The study was conducted with individual test ses-sions at one of the computer labs at Erfurt University.The technical details of the computer systems,screen-sizes, and the experimental procedures wereidentical to Experiment 1. The mean duration of theentire experiment was approximately 50 minutes; thesingle-task condition (secondary task alone) lasted 3minutes, and the dual-task condition 11 minutes 40seconds.

Results

Scoring and Analysis

As in Experiment 1, points scored in the knowledgeacquisition pretest and posttest were recorded andcompared using a paired samples t-test. The analysisof the reaction time measures was conducted Ð aftercompleting the same preprocessing as in Experiment1 Ð firstly on individual data, where the mean reac-tion times for each participant and test conditionwere calculated. As in Experiment 1, we observed adifferent number of measurements for each personand test condition (between 19 and 35; see Table 3),caused by the random generation of secondary tasks,


Figure 2. Screenshot from the materials used in Experiment 2 under the dual-task condition. In the middle isthe page of the learning system (visual-only variant); the frame of the secondary task is located above center.

individual differences in reaction times, and differ-ences in individual outliers. In a second step, the in-dividual means were again analyzed by computing aRM-ANOVA and the appropriate post hoc tests.

Primary-Task Performance

Concerning the knowledge acquired from the learn-ing system, the mean number of points was 3.2(SD = 1.69) at pretest and 10.8 (SD = 4.24) at post-test. The difference was statistically significant (t9 =6.626, p � .001). Again, the results indicate that theparticipants worked on the primary task accordingto the instructions they had received and acquiredknowledge from the multimedia learning system.


Secondary-Task Performance

Table 3 shows the individual secondary-task perfor-mance (mean, SD, number of measures) for each par-ticipant for each of the three experimental condi-tions. As can be seen, the second experiment showsthe same effect pattern as found in Experiment 1.The mean reaction times were lowest for the single-task condition (M = 466.5; SD = 74.67), followed bythe audiovisual dual-task condition (M = 789.9; SD =349.04). As in the first experiment, the reactiontimes were highest in the visual-only dual-task con-dition (M = 1063.6; SD = 382.2). The results wereconfirmed by the RM-ANOVA, which showed a sta-tistically significant main effect of the test condition(F(2, 9) = 20.356, p � .001, MSE = 43,887.32). To


Table 3. Individual Measures in Experiment 2 for the Secondary Task (reaction time in milliseconds; N, M, SD)Broken Down by the Three Test Conditions.

Subject No. Single Task Dual Task Dual Task(audio-visual) (visual only)

N M SD N M SD N M SD

1 24 459 157 31 683 308 24 1027 10872 24 537 76 33 906 250 19 1011 4363 22 602 341 33 468 116 28 627 4284 22 373 323 30 758 609 26 921 5325 22 414 74 24 595 234 32 947 6456 23 533 270 19 1600 1722 24 1955 16977 23 394 79 27 595 201 26 1063 6998 22 506 95 30 1144 620 22 1458 12659 22 437 156 35 483 149 25 740 310

10 22 410 79 30 667 465 24 887 1022

analyze these results in more detail, post-hoc paired-sample t-tests with α-adjustment (total α = .05) werecomputed. The t-tests showed that the observed meandifferences were statistically significant for the com-parison of the single-task and audiovisual dual-taskconditions (t9 = 3.078, p = .013, d = 1.53), the com-parison of the single-task and the visual-only dual-task conditions (t9 = 5.042, p = .001, d = 2.61), andthe comparison of the audiovisual and the visual-only dual-task conditions (t9 = 7.687, p � .001, d =0.75). Again, as can be seen in Table 3, this effectpattern was also present for the reaction time valuesof each individual participant.

In summary, we found that the results of our se-cond experiment, as well as the results of the firstexperiment, are in line with the underlying cognitive-load hypothesis and show a significant increase inreaction times in the secondary task when subjectswork with a visual-only multimedia learning systemas the primary task, compared with when they workwith an audiovisual multimedia system of the samecontent as the primary task.

Discussion

The goal of the experimental pilot studies presentedin this paper was the development and evaluation ofa new approach for the direct assessment of cognitiveload during learning with complex multimedia learn-ing systems. The study was motivated by a series ofrecently published studies (Brünken & Leutner,2001; 2002; Mayer & Moreno, 1998, Moreno &Mayer, 1999; Mousavi et al. 1995), which showedthat the audiovisual presentation of closely relatedpicture-based and text-based learning materials leadsto enhanced knowledge acquisition when compared


with the visual-only presentation of the same learn-ing material. All of these studies explain this effectwith reference to cognitive-load theory (Chandler &Sweller, 1991): It is assumed that the visual-only pre-sentation of learning materials results in a compara-tively higher visual cognitive load as compared withaudiovisual presentation, and this higher cognitiveload causes the observed lower amount of knowledgeacquisition.

Having applied the dual-task methodology tomeasure cognitive load during multimedia learning,the present studies show that these hitherto only as-sumed differences in cognitive load of the visual sub-system of working memory can be assessed directly,based on performance in processing a simulta-neously presented secondary task.

The present findings are in accordance with thetheoretical predictions. On the one hand, both experi-ments revealed a statistically significant difference inperforming the secondary task alone and in perform-ing the same task in a dual-task condition. The pro-cessing of the learning materials (primary task) andthe processing of the secondary task are shown touse the same cognitive resources, so that the second-ary task is a sensitive measure of cognitive load dur-ing the processing of the primary task.

Furthermore, and of central importance for ourstudies, we found, again in both experiments, that theprocessing of the secondary task during the visual-only presentation of verbal and pictorial learning ma-terials was significantly slower than during the au-diovisual presentation of the same materials. As pre-dicted by cognitive load theory, the simultaneous vi-sual presentation of texts and related pictures inducesa comparatively higher cognitive load than the audio-visual presentation of the same learning materials.

In addition, comparing the results of the two ex-periments, we found the reaction times in Experi-


ment 2 under both dual-task conditions to be longerthan in Experiment 1. This could indicate that thelearning scenario used in Experiment 2 was moredifficult than that used in Experiment 1. If a task ismore difficult in general, then it needs more cogni-tive resources (Kirschner, 2001) independent of thesensory modality used, which could be the reason forthe overall increase in reaction times in Experiment2. Though we do not have empirical information onthis topic from the present experiment, prior studieswith the learning system used in Experiment 2(Brünken et al., 2001) indicated that these materialsseem to be relatively difficult for learners.

Our findings have theoretical as well as practicalimplications. On the theoretical side, we found strongempirical evidence for cognitive load effects on learn-ing outcomes by directly measuring the performancein a simultaneously processed secondary task. This di-rect measurement is based on the dual-task method,which is well known in the field of experimentalpsychology but has not been previously applied instudies on learning and instruction with complex mul-timedia systems. On the practical side, the dual-taskapproach developed in the present studies appears tobe a suitable approach for measuring cognitive load incontexts of practical application, even beyond learn-ing and instruction. For example, to evaluate variantsof interface design drafts for a multimedia system thatdiffer with respect to the format in which informationis presented, it is conceivable that this approach couldbe used to determine the amount of cognitive load in-duced in the user by each design. On the basis of suchan ergonomic software evaluation one could deter-mine, for example, which interface design variantminimizes cognitive load.

These pilot studies of course leave many questionsopen. For instance, there are still some technical prob-lems associated with the measurement of reactiontimes under the Windows operating system that haveto be satisfactorily resolved in the future, and no im-plementation opportunities for other operating sys-tems, such as Mac OS or UNIX, exist. However, in thepresent studies sufficient progress has been made sothat the further development of the dual-task approachin measuring cognitive load in learning with complexmultimedia systems seems worthwhile.

From a more theoretical perspective, a central is-sue that has to be resolved is the simultaneous testof differences in learning outcomes (e.g., modalityeffects) on the one hand, and of secondary task per-formance on the other. As a result of the experimentspresented in this paper, we obtained empirical evi-dence for the convergent validity of the underlyingcognitive load hypothesis. In former experiments(Brünken & Leutner, 2001; 2002), in which the samelearning system, the same knowledge acquisitiontests, and comparable participants had been used, we


found modality effects on knowledge acquisition.However, using conventional analysis methods fromexperimental psychology in the present experiments,we had to choose an all-within-subjects design tocompensate for individual differences in reactiontimes. This design did not allow for the testing ofmodality effects on knowledge acquisition withinparticipants, because test items were not directlymatched with screen pages. Implementing the assess-ment tool in a between-subjects design would ne-cessitate a different and much more complex ap-proach to analyzing reaction times. However, in sucha design, our method of measuring cognitive loadshould be capable of explaining also other differ-ences in knowledge acquisition, for example, be-tween novices and experts, or differences caused bylearning material redundancy (Mayer, 2001), whichcan also be interpreted as cognitive load effects.

Further research is also necessary with respect tothe task types used for the secondary task. Until now,the secondary tasks we used involved very simple vi-sual perception that did not require any elaborate cog-nitive processing. A systematic variation of the tasktypes would not only be helpful for a validation of thepresent findings, but would also give a more preciseindication of the type of cognitive resources that isused for processing multimedia learning materials.

Finally, in the present studies, we assessed differ-ences in visual cognitive load produced by differentpresentation forms of computer-based learning mate-rials. However, with our approach, the auditory cog-nitive load produced by these systems can also beassessed through the use of an auditory secondarytask. Actual studies, such as by Moreno and Mayer(2000), show that the implementation of additionalauditory information (such as background music) ina multimedia learning system reduces knowledgeacquisition. The authors interpret this effect as anauditory cognitive load effect. We are currentlyworking on the implementation of a auditory second-ary task for our assessment tool to test for relateddifferences in reaction times during the processingof auditory information in a dual task condition.

The direct measurement of cognitive load in multi-media learning will, for the first time, allow for morethan indirect conclusions to be made regarding work-ing memory utilization in learning with multimedia,and has a high potential to provide further empiricalevidence for a multimedia dual-processing theory.

ReferencesAsymetrix (1997). Toolbook instructor II [Computer Pro-

gram, PC]. Bellevue, WA: Asymetrix Learning Sys-tems.

Baddeley, A. D. (1986). Working Memory. Oxford, UK:Oxford University Press.


Baddeley, A. D., & Hitch, G. J. (1974). Working memory.In G. H. Bower (Ed.), The psychology of learning andmotivation: Advances in research and theory (pp. 47Ð89). New York: Academic Press.

Baddeley, A. D., & Logie, R. H. (1999). Working memory:The multiple-component model. In A. Miyake & P.Shah (Eds.), Models of working memory: Mechanismsof active maintenance and executive control (pp. 28Ð61). Cambridge, UK: Cambridge University Press.

Brünken, R., & Leutner, D. (2000). Neue Medien als Geg-enstand empirischer pädagogischer Analyse: Stand derForschung und Perspektiven. New Media as a topic ofempirical educational analysis: State of the art and per-spectives] In D. Leutner & R. Brünken. (Eds.), NeueMedien in Unterricht, Aus- und Weiterbildung. AktuelleErgebnisse empirischer pädagogischer Forschung(pp. 7Ð16). Münster, Germany: Waxmann.

Brünken, R. & Leutner, D. (2001). Aufmerksamkeitsver-teilung oder Aufmerksamkeitsfokussierung? Empir-ische Ergebnisse zur Split-Attention-Hypothese“ beimLernen mit Multimedia. [Split attention or focusing ofattention? Empirical results on the split-attention hy-pothesis in multimedia learning]. Unterrichtswis-senschaft, 29, 357Ð366.

Brünken, R., & Leutner, D. (2002). Cognitive load and themodality effect in multimedia learning. Submitted forpublication.

Brünken, R., Steinbacher, S., Schnotz, W., & Leutner, D.(2001). Mentale Modelle und Effekte der Präsentatio-ns- und Abrufkodalität beim Lernen mit Multimedia.[Mental models and the effect of presentation and re-trieval codality in multimedia learning] Zeitschrift fürPädagogische Psychologie, 15, 15Ð27.

Chandler, P., & Sweller, J. (1991). Cognitive load theoryand the format of instruction. Cognition and Instruc-tion, 8, 293Ð332.

Donk, M., & Sanders, A. F. (1989). Resources and dual-task performance: Resource allocation versus task in-tegration. Acta Psychologica, 72, 221Ð232.

ERTSLab (1999). [Computer program for MS-DOS].Frankfurt a.M., Germany: BeriSoft.

Heuer, H. (1996). Doppeltätigkeiten. [Dual Tasks] In D.Neumann & A. F. Sanders (Eds.), Aufmerksamkeit(pp. 163Ð218). Göttingen, Germany: Hogrefe.

Kirschner, P. A. (2001). Cognitive load theory: Implica-tions of cognitive load theory on the design of learning.Learning and Instruction, 12, 1Ð10.

Klauer, K. J. (1987). Kriteriumsorientierte Tests. [Criterionreferenced tests] Göttingen, Germany: Hogrefe.

Klauer, K. C., Oberauer, K., Rossnagel, C., & Musch, J.(1996). Mentale Bilder und Mentale Modelle [Mentalimages and mental models]. Zeitschrift für Psycholo-gie, 204, 41Ð54.

Logie, R., Baddeley, A. D., Mane, A., Donchin, E., &Sheptak, R. (1989). Working memory in the acquisitionof complex cognitive skills. Acta Psychologica, 71, 87.

Macromedia (2000). Authorware 5 [Computer program,PC]. San Francisco: Macromedia Inc.

Mayer, R. E. (1997). Multimedia learning: Are we askingthe right questions? Educational Psychologist, 32, 1Ð19.

Mayer, R. E. (2001). Multimedia learning. New York:Cambridge University Press.

Mayer, R. E., & Moreno, R. (1998). A split-attention effectin multimedia learning: Evidence for dual processing


systems in working memory. Journal of EducationalPsychology, 90, 312Ð320.

Mayer, R. E., Moreno, R., Boire, M., & Vagge, S. (1999).Maximizing constructivist learning from multimediacommunications by minimizing cognitive load. Journalof Educational Psychology, 91, 638Ð643.

Microsoft (1996). Windows 95 operating system [Com-puter Software, PC]. Redmond, WA: Author.

Miyake, A., & Shah, P. (Eds.) (1999). Models of workingmemory. Mechanisms of active maintenance and exec-utive control. Cambridge, UK: Cambridge UniversityPress.

Moreno, R., & Mayer, R. E. (1999). Cognitive principlesof multimedia learning: The role of modality and conti-guity. Journal of Educational Psychology, 91, 358Ð368.

Moreno, R., & Mayer, R. E. (2000). A coherence effect inmultimedia learning: The case for minimizing irrele-vant sounds in the design of multimedia instructionalmessages. Journal of Educational Psychology, 92,117Ð125.

Mousavi, S. Y., Low, R., & Sweller, J. (1995). Reducingcognitive load by mixing auditory and visual presenta-tion modes. Journal of Educational Psychology, 87,319Ð334.

Paivio, A. (1986). Mental representations: A dual codingapproach. New York: Oxford University Press.

Paas, F. G. W. C., & van Merrienboer, J. J. G (1993). Theefficiency of instructional conditions: An approach tocombine mental-effort and performance measures. Hu-man Factors, 35, 737Ð743.

Schnotz, W., & Bannert, M. (1999). Einflüsse der Visuali-sierungsform auf die Konstruktion mentaler Modellebeim Text- und Bildverstehen [Influence of the type ofvisualization on the construction of mental models dur-ing picture and text comprehension]. Zeitschrift für Ex-perimentelle Psychologie 46(3), 217Ð236.

Schnotz, W., Böckheler, J., & Grzondziel, H. (1999). Indi-vidual and co-operative learning with interactive ani-mated pictures. European Journal of Psychology ofEducation, 16, 245Ð265.

Sweller, J. (1988). Cognitive load during problem solving:Effects on learning. Cognitive Science, 12, 257Ð285.

Verwey, W. B., & Veltman, H. A. (1996). Detecting shortperiods of elevated workload: A comparison of nineworkload assessment techniques. Journal of Experi-mental Psychology: Applied, 2, 270Ð285.

Wickens, C. D. (1984). Processing resources in attention.In R. Parasuraman & D. R. Davies (Eds.), Varieties ofattention (pp. 63Ð102). London: Academic Press.

Wittrock, M. C. (1990). Generative processes of compre-hension. Educational Psychologist, 24, 345Ð376.

Dr. Roland Brünken

Independent Research GroupErfurt UniversityPostbox 900221D-99105 ErfurtGermanyTel.: +49 363 7371405Fax: +49 363 7371031E-mail: [email protected]

Assessment of Cognitive Load in Multimedia Learning Using...

Documents

Transcript of Assessment of Cognitive Load in Multimedia Learning Using...