EuropeanJournal,J! Psychology of Education1995, Vol, X ,,"2, 181-196e 1995, I,S,P,A.

Validation of a theory about functions and effectsof embedded support devices in distance educationlearning materials

Rob MartensMartin ValckeOpen University, Heerlen, The Netherlands

In this contribution a theoretical model is explored and validatedthat describes and explains learning based on learning materials wherethe basic content is enriched with embedded support devices. Embeddedsupport devices (ESD) are elaborations of the basic content ofa course,intended to support students learning in a distance educational context.Examples are: questions with fe edback, advance organizers and studyguidelines. The model represents the interaction ofa large set of vari­ables and processes and has been tested using a variety of researchstudies, each fo cusing on specific interrelations and/or variables.

The research results support the positive impact ofembedded sup­port devices in written learning materials on study outcome.Furthermore the results indicate the impact ofindividual variables, suchas prior knowledge and attitude towards computers . These variablesinfluence the way students process the learning materials and/or use thetask environment and have an interaction effect on study outcomes,

Introduction

One of the basic characteristics ofleaming materials in distance education contexts, is theprovision of embedded support devices (ESD) that have been added to the domain specificknowledge base (basic content). These devices comprise questions, content pages, illustra ­tions , examples, activ ities, tests , examples, schemes, cases , etc . Given the large amount ofESD in learning materials, they can be clustered considering hypothetical basic functions andeffects as suggested by Valcke, Martens, Poe1mansand Daal (1993) :

- orienting ESD: learning objectives, references to other learning materials, referencesto required prior knowledge, etc.

- processing ESD: indexes, additional learning materials , advance organizers, illustra­tions, glossaries, introductions, study advice, summaries , tables, examples , etc.

- testing ESD: self-test items, exercise items, answers and feedback .

Acknowledgments: The authors wish to thank H. Dekeyser, K. Goeminne and G. Schuyten from the Univers ity ofGent(Belgium) for their contributions in theresearch on interactive learning environments.

182 R. MARTENS & M . VALCK E

The support devices can make up to 45% of the learning materia ls.Learni ng materials developers design their materials, building on expectations about the

hypothetical functions and effects of ESD. These devices form the base of "course models"(learning unit model, text book, working book model, etc.) and are aspects that are integratedinto templ ates to design the materials. The learn ing materials, discussed in this article areeither presented in printed format (printed learning environment PLE) or presented on a com­puter screen (interactive learning environment ILE).

The weak theoretical base for ESD

ESD have been the subject of a vast body of research since printed texts playa major rolein learning materials. In the seventies and eighties some major reviews of this research havebeen elaborated (cf. Alesandrini , 1984; Barnes & Clawson, 1975; Hartley & Davies , 1976;MacDonald-Ross, 1977; Rothkop f, 1970; Waller, 1979; etc.). This sounds to be in contrastwith our statement: there is no theoretical base nor a c1earcut empirical base to found the useof ESD. An analysis of the nature and design of this research body is needed to understandthis statement:

- Earlier research, was rarely aimed at distance educat ion or focused on independentlearning .

- Research set-ups did not reflect natural learning contexts and were mainly set up inisolated lab-like settings (Lockwood, 1992).

- This research focused on single, isolated support devices in contrast to educationalpractice where the complex mixture of devices support s independent learning .

- Researcher focus especially on "product" effects of the use of embedded supportdevices; for instance study outcomes or posttest scores. As a consequence , researchershardly know what learners actually do when they are presented with text-embeddedsupport (cf. Driscoll , Moalle m, Dick, & Kirby, 1994). It is for instance not surewhether studen ts use or know how to use these support facilities (Winne, 1983).Despite the growth of distance education, little systematic knowledge is available con­cern ing the ways in which studen ts use the distance teaching mater ials provided tothem (Clyde & Crowther, 1983; Lowijck & Simons, 1991; Marland , Patching, Putt, &Putt , 1990; Schuemer, 1993). Students - when workin g at a distance with printedlearning mate rials - have complete veto power over learning (Rothkopf, 1970 ).Therefore, researchers sometimes train students to use ESD in a certain way. A similarapproach is adopted by Bernard (1990) and De lon g and Simons (1988).

- The question is not only whether students use the embedded support devices orwhether they know how to use them, but also whether they need the support provided .Waller (1979) mentions e.g., different user objective s when dealing with study texts :entertainment, recapitulat ion/reviewing, browsing , studying in depth , searching for aparticular item. Each user perspect ive affects the potent ial effects/functions of ESD.Most ESD are des igned to evok e some kind of behav iour of students. Wade andTrathen (1989, p. 40) however state: " (...) despite their popularity, there is no consis­tent empirical evidence that these technique s are any more effect ive than more passivemethods of reading-only or repetitive reading...".

- Even when - from a theoretical point of view - functions and effects are assigned tospecific ESD, there is the problem Meyer and Watson (1991) and Schuemer (1993)signal: practical implications derived from this are often not useful.

Towards a theoretical model

In our opin ion it is crucial to develop a theoretical model that describes and expla ins how

FUNCTIONSAND EFFECTS OF EMBEDDED SUPPORT DEVICESIN LEARNING MATERJALS 183

students learn from learning materials, considering the central role of ESD, but also appraisingother key processes and variables . Figure I represents our theoretical model. This model pic­tures the interaction between variables and processes :

- A key variable is constituted by the learning materials themselves consisting of basiccontent enhanced with ESD.

- The materials have been developed to influence cognit ive aptitudes such as the learn ­ing process .

- Moreover, since the learning materials are text -based, also the cognitive aptitude"reading comprehension" is influenced. The reading process is expected to interactwith the learning process .

- Non-cognitive aptitudes, such as motivation and attitudes are expected to interact.

- Variables in the individual learner, such as prior knowledge , reading compre hensionlevel, educational level, etc. are expected to influence directly or indirectly the learn­ing process .

- The specific delivery mode of the learning materials (ILE or PLE) determines the taskenvironment.

"'IIddvo... ~opI"

F= ,-~I(AC '" fC

lDHbl..tohloo~

~~ -- ~II =I .-.., ......

IIIk tll _

11I tnt + ESD

Figure 1. A model to describe and explain learning based on learning materials enriched withembedded support devices .

This article is about the validation of the model presented. We will focus on the possiblefunctions and effects of ESD, on the intervening influence of individual varia bles (studentaptitudes) and on the relation between approachi ng learning materials with ESD and studyresults. First we will elaborate in short the key processes and variables in our model as depict­ed in Figure 1.

Key variables and processes

The learning process. The theory of cognitive functioning of Sternberg (1986, 1988) isused as a base to approach the learning process. Cognitive functioning is - according toSternberg - described with the concept "components". A component is "an elementary infor-

184 R. MARTENS & M. VALCKE

mation process that operates upon internal representations of objects or symbols" (1986, p.225). Sternberg distinguishes three groups of components:

- Metacognitive components (MC): Metacognitive components control, evaluate andplan the activities of the other components, They are executive processes.

Performance components (PC) : Performance components are those processes thatoperate when a task is actually executed. The PC are grouped as stages when execut­ing a task .

Knowledge acquisition components (KAC): The knowledge acquisition componentsare involved when new information is getting integrated in already available informa­tion in memory . This integration process implies encoding, combining and comparing.

At a theoretical level, the theory of Sternberg has been used to derive functions andeffects that can be assigned to specific ESD ; e.g., "questions" activate the knowledge acquisi­tion components. In a specific research set-up we will test these hypothetical theoretical func­tions and effects.

The reading process . In the literature, the theoretical view on discourse comprehension ofKintsch and van Dijk (1978) is dominant in theory and research about the nature of the read­ing process. In short, their approach can be described as follows : "Discourse comprehension,from the viewpoint of a computational theory , involves constructing a representation of a dis­course upon which various computations can be performed, the outcomes of which are com­monly taken as evidence for comprehension. Thus, after comprehending a text, one might rea­sonably expect to be able to answer questions about it, recall or summarize it, verify state­ments of it, paraphrase it, and so on." (Kintsch, 1988, p. 163). To date this view on discoursecomprehension remains the most widespread and accepted view (cf. Rayner & Pollatsek,t989; Britton & Gulgotz, \99\; van Oostendorp & Peeck, 1991).

Reading (discourse comprehension) and learning are closely interrelated as already theKintsch-text suggests : learning can be seen as a special form of reading with the intention tounderstand and/or reproduce the text. As a consequence reading ability does correlate withschool achievement and general intelligence indices (Rost, 1989). Individual differences inreading ability are therefore expected to be an important interaction process and variable in thelearning process (Oostendorp &: Peeck , 1991). When validating the model, interaction effectsof reading comprehension measures will be considered.

The task environm ent. In our theoretic al model, the learning materials are presented with­in the context ofa task environment. It was stated that the specific delivery mode of the Jearn­ing materials determines the task environment. The particular learning environment can bevery different in a distance education context. In traditional distance education approaches, thematerials are especially delivered in printed form. In the seventies there was a growing ten­dency to add audio-visual components. Since the eighties, the computer has enriched thelearning environment. Nowadays multi-media and electronic communication media influencethe task environment.

In our model, we focus on the way text, enhanced with ESD, is delivered. Consideringthe context of the Open University (Ol,J) of the Netherlands, two main delivery modes areresearched: printed learning environments where the materials are printed in textbooks, work­ing books, etc . versus interactive learning environments where the learning materials aredelivered by making use of computers. The delivery mode is expected to influence to a largeextent the learning process of students. The degree of interactivity with the learning materialsis for instance completely different. In electronic learning environments, students can activelymanipulate or select text parts, browse through the materials, etc . This is expected to be ofparticular importance in relation to the use of ESD.

Next to the delivery mode, also the way ESD are presented is important: are they embed­ded into the basic content of the learning materials and as a consequence hardly discernable,or are they clearly discernable and put next to the basic content?

FUNCTIONS AND EFFECTS OF EMBEDDED SUPPORT DEVICES IN LEARNING MATERIALS 185

The theoretical model does not suggest that delivery mode, the degree of interactivity andthe degree of discernability have a straightforward impact on study outcome. The interrelationbetween the processes and variables in the model also indicates that the effectivity of a partic­ular task environment can depend on individual variables and is to be related to the cognitiveand non-cognitive aptitudes.

Individual variables. The model stresses the importance of individual variables in rela­tion to the other processes and variables. It is hypothesized that differences in students influ­ence the use and effectivity of ESD. This suggests that the optimal choice and/or combinationof ESD could be different for different students and might depend of for instance prior know­ledge level, learning style. The importance of the individual variables may not be underesti­mated in a distance education context. The student population of the Dutch Open university isfor instance very heterogeneous. Espec ially the "open" character of the education paradigminvokes a var iety of student needs and demands that have to be met. In this section we willintroduce a set of potentially relevant individual variables.

Only a subset will be manipulated in our research set-up to validate the theoretical model.The variables discussed originate from widely different theoretical positions or frames of ref­erence.

There is a comprehensive theoretical base to found the impact of prior knowledge on theacquisition of new knowledge (e.g., Ausubel, 1968; Mayer, 1979; Reigeluth & Stein , 1983).Prio r knowledge at the beginning of a study is an important variable to predict studentsprogress and study results. Not only does prior knowledge reflect the availability of relevantskills and knowledge in students, but it can also reflect general ability or interest in the subjectof a course (Tobias, 1994). Dochy (1992) has tried - in a distance education context - to useprior knowledge state tests to orient the learning proce ss. He approached prior knowledge as amulti-dimensional concept and distinguished two major educational-psychological dimensionsin prior knowledge: a behavioral dimension (knowing - insight - application) and a know­ledge dimension (facts , concepts, relations, structures, methods). These knowledge profiledimensions have been empirically validated and are helpful to differentiate between groups(Dochy & Valcke, 1992; Keeves, 1988).

Learning style is a multi-componential construct, containing several cognitive style com­ponents. Van Rijswijk and Vermunt (1987 a.b,c) and Vermunt (1992) distinguish three learn­ing regulation styles: external regulation, self regulation and no regulation. They describe fourtypes of learning style. Regarding the concept of learning style, there are many relations withabi l i ty , aptitudes, strategy and motivation (cf. Kini, 1994; De Diana & van der Heiden, 1994) .As a result the construct validity of the concept learning style is sometimes questioned (Corno& Snow, 1986). The use of computers as learning medium makes it possible to monitor whatstudents are actually doing while studying. For instance it can be monitored by means of logfiles what support devices are actually used by students, thus providing information aboutsome aspects of students' learning style s.

In literature, field dependencelfield independence is put forward as an important individ­ual variable . According to Kini (1994) it is the most important independent dimensions ofcognitive style, different from intelligence and creativity . Individuals classified as field inde­pendent are aole to overcome the embedding context of a field and tend to perceive certainparts of the field as discrete from the surroundings. Individuals labelled as field dependent areguided by the field as a whole and are influenced by the field factors and complexity of thesurrounding visual area . They depend on external cues to make judgements. It is found thatthis variable affects educational achievement.

When discussing the task environment, we already indicated the importance of individualvariables to explain the effective use o f learning environments or the impact of particulardelivery modes, The individual variable "preference for a perceptual mode" can be directlyrelated to the latter. Preference for a "verbal" or "visual" representation differentiates betweenstudents: students using sequential methods of processing information rely on verbal represen­tations; students adopting parallel processing methods rely rather on visual representations.

186 R. MARTENS & M. VALCKE

Not surprisingly, individuals classi fied as visual are better than their verbal counterparts in therecall of highly visual materi al (Kini, 1994).

Metacognit ion is used to describe the conscious steps a learne r has to take to direct andmonitor his or he r learning ac tivi ties . Man y research has show n that students di ffer inmetacogn itive abilit ies (e.g. , Elen, 1992). It is not clear to what extent this is related to theother aptitudes that are indicated in this section (e.g., motivation, prior knowledge, learnin gsty le). Moreo ver, since it is difficult to make a clear distinction betwee n metacogni tion andbehaviours that are related to learn ing sty le or cognitive style we will not trea t metacognitionas a separate individual variable aptitude .

The "open" nature of the Open university educational provisions has as a consequencethat students can start studying at university level without specific entrance requirements.Educational level is therefore an important variable in fluenc ing the learning process.Embedded support devices can playa major role in this conte xt since these devices give stu­dents help in gett ing access to the learning materials, help to process the knowledge base andhelp students to test their mastery . Reviewing the literature, the impact of educational level isstill not clear. De Bruijn states for instance (1993, p. 12): "( ...) it appeared that the relativelyhigh educated adults had a stronger need for abstract conceptualisation , reflecti ve observation,and active experimentation than the less educated subjects". Also Daal (1991) found in herstudy with au students a positive relation between educational level and study progress. But ,Poelmans & Jans sen (1994) signal that there is only a weak relationship between educationallevel and drop out in distance education.

Another consequence of the open chara cter of the Dutch Open university approach, is thedifference in age levels of students ( 18 to 82 years). One can quest ion whether this influencesthe learning process and the way students work in specific learning environments.

The Dutch Open university deplo ys an act ive policy to promote emanc ipat ion of f emalestudents (gender). For instance. when designing Ol.I-courses, the course develop ers have toindicate explicitly what they will do to take the gender variable into account. In the literatureexplici t reference is made to differences in learning betwee n male and female students. DeBruijn (1994, p. 12) found: "( ...) it appeared that the female subjects had a stronger need forconcrete experience and active experimentation, while the male subjects ' primary need wasfor abstract conceptual isation ." Long, Gaynor, Erwin and Williams add to this (1994, p. 20) :" Intrinsic mot ivation may prove an important se1fmanagement issue for females more than formales, whereas males ' self-manage ment may be linked to a more broadly-based sat isfacti onwith their academic life" . Research at the au of the Netherlands revealed that female studentstend to do more and in line they were instructed to do. They also tend to make more use ofstudy guidance facilities in distance education (Poelmans & Janssen, 1994). The same authorspoint at the interrelation between gender and work load : women have more problems in savingenough time from household activities for study time than male students.

To the former list of individual variables that might play a role in the way students learnand use learning materials in a distance education setting, a long list of other variables dis­cussed by a variety of authors can be added (Clyde & Crowther, 1983; Como & Snow, 1986;De Bruijn, 1993; Elen, 1992; Holmberg, 1986; Kember & Gow, 1989; Marland et aI., 1990;Pask, 1976; Van Joolingen, 1993; Vermunt, 1986; Willems, 1989). This list indicates how dif­ficult it is to set up research in educational contexts. Control of all these variables, especiallywhen researching in ecologically valid contexts is hardly possible :

- intelligence

course-focused versus interes t-focu sed;

holist learne rs, ser ialist and versatile learners ;

- deep- level processing versus surface-level processing;

deep level, concrete and stepwise processing;

studen t concep tions abou t the study process;

experienced versus novice students;

FUNCTIONS ANDEFFECTS OF EMBEDDED SUPPORT DEVICES IN LEARNING MATERIALS 187

need for concrete experience versus need for abstract conceptualisation;

- cognit ive comple xity versus simplicity (the degree of differentiation or hierarchic inte­gration)

- reflectivity versus impulsivity

- willingness to take risks in pursuing goals

convergence versus divergence (the tendency to converge toward or diverge from theobvious in hypothesis generation);

intrinsic versus extrinsic motivation

learning intentions (performance versus pleasure in learning)

experi menter - theorist

Processes and variables in action

In the previous sections we discussed a large number of individual variables that possiblyaffect the learning process . These variables are expected to interact with cognitive and non­cognitive processes as supported by a specific task environment. We also already stressed theimportance of the task environment (e.g., ESD, delivery mode) in dealing with individual vari­ables to support the learning process .

Considering the importance we attach to these individual variables and especially the factthat we suggest that specific learning environments are expected to be better suited for specificstudents, our research fits into the research tradition called Aptitude-Treatment-Interaction(ATI). In ATl-research (Snow \992; Cronbach & Snow, \977) the relation between studentabilities or characteristics (Aptitudes), educational settings or learning methods (Treatment)and learning outcome s is investigated . Crucial is the assumption that the effects of treatmentsare related to aptitudes (Boekaerts & Simons, 1993). A'Tl-research outcomes often contrad icteach other , results often cannot be replicated or are ambiguous (e.g., Simons , 1982; Crombag,1979; Thijsser. & Span, 1985). This is in part due to the vague nature of the concept "aptitude"that leads to an almost infinite collection of concepts , methods, etc. that might affect students 'academic performance (Long et ai, 1994).

In the context of this article a further elaboration of the ATl-paradigm is not relevant.The focus on the validat ion of our theoretical model puts forward the validation of other inter­relations and effects , next to the ATl-prepositions, Moreover, our focus on the exploration ofpotential interrelations within the model does as yet not imply that students are assigned tospecific treatment conditions on the basis of individual variable measures to attain a specificstudy outcome .

From theory to research

The elaboration of the theoretical base for this study and especially the review of ESD­related research resulted, next to a theoretical model, in a set of guidelines to direct moreeffective and effic ient research. We summarize these guidelines as far as they have been incor­porated into our consecuti ve research designs:

Research the actual learning behaviour in relation to study outcomes. Often there is aweak relation between behav iour as measured by aptitude tests and the behaviourrequired for achieving learning outcomes (Boekaerts & Simons, 1993)

- Assess time investment and learner engagement as potential co-variables (cf. Como &Snow, 1986).Check particularities of the study process that might reflect compensation for less suit­ed learning condit ions (e.g., studying longer and with more effort).Examine long term effects of treatments (cr. Lohman, 1993).Choose for a high ecological validity of the research setting.

188 R. MARTENS & M. VALCKE

Since aptitudes are interrelated, multi variate analysis techniques should be used toanal yze the research data. The statistical techniques should also be helpful to unravelthe influence of interaction and co-variables.Design different experimental conditions so that they can lead to differences in out­comes. Often conditions in research are rather similar (cf. Crombag, 1979) .Use basic concepts to describe individual differences. The concept of "reading com­prehension" for instance is more basic then the concept "learning style".

- Differences between experimental treatments should be easy to interpret and general­izable.Make use of on line measurements to grasp what students are actually doing whilestud ying .

Review ofthe validat ion research: printed learning environments

The findings of four research projects are reported here that are helpful to validate orexplore aspects of our theoretical model. After a brief review of each set-up and the results, aconclusion is drawn in relation to the model. These four researches have in common that theybuild on printed learning environments when focusing on the functions and effects of ESD inthe learning materials. In a next section we add the results of research with interactive learningenvironments. We repeat that the set-up, analysis procedure and results section in relation toeach research project will only be discussed in brief. We refer to specific research reports for amore elaborated presentation.

Research with the interview method. In a first research set-up twenty-five law students ofthe au of the Netherlands were interviewed in detail about the way they study Ol.l-Iearningmaterials (Valcke, Martens, Poelmans, & Daal, 1993).

The research results indicated that students use ESD to a high extent (varying from 60%to 100%) and highly appreciate them . A distinction can be made referring to aspects of studyregulation: surface level versus deep level studying (Marton & Saljo , 1984) . Students who useESD to a high extent and at a deep level, attain significantly higher study outcomes. Studyingat a surface level implies "only" reading and learning by heart; deep level learning putsemphasis on comprehension. It was found that especially higher educated female students tendto make more use of ESD. The very high general use of ESD made it difficult to find interac­tions between use of ESD, student aptitudes and effects of use of ESD .

This research set-up also checked whether students assign functions and effects to theembedded support devices in line with functions and effects as derived from our theory. Theresults are clear: students assign personal functions and effects to embedded support devices.Moreover, they approach them differently according to the phase in their learning process.The results of this study indicate the high importance students attach to the embedded supportdevices. They also indicate the importance of individual variables in our model where theyaffect the use of embedded supported devices . But, it remains unclear how ESD affect thelearning process in terms of components of cognitive functioning.

Research with questionnaire method At the au of the Netherlands, the department OlDsets up - yearly - an evaluative study invol ving about 2000 students, based on the use of ques­tionnaires. Students taking courses in different subject domains participate in these studies. In1992, a part of the questionnaire consisted of questions about ESD use and the link of ESD­use with stud y success (Martens, Poelmans, Daal, van Staa, & van Meurs , 1993; van Staa , &van Meurs, 1992).

Student appreciation of ESD is high . Only 7% of the students prefer learning materialswithout ESD . Students studying "Cultural Sciences" appreciate ESD to a lesser extent(Pearson correlation coefficient -.24, p<O,O 1). No interrelations between the use of ESD andthe level of stud y success can be found . This might be partly due to the way study success wasoperationalised: the time it takes to attain study credits.

FUNCTIONS AND EFFECTS OF EMBEDDED SUPPORT DEVICES IN LEARNING MATERIALS 189

No specific conclusions can be drawn from this research to validate the theoreticalmodel.

Diffirent versions ofone course. In this research project, the main focus was on the cen­tral role of ESD to influence learning: Does making use of ESD during the learning processhave a positive effect on study outcomes? Two versions of a learning unit of the OU-course"Introduction to governmental law" were elaborated (Martens, Poelrnans, Daal, & Valcke,1993). A first version of the course was a replication of the original printed OU-textbook. Thisversion incorporated a large set of embedded supported devices. The second version proved tobe about 40% shorter in length after deleting most embedded support devices (such asschemes, summaries and learning objectives). Care was taken that this omission did not alterthe basic content and did not affect the coherence of the text. Thirty-six students from theUniversity of Limburg (RUL) studied one of both course versions. Several tests and question­naires were developed to measure the effects of course versions on study success and thepotential interrelation of study success with independent student variables. The tests used tomeasure study success were carefully constructed in order to detect differences in masterylevel (knowing, insight, application) or differences in mastery of specific knowledge types(facts, concepts , relations, structures, methods).

No significant effects in use of study time, student evaluations of the learning accesslevel and "knowledge" questions could be observed. But students who studied the course ver­sion with ESD, achieved significant higher scores on "insight" questions (Anova,F(I,34)=3.47, p<0.05). Interrelations with the independent variables prior knowledge, timeused, reading comprehension, educational level and gender were tested. These variables wereadded to the ANOYA models as covariables. No relevant interactions were detected .

The results of this study indicate the importance of embedded support devices in ourmodel as they relate to the learning process. Since the effect on study outcomes is especiallyfound in relation to "insight questions", one can suggest that ESD especially influence particu­lar components of cognitive functioning .

Eye movement registration. Eleven subjects participated in this in-depth research(Martens , Poelmans, Daal, Valcke, & Stolk, 1994). The students studied one learning unit ofan OU-course in the law-domain. In the learning materials used, ESD were integrated into thetext. The eye movements were monitored by means of the corneal reflection/pupil centre tech­nique with an infrared camera (Stolk, Smulders, & Belsmaeker, 1991. Eye movement patternswere compared between text parts with ESD and those without ESD. For each text part a com­puter program computed the total reading time per word. Prediction hypotheses were formulat­ed as to the reading time ofeach specific ESD as compared to the reading time for regular text.

It was found that the average reading time was for most ESD in line with the hypotheses.For instance content pages have shorter reading times per word and questions are read longerper word, thus indicating a deep level use. Furthermore it turned out that every ESD analyzedwas looked at, at least once by each student.

The very reliable research set-up confirmed the earlier results indicating a very high levelofESD-use.

Review ofthe validation research: interactive learning environments

When analyzing the use of printed learning environments , the possibilities have not beenresearched to adapt learning materials to student needs or questions. The research projects inthe next section of this article either focus on the adaptation feature or compare student activi­ties and outcomes of groups making use of these features.

We will not discuss the design of interactive learning environments since this will be inpart done in the contributions of Portier & Van Buuren (this issue) and Yuist (this issue). Wewill only focus on some basic characteristics of these environments as they are related to ourtheoretical model.

190 R. MARTENS & M. VALCKE

Interactive learning environments. To research the impact of ESD in lLE, we used acomputer environment to present learning materials to students and to give them the possibili­ty to manipulate the learning materials. The explicit distinction between basic content andESD, the level of ESD-integration and the degree of interaction in manipulating ESD form thebase of our studies with ILE:

The database of learning materials in the computer program reflects the basic distinc ­tion between basic content and the embedded support devices in the learning materials .

- In printed learning environments ESD are integrated into the basic content wherecourse developers consider them to be of relevance. In ILE, variations in levels of inte­gration or the discernability can be realized. Two ILE with a different discernabilitymode have been developed. In the first environment the ESD are hardly discernablefrom the surrounding asic content. In the second environment the ESD are explicitlydiscernable from the basic content and are preceded by a header indicating the type ofESD.

- In printed learning materials, ESD are available depending on the design-decisions ofthe learning material developers. In interactive learning environments, we transfer thedecision and responsibility to activate and incorporate ESD into the basic content tothe student. The degree of "interactivity" is considered as a basic feature of ILE.Therefore, two versions of an ILE with embedded support devices have been elaborat­ed. In a first version, the interaction possibilities with the ESD have been blocked. Inthis way the interactive learning environment is reduced to a simple , straightforwardelectronic textbook. In the second version, students have full control to (de)activate thesupport devices at a global and/or local level. Moreover, comparing ILE with PLE inthe same research design also builds on the differences in interactivity discussed here.

Interactive learning environments, a first research set-up. Portier and van Buuren (thisissue) describe this research set-up in detail, thereby focusing on design features of lLE and aspecific set of research hypotheses. A sample of 46 students participated in this study wherenext to students working with ILE, a control group participated in regular face-to-face lecturesand working-groups. Computer interactions were continuously logged in a personal activitiesfile. All students in the experimental condition used the same type of ILE.

The results reveal that study outcomes of students in the control and experimental condi­tions are not significantly different. Students in the independent learn ing situation attainedcomparable study outcomes .

Analysis of data in the log-files is helpful to link ESD-use to individual student variables.Students with a high level of prior knowledge interacted significantly more with the embeddedsupport devices than students with a lower prior knowledge level.

Relating the results to our theoretical model, we see that the impact of individual vari­ables is stressed again in the interaction with the task environment and the learning process .

Interactive learning environments, a comprehensive study. In this second study, interac­tive learning environments were contrasted with printed learning environments and face-to­face lectures (Portier & van Buuren, this issue) . Moreover, types of ILE and PLE wereresearched, considering the discernability and interactivity modes described earlier. A largeset of variables was controlled for, although the research was completely integrated into thenormal learning set-up of the students . Students studied during three working sessions threechapters of a course on statistics. The independent learning sessions were set up at the univer ­sity. Students did not get learning materials to take home. Automatic log-files stored the inter­actions of students working with the ILE and help to discern between low and high ESD-use.

All first-year students (n=502) of the university of Gent, Belgium (Faculty ofPedagogical and Psychological Sciences) participated in the study. Students were selected atrandom to participate in one of five research conditions (Table I). The sessions were part ofan obligatory part of the study program. Students could not take home the study materials .

FUNCTIONS AND EFFECTS OF EMBEDDED SUPPORT DEVICES IN LEARNING MATERIALS 191

Table IFirst-year student population at the University of Gent , Faculty of Psychology andEducational Sciences, Course Statistics I (N=502)

Type ofleaming environment

Experimental Groups---------

Discemable ESD Non-discemable ESD Control group

Interactive (lLE)

Printed (PLE)

I (n=40)

II (n=40)

lIla (n=20)IIIb (n=20) non interactive

IV (n=40)

V (n=342)

Reading comprehension level, prior knowledge level and a set of other variables weremeasured and controlled for. At the end of the research period a posttest was used to measurestudy outcome. The final examination scores were used to measure long term effects .

We focus here on the results of students in condition I, IlIa and I1Ib. This implies that dif­ferences in discernability mode and interactivity mode are expected. Embedded supportdevices are clustered according to the clusters discussed earlier: orienting ESD, processingESD and testing ESD .

Comparing study outcomes of students in the two different discernability conditions, inrelation to the use of ESD results in a clear picture (Table 2).

Table 2Study outcomes as a function of ESD-use (low/high) and discernability of ESD (low/high),with size per group

Orienting Processing Testing

Discernability low high low high low high

Low 7.90 7.40 6.00 8.55 6.45 9.38(10) (10) (8) (II) (II) (8)

High 7.77 7.36 8.17 7.00 7.77 7.60(13) (11) (12) (12) (13) (10)

Multivariate analysis of variance reveals no significant main effects of discernability, noruse of several types of ESD on study outcome. However, if the two-way interactions are takeninto consideration, the analyses revealed a significant interaction between discernability andthe use of processing ESD (F(I,42)=5.66, p<.05) on study outcome. In addition we find atrend interaction between discernability and use of testing ESD (F( 1,41)=3.67, p=.06). Bothinteractions can be interpreted as follows: students who use much processing and testing ESDbenefit from a condition (treatment) in which ESD are not discernable, i.e., integrated into thebasic content. In contrast, low ESD-users seem to benefit from conditions in which ESD canbe clearly identified.

Comparing the study outcomes of group I and IlIa with the results of group IIIb helps totest the impact of the interactivity mode . Analysis of variance reveals no significant maineffects nor interactions of the independent variables on study outcome.

Reading comprehension levels do not interact with the use of embedded support devices.Portier and van Buuren (this issue) also report on the impact of prior knowledge in

making use of ILE. They report that there is a significant interaction between PKS and ESD­use: one's level of prior knowledge influences the extent to which ESD are used . This effect issignificantly greater for processing ESD and testing ESD (Wilks 1\=.88, p<.05).

Extending the analysis to a comparison between students studying in ILE and PLE, theanalyses reveal no significant differences. Also a comparison between independent learning

192 R. MARTENS & M. VALCKE

situations and traditional face-to-face lectures docs not result in significant differences instudy outcomes.

Returning to our theoretical model, the results of this research stress again the impact ofindividual variables on learning in a specific environment and on the learning process. Priorknowledge levels influence the use of embedded support devices. Task environments that canbe adapted to the needs of students (discernability) fit into this picture and meet the demandsof specific groups of students .

Discussion

Combining the results of the research studies, we can conclude the following:

Embedded support devices are used to a very high extent by most students.

The use of ESD depends on individual variables, such as gender, educational back­ground and especially on the variable prior knowledge.

ESD influence the learning process and study outcomes.

There is an interacting influence of student aptitudes in the relation between the taskenvironment and learning outcomes.

Active manipulation of features of the task environment, such as the discernability ofparts of the learning materials are related to individual variables and the learningprocess.

We were not able to validate the hypothetical functions and effects of the embedded sup­port devices. The results of the interview-based study learn us that these functions and effectsseem to be different for different students and even depend on the phase in their learningprocess.

The question now arises whether or not our model has been validated? We may concludethat the impact of the task environment (with ESD) in the interrelation between individualvariables and the learning process has been validated. But, at a more important level, theresearch results also have shown us a clear direction for future research . The value of certainflexibility-features of interactive learning environments has been revealed. In this type ofenvironments students can shape, adapt, select/deselect learning materials in accordance totheir individual needs . This is in line with for instance de Diana and van der Heiden (1994):" ... an increasing need is felt for adaptable educational software products that can be modifiedto the need of users" (p. 116). ILE also provide many additional features such as the possibili­ty to make use of hypertext links (e.g., Picking, 1994).

References

Alesandrini , K. L. (1984) . Pictures and adult learning. Instructional Science. 13,63-77.

Ausubel, D. P. (1968) . Educational Psychology: a cognitive view. New York: Holt, Rinehart and Winston.

Barne s. B. R., & Clawson. E. U. (197 5). Do advanced organi zers facilitat e learning? Recommendations for further

research based on an analysis of 32 studies . Review ofEducational Research. 45. 637-659.

Bernard, R. M. (1990). Effects of processing instructions on the usefulness of a graphic organizer and structural cuing in

text. Instructional Science. 19. 207-217 .

Boekaerts, M., & Simons, P. R. J. (1993) . Leren en instructie. Psycholog ie Van de leerling en het leerproc es. Assen :

Dekker & van de Vegt.

Britton, B. K., & Gulgotz, S. (1991). Using Kintschs Computational model to improve instructional text: Effects of

repair ing inference calls on recall and cognitive structures. Journal of Educational Psych ology. 83, 329-345.

FUNCTIONS AND EFFECTS OF EMBEDDED SUPPORT DEVICES IN LEARNING MATERIALS 193

Clyde, A., & Crowther, H. (1983). How students use distance teaching materi als : an institutional study. Distance

Education. 4,4-26.

Como, L., & Snow, R. E. (1986). Adapting teaching to individual differences among learners. In Wittrock, M. C. (Ed .),

Handbook ofresearch on teaching (PI' . 605-629) . Third edition . London : Collier Macmillan Publishers.

Crombag, II. F. M. (1979) . ATl : Perhaps not such a good idea after all. Tijdschrift voor Onderwijsresearch, 4, 176·183 .

Cronbach, L. J., & Snow, R. E. (1977) . Aptitudes and instructional methods: a handbookfor research on interactions.

New York : Irvington .

Daal, M. M. (1991). De relatie tussen vooropleiding en studievoortgang van studenten van de Open universiteit . Een

verkennende analyse. Heerlen: Open universiteit , OlD .

De Bruijn, H. (1993) . Situated cognition in a computerized learning environment for adult basic education students.

Unpublish ed doctoral thesis. University Twente, The Netherlands .

De Diana, I. P. F., & Van der Heiden, G. (1994) . Electronic study books and learning style. Journal ofcomputer assisted

learning. to. 113-124.

De Jong , F. P. C. M., & Simons, R.-J. (1988). Self-regulation in text process ing. European Journal ofPsychology of

Education. 3. 177-190.

Dekeyser, 11 ., Goeminne, K., Schuyten, G., Martens, R. L., Portier, S. J., Valcke, M. M. A., & van Buuren, J. A. (in

print ). Quality of a learning environment for an applied statistics course of students and educational sciences

(UG-Report). Gent: University ofGent.

Doch y, F, J. R. C. (1992). Assessment ofprior knowledge as a determ inant for future learning. Utrecht/London:

LEMMA, Jessica Kingsley.

Dochy, F. J. R. C., & Valcke, M. M. A. (1992) . Validation ofknowledge profile dimensions: Lookingfor empirical evi­

dence (011C Research Report 33). Heerlen: Centre for Educational Technology and Innovation.

Driscoll , M. P., Moallem, M., Dick, W., & Kirby, E. (1994) . 110w does the textbook contribute to learning in a middle

school science class? Contemporary educational psychology. 19, 79- I00.

Elen, J. (1992). Toward prescriptions in instructional design: a theoretical and empirical approach. Unpubl ished doc­

toral thesis. Katholieke Universiteit Leuven, Belgium.

Hartley, J., & Davies , I. K. (1976) . Preinstructional strategie s: The role of pretests , behavioral object ives, overviews and

advanced organizers. Review ofEducational Research. 46. 239-265 .

Holmberg, B. (1986) . Improving study skills for distance students . Open Learning. November 1986.

Keeve s, J. P. (Ed .), (1988). Educational Research . methodology and measurement: an international handbook.

Oxford/New York: Pergamon Press.

Kember , D., & Cow, L. A. (1989) . model of student approaches to learning encompassing ways to influence and change

approaches. Instructional Science, 18. 263-288.

Kini, A. S. (1994) . Effects of cognitive style and verbal and visual presentation modes on concept learning in CBI.

Paper presented at the 1994 Annual Meeting ofthe American Educational Research Association. New Orlean s

July 23-25.

Kintsch, W. (1988) . The rol e of knowledge in d iscourse- comprehension : a construc ti on-i ntegration model.

Psychological Review, 95, 163-182 .

Kintsch , W., & 'Ian Dijk, T. A. (1978) . Towards a model of discourse comprehension and production . Psychological

Review. ss. 363-394 .

Lockwood. F. (11)92). Activities in self-instructional texts. London: Kogan Page.

Long, J. D., Gaynor, P., Erwin, A., & Williams , R. L. (1994). The relationship of self- management to academic moti­

vation, study efficiency, academic satisfaction, and grade point average among prospe ct ive education majors .

Psychology, 31, 12-22.

194 R. MARTENS & M. VALCKE

Lowijc k, J., & Simons, 1'. R. J. ( 1991). Inleidin g themanummer zelfinstructie: ecn cogniiief-psychologisch perspectief

Pedagogische studiec n, 68, 28 1-284.

Lohm an, D.F. (1993 ). Teachi ng and testing to develop fluid abili ties. Educationa l resea rcher, 22, 12-23.

Macdon ald-Ross, M. (1977 ). Graphics in texts. Review ofResearch in Education, 5, 49-85.

Marl and, P., Patching, W., Putt, I., & Putt, R. (1990) . Distance learn ers ' interact ions with text while studying. Distance

Educati on, I, 71- 91.

Marten s, R. L., & Valcke, M. M. A. (1993). Validation ofa theory abou t the functions and effec ts o f em bedd ed support

devices in written study materials. In M. Valc ke & F. Lockwood, (Eds. ), Research in relation to ne w develop­

ments in distance education materials (pp. 66-84). Hecrlen : Open univers ity.

Marte ns, R., Poelrnan s, P., Daal, M., van Staa, A., & van Meurs, C. (1993). Leren iO! verschillende leerstofge bieden:

consequenties van de dldactische uitwerking. Unpubl ished manuscript. Heerl en: Open universi teit.

Martens, R. L., Poelrn ans, P. H. A G., Daal, M. M., & Valcke, M. M. A. (1993). Varying embedded support devices in

a course: what is the effec t? (OTIC Research Report 44). Hecrlen : OTIC, Open university.

Martens, R. L., Poelrnans, P. H. A. G., Daal , M. M., Valcke, M. M. A , & Stolk, J. (19 94) . Eye movement regis tration

dur ing studying: an explorative study of the use of embedded support devices (OTI C Research Report 64) .

Heerlen: OTIC , Open unive rsity.

Marton , F., & Saljo , R. (1984) . Approaches to learnin g. In F. Marton , D. Hounsell & N. Entwistle (Eds .). The expe ri­

ence of learning (pp. 36-55). Edinburgh: Scottish Academic Press.

Mayer , R. E. (1979). Twenty yea rs of research on advance organi zers . Instructional Science. 8,133-167.

Meye r, J. H. F., & Watson, R. M. (1991 ). Evaluating the quality of student learning. II - study orchestration and the cur­

riculum. Studies in high er education, 16, 251-275.

Pask, G. (1976). Styles and strategies of learning. British Journal ofEducational Psychology, 46, 126-148.

Picking, R. ( 1994). A comparative study of computer-based document manipulation techn iques. Hypermedia. 6, 87-100.

Poel mans , P., & Jan ssen , J. (1994) . Panelonderzoek naar studie-uitval. Deell: Theoretische ana lyse van studie-uitval.

Heerlen: OlD, Open universiteit.

Rayner , K., & Pollatsek, A. (1989). The psychology ofreading. Englewood Cliffs, New Jersey: Prentice Hall.

Reigeluth, C. M., & Stein, F. S. (1983) . The elaboration theory of instru ction. In C.M. Reigeluth, (Ed .), Instructional­

design models : an overvie w of the current status. Hillsdal e, N. J.: Erlbau m.

Rost, D. H. (19 89). Reading comprehension: skill or skills? Journal ofResearc h in Reading. 12 (2) ,87-113.

Rothkopf, E. Z. ( 1970) . The concept of mathema genic activ ities. Review ofEducational Research. 40. 325-336 .

Schuemer, R. (1993). Some psycho logic al aspects ofdistance education . Hagen : ZI FF.

Simon s, P. R. J. (19 82). Welke leerlingen profiteren van het presenteren van analogieen bij leerstof? In E. de Corte

(Ed.), Onderzoek van onderwijsleerprocessen (SVO-reeks nr.5 3) . Harlin gen: Flevodruk.

Snow, R. E. (1992). Aptitude theor y: yesterday, today and tomorrow. Educational psychologist. 27. 5-32 .

Sternberg, R. J. (19 86). What is intelli gence ? Contemporary viewpoints on its nature and definition . Norwood, New

Jersey: ABLEX.

Sternberg, R. J. ( 1988). The triarchic mind. New York: Viking Penguin Inc.

Stolk, J., Smulders, M. A. M., & Belsmaeker, A. R. Th. (1991) . "BUKVANGER" eindrapportage tweede fase (SPIN

projectnummer 0710 .161). Heerlen: Open unive rsity.

Thijssen, J. G. L., & Span, P. ( 1985). Diff erentiatie-effccten in de schoolpraktijk : een ATl-onderzoek in veri eng de

brugklassen. Pedagogisch« studieen, 62, 358-371.

Tob ias (l994).lnterest. pr ior knowledge . and learn ing . CUNY: City College .

FUNCTIONS ANDEFFECTS OF EMBEDDED SUPPORT DEVICES IN LEARNING MATERIALS 195

Valcke , M. M. A ., Martens, R. L., Poelmans, P. H. A. G., & Daal, M. M. (1993 ). The actual use of embedded support

devices in self-study materials by students in a distan ce education setting. Distance education, 14, 55-84 .

Van Joolingen, W. R. (1993). Understanding andfacilitating discovery learn ing in computer-based simulation environ ­

ments. Unpublished docto ral thes is. Eindhoven University of technology, The Netherlands.

Van Oostendorp, II. , & Peeck, J. (1991). Leren uit teksten . In A. J. W. M. Thomassen, L. G. M. Noordman, & P. A. T.

M. Eling (Eds.), Lezen en begrijpen. De psychologie van het leesproces (pp. 153-172). AmsterdamlLisse : Swets

& Zeittinger.

Van Rij sw ijk , F., & Vermunt, J. (1987a) . Vaardig studeren: studie-aanpak adviespakket VOOr Ou-studenten . I .

lnventa ris (eerstijlen.Heerlen: Open university.

Van Rij swijk F.. & Vermunt, J. (1987b). Vaardig stude ren: studie-aanpak adviespakket VOOr Ou-student en . 2.

Handleidirg VOOr studenten. Heerlen : Open university.

Van Rij sw ijk , F., & Vermunt, J . ( 1987c ). Vaurdig s tuderen: studie -aanpak adviespakket VOOr Ou-studenten. 3.

Handleidirg voor consulenten en studiebegeleiders. Heerlen : Open university.

Van Staa, A., & van Meurs , C. (1992). Didactische hulpmiddelen en Ou-cursussen: meetproblem en, gebruik en

waardering. Heerlen: Open university, OlD .

Vermunt, J. D. H. M. (1986). Leerstijlen, concepties, orientaties en de Open universiteit (Eindrapport eerste fase project

" Diagnose Leestijlen OU") . Tilburg: KUB .

Vermunl, J. D. H. M. (1992) . Leerstijlen en het sturen van leerprocessen in het onderwijs. Naar procesgerichte instructie

en zelfstandig denken. AmsterdamlLisse: Swets & Zeitlinger.

Wade , S. E., & Thrathen, W. (1989 ). Effect ofself-selected study method s on learning. Journal of educational psy cholo ­

gy. 81. 40-H.

Waller, R. H. W. (1979). Typographic acce ss structures for educational texts . In P. A. Kolers, M. E. Wrolst ad , & H.

Bouma [Eds.), Processing of Visible Language (pp . 175.187). Ne w York : Plenum Press.

Willems , J. M. II. M. (1989). Sturen van leerprocessen met behulp van studietaken. In P. Span , E. De Corte, & B. van

Hout-Wolters (Eds .), Onderwijsleerpro cessen. Strategieen voor de verwerking van informatie (pp. 113-123).

Amsterdam! Lisse : Swets & Zeillinger B. V.

Winne , P. II. (19&3). Training students to process text with adjunct aids . Instructional Science, 12,243-266.

Key words: Instructional design, Interactive learning environment, Learning materials , Printedlearning materials, validation research .

Received: October 1994

Revision received: December 1994

Rob Martens. Open University, Centre for Educational Technology and Innovation, P.O. Box 2960,6401 DL Hee rle n, The Netherlands.

Current theme ofresearch:

Optimizing the embedded student support in printed and electronic self-study materi als.

196 R. MARTENS & M. VALCKE

Most relevant publications in the fie ld ofPsychology ofEducation:

Martens, R. L., Poelmans, P. II. A. G., Daal, M. M., & Valcke, M. M. A. (1993). Varying embedded support devices in

a course: what is the effect? (OTIC research report 44). lIeerlen: OTIC, Open university.

Martens, R. L., Poelmans, P. H. A. G., Daal, M. M., Valcke, M. M. A., & Stolk, J. (1994). Eye movement registration

during studying: an explorative study ofthe use ofembedded support devices (OTIC research repon 64). Heerlen :

OTIC, Open university.

Martens , R. L., Portier, S. J., Valcke, M. M. A., van Buuren, J. A., Dekeyser, H., Goeminne, & K. Schuyten, G. ( 1994).

The use ofembedded support devices in interactive learning environments: The impact ofstudent characteristics.

Heerlen: Open university.

Valcke, M. M. A., Martens, R. L., Poelmans, P. H. A. G, & Daal, M. M. ( 1993). The actual use of embedded support

devices in self-study materials by students in a distance education setting. Distance Education 14 ( I), 55-84.

Martin Valcke. Centre for Educat ional Technology and Innovation , Open University of The Netherlands,Valkenburgerweg 167, NL 6419 AT Heerlen, The Netherlands.

Current theme ofresearch:

Instructional design and evaluation of learning environments. The theoretical base of distance learning and the provisionof distance education learning materials.

Most relevant publications in the fi eld ofPsychology ofEducation:

Valcke, M., Martens, R., Poelmans, & Daal, M. ( 1993). The actual use of embedded support devices in se lf-study mate­

rials by student s in a distance education setting. Distance Education. 14 (1),55-84.

Kirschner, P., & Valcke, M. (1994). from supply driven to demand driven education: New conceptions and the role of

information technology therein. Computers in Human Services, 10 (4), 31-53.

Valcke , M., & Lockwood, F. (1993). Research in relat ion to new develop ments in distance education materials .

Heerlen: Open universiteit.

M. Valcke (1993). Knowledge representalions and the learning process: Taking account of developmental features and

support features in computer learning environments. In B. Denis (Ed.), Control Technology in Education (ASI

NATO Series, PI'. 13-27). BerlinHeidelberg: Springer Verlag.