ABSTRACT

Reuse of educational materials is integral to many educatortasks, from designing a course to preparing for a lab orclass. We are studying the reuse of educational materials inthe context of the Digital Library for Earth System Educa-tion (DLESE). DLESE is a community-owned and gov-erned facility offering high-quality teaching and learningresources for Earth system education. Our studies build ona cognitive framework that posits that people engage inthree cyclical processes—location, comprehension, andmodification—when reusing resources from large digitalrepositories. Our formative evaluations and cataloging ex-periences in DLESE suggest that the ‘findability’ and reus-ability of community-created digital educational resourcesis highly dependent on the presentational and structuraldesign of the resources themselves. Educational resourcedesigners often do not develop components with reuse inmind, making it more difficult or impossible for other edu-cators to find and use their material. We share the results ofthese studies to help educators create digital materials thatare more easily shared and used by others. For example, werecommend that all resources clearly state the creator’sname and contact information; relevant copyright restric-tions; the most significant date for the resource (specifyingcreation or revision); and the intended grade level.

Keywords: Education–Computer Assisted,Education–Geoscience

INTRODUCTION

The sharing and reuse of resources are activities deeplyembedded in the way earth system science educators worktoday. Reuse of educational materials is integral to manyeducator tasks, from designing a course to preparing for alab or class. With rapid developments in informationtechnology, particularly the World Wide Web, digitallibrary projects have emerged to provide access to a wideand rich variety of on-line educational components. Astraditional publishing methods are replaced by desktopand web publishing, the responsibility of preparing aresource for widespread use falls on digital libraries andresource creators working in partnership. In designingtheir resources for reuse creators can contribute to the

improvement of educator productivity and the quality ofeducation.

We are studying the reuse of educational materials inthe context of the Digital Library for Earth System Educa-tion. DLESE is a grassroots, community-led project toprovide searchable access to high-quality, onlineeducational resources for K-12 and undergraduate earthsystem science education (Marlino et al., 2001). Theseresources include objects such as maps, lesson plans, labexercises, data sets, virtual field trips, and interactivedemonstrations. The holdings of DLESE are created by awide variety of individual faculty members, agencies, andinstitutions. These resources are held (stored) on localservers and are accessed through the library via a databaseof searchable metadata records that describe them.

Based on our experiences with the DLESE project, weshare lessons learned in educational resource design inorder to help educators create digital materials that aremore easily reused by others. Reuse can improve educationby providing educators with a broader pool of qualityteaching resources. A goal of DLESE is to facilitate resourcesharing by providing well-designed access to suchresources, and encouraging creators to contribute theirwork to this effort.

A MODEL OF REUSE - THE LOCATION-COMPREHENSION-MODIFICATION CYCLE

In the context of DLESE, reuse refers to the process ofeducators incorporating existing resources held in thelibrary (images, tutorials, etc.) into their teachingprocess. Reuse can happen at many different levels ofgranularity, with an educator choosing to adopt anentire lab activity or a single diagram. Prior research(Fischer et al., 1991; Ye et al., 2000) suggests that whencreating new resources from existing resources, reuseinvolves three closely intertwined cognitive activities:location, comprehension, and modification.

Location - refers to the process of finding potentiallyuseful educational items for the classroom. This can bein the form of getting tips from a colleague, using adigital library, or searching the web.

Sumner et al. - Creating Reusable Educational Components 25

CREATING REUSABLE EDUCATIONAL COMPONENTS: LESSONS

FROM DLESE

Tamara Sumner Center for LifeLong Learning and Design, Campus Box 430, University of Colorado, Boulder

CO 80309, sumner@colorado.edu

Melissa Dawe Center for LifeLong Learning and Design, Campus Box 430, University of Colorado, Boulder

CO 80309, sumner@colorado.edu

Holly Devaul DLESE Program Center, University Corporation for Atmospheric Research, P.O. Box 3000,

Boulder, CO 80307, devaul@ucar.edu

Comprehension - involves not only making relevancejudgments, but also understanding the function,structure, and context for use of the resource. It is crucialthat educators are able to rapidly assess whether aresource can be used in a specific classroom setting, orfor a particular group of learners.

Modification - can take many forms, from editingcontent, to partitioning more complex resources intosimpler parts for easier use. Resources often cannot beused as-is, but instead need to be tailored to better fit thenew context of use (Spohrer et al., 1998; Roschelle et al.,1999).

These three processes form a cycle, in that newresources are created that can then be shared and reused.DLESE is designed to facilitate this cycle by offering astructured search environment with supportinginformation to enhance both location andcomprehension. We include several key metadatacategories, such as keyword, grade level, and resourcetype (e.g. lesson plan, lab activity), as part of thediscovery system of the library. Users can search orbrowse by these categories to locate resources andconstruct a general or more specific query (Figure 1). Thesearch results contain additional information thatcharacterizes the resources, including the URL and ahuman-crafted description. Feedback from DLESE userssuggests that the metadata description plays animportant role in resource comprehension by providinga concise overview of a potentially large and complexresource (Sumner and Dawe, 2001).

Our formative evaluations and catalogingexperiences show that the reusability of digitaleducational resources is highly dependent on both: (1)the design of the digital library’s discovery system, and(2) the design of the resources themselves. Here we donot refer to pedagogical design, but rather topresentational and structural aspects of the resource,including, but not limited to, text-based content. Wehave observed that educational resource designers oftendo not develop components with reuse in mind, makingit difficult or impossible for other educators to find anduse their material, and also more difficult for digitallibraries to catalog and accession them. In this article, wedescribe our research approach, and present findingsand recommendations for resource creators pertinent tothe presentational and structural aspects of resourcedesign. A discussion of the implications of these studiesfor the design of the library’s resource discovery systemhas been presented elsewhere (Sumner and Dawe, 2001).

RESEARCH APPROACH

The DLESE project is committed to engaging in acommunity-centered design process that encouragesbroad-based participation in design and is focused onaddressing concrete user needs. Towards this end, wehave adopted a task-centered design methodologyconsistent with best practices in designing highlyinteractive systems. The central tenets of thismethodology are: early and continual involvement ofusers in the design process, a focus on real users andtheir tasks, and iterative design guided by frequentformative evaluations systems (Lewis and Rieman, 1993;Gould et al., 1991; Nielsen, 2000). Table 1 shows theformative studies conducted as part of this overallmethodology to help us understand community needsand to evaluate the evolving library interface.

Semi-structured interviews were conducted withtwo K-12 instructors, five university faculty, and twoundergraduate earth science students. Each interviewlasted between 60 and 90 minutes and was taped andtranscribed. In semi-structured interviewing, a set ofopen-ended questions is used to guide the overalldiscussion but not to dictate the order in which topicsare covered (Yin, 1984).

The interviews focused on understanding howparticipants currently located and selected digitaleducational resources on the World Wide Web usingexisting search engines or other means. The questionswere designed to have participants reflect on recentcritical incidents, such as remembering the last time theyused a digital resource in their teaching and talkingabout how they located the resource, thedecision-making process of its selection, andconsidering what changes, if any, were made to their

26 Journal of Geoscience Education, v. 50, n. 1, January, 2002, p.25-30

Formative Study Participants

Semi-structured Interviews 7 teachers,

2 students

Think-aloud Obeservations:

Round 15 participants

Think-aloud Observations:

Round 210 participants

Rubric Evaluation 22 participants

Resource Analysis 140 resources

Table 1. Formative studies conducted

Sumner et al. - Creating Reusable Educational Components 27

teaching practices or other course products as a result.Interviews were conducted in the participants’ place ofwork or study and, where possible, we asked to seeconcrete demonstrations of these digital resources orwalked through re-creations of recently conductedsearches. Prior to the interview, participants were askedto fill out a questionnaire about their current use ofInternet and web technologies. The interview transcriptswere analyzed to look for consistent patterns of behaviorand decision-making when engaging in resourcelocation, comprehension, and modification activities.The results from these interviews contributed to theinitial requirements for the first DLESE prototype.

Further studies involved two rounds of usabilitytesting with K-16 earth science educators, one roundwith five participants and one with ten. Each participantwas asked to “think aloud” as he or she performed aseries of class and course preparation tasks using thecurrent DLESE prototype. Detailed observational noteswere independently taken by two observers andcompared after each session to note recurring usability

problems or misconceptions arising from the libraryinterface. After each round of testing, the results wereused to guide immediate changes to the current DLESEprototype and to formulate new design requirements forthe next generation of the library.

A rubric evaluation exercise was conducted with 22participants, 19 K-12 teachers and 3 pre-service teachers.The participants were divided into three groups of 7-8people, with one pre-service teacher per group. Eachgroup spent 90 minutes using, discussing, and analyzingthe current DLESE prototype according to a preliminaryPortal Evaluation Rubric prepared by the Long BeachUnified School District. The rubric asked participants toevaluate the system along a variety of dimensions,including the usefulness and usability of the discoveryinterface, the accuracy of the search results, and thequality of the library contents. The observation andanalysis process for the rubric evaluation was similar tothat used in the think-aloud rounds.

Finally, a preliminary cataloging exercise wasconducted as part of designing the library’s metadata

Figure 1. DLESE Discovery system. Using the basic search interface shown in the upper left, a teacher could

search for 9-12 grade level resources about hydrology. As shown in the bottom window, 114

resources matched this query. Each matching resource is presented in a brief display with

information describing its content and location. The teacher could narrow these results using the

advanced search interface shown in the upper right. Selecting “Tutorial” narrows the results to

twelve grade-level appropriate tutorial-based resources about hydrology.

framework and developing the guidelines for catalogingbest practices; i.e., formulating the steps a person shouldtake to produce a consistent and coherent resourcemetadata description. A testbed collection wasassembled with 140 resources that spanned both theearth system science disciplines and a wide variety ofeducational resource types. The process of analyzingand cataloging items in this collection providedimportant insights into the challenges of discovering thedescriptive information necessary for complete andaccurate metadata to be included in library records. Thisinformation is crucial for effective resource location andcomprehension in both digital library systems and incommercial search engine systems. Given thenon-standard design of web-based resources, thisnecessary information is often not evident, or notpresented in a way that a human cataloger or anautomatic search engine can tap into it.

When considered together, the results from thesefive qualitative studies provide a convergent account ofthe major challenges educators face in locating andusing digital components in their classrooms. Thisaccount has aided us in developing metadatadescriptors for the library, and has resulted inrecommendations for resource creators who wish tofurther assist educators in the location, comprehensionand reuse of their materials. These recommendations arerelevant not only within the context of DLESE, but forgeneralized web searching as well. We believe thatfollowing these recommendations can: (1) enhancegeoscience education by providing teachers andstudents with improved access to high-quality, onlineeducational resources, and (2) help resource creatorsmake the fruits of their creativity and intellect morewidely accessible and available.

FINDINGS AND RECOMMENDATIONS FOREACH STEP IN THE CYCLE

Location - The first step in locating a resource isdeciding where to look. Through our interviews anduser testing, we found that educators primarily rely onpersonal interaction to locate new material: they askcolleagues, exchange ideas and materials at workshopsand conferences, or consult journals. A crucial aspect ofthis process is the reliance on a trusted source forresource location and evaluation. Educators appear toconsider the source of the material (either the individualcreator, or an institution) as a primary factor in assessingits quality and effectiveness. We observed that thisaffects how educators search the web. Rather than usingsearch engines, they prefer to go to sites that theyalready know and trust. Examples of trusted sitesinclude colleagues’ personal home pages, or knownorganizations’ websites, like NASA, or the USGS.

Recommendations for Designers:

Include contact information - Surprisingly, creator orauthor information can be very hard to find inweb-based resources. Often this information is locatedon the creator’s homepage or elsewhere, but is notavailable from the resource directly, making it verydifficult to find when reached through the digitallibrary. It also makes cataloging of the resource moredifficult, as the cataloger must search for thisinformation. Providing brief contact information, suchas the creator, institution, and an email address directlywithin the resource helps others locate and evaluateyour resource. Ideally, this information is on a separatepage linked to the main resource page, to facilitate reusewith a minimum of modification needs.

Create metadata for the resource - By metadata, wemean information about the resource; this includes thetitle, description, intended use and grade level.Metadata can be added to a resource in a number ofways. Resource creators can generate rich metadata fortheir resource using the DLESE cataloging tool, found athttp://www.dlese.org (select ‘Contribute a resource’,then ‘Catalog a resource’ from the left-hand navigationmenu). This tool allows you to craft a description thataccurately reflects the content of your work, choose anappropriate resource type and grade level assignment,and ensure that your contact information is correctlyrecorded. Submitting a record to the library means thatusers will be able to find your resource more efficiently,and will better understand its use when they do.Contributing to the library is an easy way for creators tofacilitate reuse for their colleagues.

It is also extremely helpful to potential resourceusers to have some metadata embedded directly in thetext of the resource. Having a brief description of thelearning goals, intended use, and grade level readilyvisible makes it much easier for a potential user to judgewhether or not a resource is appropriate for theirpurposes. Embedding this textual overview informationinto a web-based resource will also help a resource to bebetter indexed by web crawlers or search engines sincemany of these systems rely on the location andfrequency of keywords in the HTML text to indexresources (Sullivan, 2001). Including this information isespecially important for non-textual resources, likesimulations or images in which captions may exist onlyas part of the image itself, and hence are unavailable tosearch engines or library discovery systems.

A popular way to embed simple metadata inweb-based resources is through metatags, which areHTML elements that can be placed in the header of anHTML document. Metatags contain information such as

28 Journal of Geoscience Education, v. 50, n. 1, January, 2002, p.25-30

Sumner et al. - Creating Reusable Educational Components 29

the author’s name, keywords, and description. Metatagsallow authors to specify the page title that appears atop abrowser window; they also facilitate discovery by somesearch engines. Technology is currently beingdeveloped that could potentially create a metadatarecord automatically from these metatags such that theresource could be accessioned into a number of digitalrepositories. A resource creators’ guide on the DLESEwebsite provides more detailed information(http://www.dlese.org/Metadata).

Comprehension - Comprehension of a resourceincludes evaluating it for use, and recognizing whatmodifications need to be made. An importantobservation is that comprehension is a distributedprocess, involving both library-provided informationabout an educational resource and information withinthe resource itself. In both the think-aloud studies andthe rubric evaluation, participants were observed tomake extensive use of the descriptive metadata recordsas comprehension aids, but then struggled to find thepiece of interest because the resource was not as clearlystructured as the metadata.

Recommendations for Designers:

Include comprehension aids in your resource -Difficulty in comprehension at the resource level canoften be viewed as a ‘packaging’ problem (Bannon andBodker, 1997). Many faculty are placing resources on theweb so that they can be used by students (notably theirown), but they are not taking the extra step to do the‘packaging’; i.e., to prepare the useful supplementarymaterials such as tables of contents, indexes, summaries,and instructor’s guides that make complex resources,such as courses, reusable by other faculty. Even simpleritems like interactive applets do not commonly includesyllabi or lesson plans demonstrating how it might beused in a course (Marion and Hacking, 1998).Traditionally, these ancillary materials are developedand integrated into the resource as part of the formalpublishing process (Marion and Hacking, 1998;Bondaryk, 1998). Without them, the cost ofcomprehension can be prohibitively high, resulting in afailure of reuse. A paragraph providing an overview ofthe intended audience (grade level) and a site map ortable of contents of the concepts contained therein is anextremely useful way for a user to quickly determine ifthe resource is relevant to their goals. If the resource hassignificant scientific material, include the date ofcreation or revision and the source of the data orstatistics to assist potential users in assessing howup-to-date and reliable it is, (e.g., if a resource is older,one may want to inspect it more closely for scientificaccuracy). Adding this information can assist a felloweducator in the decision whether to use the material; it

also makes cataloging a resource more accurate andswift.

Modification - Modification is the process of adaptingan existing component for use in a specific teachingcontext. Our studies, and experiences of othereducational component libraries such as the EducationalObject Economy project (http://www.eoe.org), suggestthat many educational resources, even simple textualresources, need to be modified in some way before theycan be used (Spohrer et al., 1998). However it is also clearthat modification is difficult, especially for non-textualresources. When designing a resource, there are simplethings you can do to reduce the work of modification.

Recommendations for Designers:

Don’t embed overly-specific information in yourresource - We have found that many resources containcontext-specific information, such as the name and officeof the professor of a course. Consider separating thecontextual and time-dependent information (thesemester it is used, the instructor, etc.) in a separatedocument that is linked to from the main resource. Themain content of the resource can then be reused moreeasily in another setting without major modifications. Itmay also be easier for the creator to locate and updatethis contextual information as needed.

Provide copyright information - We’ve also found thateducators (like most of us) do not have a lot ofexperience with nor want to think about copyrightrestrictions. If your resource has copyright restrictions,stating them explicitly and clearly will protect the rightsof the creators, and will also guide others in theappropriate use and modification of the material. Ifthere are no restrictions, state that clearly as well.

Modularize your resource into independentcomponents - Consider organizing your resource insuch a way that smaller components can be extractedand used independently from the larger piece. Forexample, if you have developed a useful Java applet, orhave an illustrative image of a geological eventembedded within a larger resource, consider designingthose items as separable components. Our studies showthat due to individual teaching styles and non-standardcourse content, educators are more likely to utilize smallpieces, like photos and lesson plans, and incorporatethem into the existing framework of their curriculum,than to use an entire course.

CONCLUSION

We have presented a series of formative studies guidedby the task-centered design methodology, the results of

which have contributed to our understanding of howeducators and students engage in the Location -Comprehension - Modification cycle when searching forappropriate, reusable learning materials. We have alsoidentified several ways that resource creators can easilyaid in the dissemination of their on-line resources. Theserecommendations address structural design and contentissues that are simple to include and can have significantimpact on making resources accessible and useable. Wehope that the promise of a richer collection of reusableeducational components will be an incentive foreducators to create resources that facilitate reuse.

ACKNOWLEDGEMENT

The research described here was funded underCooperative Agreement #ATM-9732665 between theNational Science Foundation (NSF) and the UniversityCorporation for Atmospheric Research (UCAR), andGrants #9978338 and #0085600. The views expressedherein are those of the authors and do not necessarilyreflect the views of NSF, its sub-agencies, or UCAR.

REFERENCES

Bannon, L. and Bødker, S., 1997, Constructing commoninformation spaces. In Hughes, J. ed., FifthEuropean Conference on Computer SupportedCooperative Work, (Sep 7-11, Lancaster, UK),Kluwer Academic Publishers, The Netherlands, p.81-96.

Bondaryk, L., 1998, Publishing new media in highereducation: Overcoming the adoption hurdle.Journal of Interactive Media in Education, v. 98, no.3, (9/15/01)

Devaul, H., Ginger, K., and Kelly, K., 2001, Resourcecreator information page, http://www.dlese.org/Metadata/index.htm. (9/15/01)

Fischer, G., Henninger, S.R., and Redmiles, D.F., 1991,Intertwining query construction and relevanceevaluation, in Human Factors in ComputingSystems, CHI’91 Conference Proceedings, NewYork. p. 55-62. (Available in the Association forComputing Machinery Digital Library).

Gould, J.D., Boies, S.J., and Lewis, C., 1991, Makingusable, useful, productivity-enhancing computerapplications. Communications of the ACM, v. 34, p.74-85.

Lewis, C.H. and Rieman, J., 1993, Task-centered userinterface design: A practical introduction, http://home.att.net/~jrieman/jrtcdbk.html. (10/06/01)

Marion, A. and Hacking, E., 1998, Educationalpublishing and the World Wide Web. Journal ofInteractive Media in Education, v. 98, http://www-jime.open.ac.uk/98/2. (10/06/01)

Marlin, M.T., Sumner, T., Fulker, D., Manduca, C.A., andMogk, D., 2001, The Digital Library for Earth SystemEducation: Building Community, Building theLibrary, Communications of the ACM, vol. 44, p.80-81.

Nielsen, J., 2000, Designing web usability, Indianapolis,IN, New Riders Publishing.

Roschelle, J.C., DiGiano, C. Koutlis, M., Repenning, A.,Phillips, J., Jackiw, N., and Suthers, D., 1999,Developing educational software components, IEEEComputer, v. 32, p. 50-58.

Spohrer, J., Sumner, T.R., and Buckingham Shum, S,1998, Educational authoring tools and theeducational orbit economy: Introducation to thisSpecial Issue from the East/West Group, Journal ofInteractive Media in Education, v. 98,http://www-jime.open.ac.uk/98/10. (10/10/01)

Sullivan, D. 2001, http://www.searchenginewatch.com.

Sumner, T.R. and Dawe, M., 2001, Looking at digitallibrary usability from a reuse perspective. InACM/IEEE Joint Conference on Digital LibrariesProceedings (JCDL ‘01), p. 416-425.

Ye, Yunwen, Fischer, Gerhard, Reeves, Brent, 2000,Integrating active information delivery and reuserepository systems. In Eighth InternationalSymposium on Foundations of SoftwareEngineering for Twenty-first Century Applications,ACM Press, p. 60-68. (Available in the Associationfor Computing Machinery Digital Library).

Yin, R.,1984, Case study research: Design and methods,SAGE Publications, Newbury Park, CA.

26 Journal of Geoscience Education, v. 50, n. 1, January, 2002, p.25-30