Challenges in Defining,

Designing, and Measuring

“Digital Literacy” Development in

Learners:A Proposed Framework

AERA, PHL, April 2014Rebecca Reynolds, Assistant Professor

School of Communication & InformationDept. of Library and Information Science

Rutgers UniversityJohn Wolf, Ph.D., Instructor, NJIT

“Digital Literacy” in Naturalistic vs. Prescriptive ContextsNaturalistic Phenomena in General Population• Digital Divide, Levels 1 and 2 (Hargittai & DiMaggio, 2002; Barzilai-

Nahon, 2006)

• Important cultural, social and political activities occur in online environments, and participation in such activities requires digital skills and knowledge (Mossberger, Tolbert, and McNeal, 2007; Jenkins, 2009, Hobbs, 2010; Horrigan, 2011)

• More sophisticated forms of content creation, participatory engagement, digital knowledge and cultural benefits have been associated with privilege: higher socio-economic status and level of education (Pew, 2007; Hargittai and Walejko, 2008)

• Connects digital literacy to capacity for social mobility (Hargittai, 2010)

“Digital Literacy” in Naturalistic vs. Prescriptive ContextsPrescriptive Instructional Contexts• Strand 2 of the U.S. FCC’s NBP of 2010 focuses on

awareness raising and digital literacy efforts, esp. in schools• NETP of 2010: Optimizing learning and instructional design in

US schools with technology-integrated project / problem-based learning interventions.

• Prepare learners for 21st century work cultures• Role-taking, epistemic learning• Learning (engage & empower), Assessment (measure what

matters), Teaching (prepare & connect), Infrastructure (access & enable), Productivity (redesign & transform)

• AASL’s Standards for 21st Century Learner and ISTE National Education Technology Standards for students and teachers: Discuss student-level dispositions and latest versions bring in creativity and productivity

Definitions and Operationalizations Vary• Example 1, scholarship: Web-use skill

(Hargittai,2005): “a user’s ability to locate content on the web effectively and efficiently” with no definition offered in her updated work in 2009; operationalizes digital literacy as a user’s familiarity with terminology reflecting certain technology concepts (JPEG, frames, preference settings, newsgroups, PDF, refresh/ reload,

advanced search, weblog, bookmark, bookmarklet, spyware, bcc (on e-mail), blog, tagging, tabbed browsing, RSS, wiki, malware, social bookmarking, pod-casting, phishing, web feeds, firewall, cache, widget, favorites, torrent)

Definitions and Operationalizations Vary• Example 2, scholarship: Barron (2004) defines

technological fluency as learners’ ability “to express themselves creatively, to reformulate knowledge, and to synthesize new information. [The concept] entails a process of lifelong learning in which individuals continually apply what they know to adapt to change and acquire more knowledge to be more effective at applying technology to their work and personal lives.” Operationalization: students’ self -reported frequency of participation in the following twelve constructive activities

Definitions and Operationalizations VaryExample 3, scholarship: Eshet & Aviram• Skeptical hypothesis 2 centers on technology

situatedness of the definition one uses of digital literacy.• The definer must prioritize particular technology contexts of

the moment, and any definition reflects the norms and values of the definer in assigning important to given technologies

• “Clash of civilizations”

”Old, modern, rationalist, linear, conceptual, book-based culture of Western societies is CLASHING with new emerging post modern, branching, multi-media based reproduction oriented culture that has been developed in the last 20 years largely due to the spread of technology”

Definitions and Operationalizations VaryExample 1, policy: AASL Standards: “The definition of information literacy has become more complex as resources and technologies have changed. Information literacy has progressed from the simple definition of using reference resources to find information. Multiple literacies, including digital, visual, textual, and technological, have now joined information literacy as crucial skills for this century.”

Definitions and Operationalizations VaryExample 2, policy: NETS Standards

Definitions and Operationalizations VaryExample 3, scholarship: Eshet & Aviram• Conservative hypothesis in most studies rests on

assumption that digital literacy is a set of distinct skills that we can measure in the population.

• Theoretical questions (are skills distinct or are they really something else, like intelligence or motivation?)

• Operation-oriented questions (how to measure)• Didactic questions (can they be taught?)• Development and design questions (how are they best taught?)

• Skeptical hypothesis 1: No such thing as “digital literacy;” it is orthogonal with individual level capacities:

• Motivation / Curiosity, Learning styles, Creativity, Intelligences, Capacities, Personality types

• This hypothesis places the individual and his/her expertise at forefront

Critical Issues• Reynolds (2008): Research and policy definitions hold significant

limitations for advancement of research because of the definitional variation, technology determinedness of operationalizations resulting in a need for constant update, and in policy, limited structural, material means for realizing ends specified (e.g., time, curricula, budgets).

• Yet, given Digital Divide, and learning opportunities technology poses, the need for intervention clearly remains.

• To design, implement and study processes and effects of interventions that inculcate “digital literacy” in learners requires clearly defining learning objectives and outcomes for the intervention, and developing the appropriate pedagogy.

• Eshet & Aviram’s skeptical hypotheses advise (a) a focus on individual expertise, and (b) problematization of technology situatedness

Critical Issues• To take it 1 step further, a learning sciences approach to

designing digital literacy intervention requires theoretical grounding of the instructional design principles

This Study• The definition and framework for digital literacy that follows

emerged out of an existing learning innovation called Globaloria that applies social cognitive theory (e.g., Vygotsky) and Constructionism (e.g., Papert & Harel, 1991) in its design.

• Reynolds & Harel Caperton 2009 outline the theoretical linkages between G’s instructional design principles and Constructionism

• Reynolds & Hmelo-Silver (2013) identify conceptual linkages among Globaloria’s design, Constructionism and the knowledge building literature of Scardamalia & Bereiter.

• The definition of “Contemporary Learning Abilities” we propose is student centered, and clearly distinguishes between the abilities and the technology contexts out of which they may emerge.

Constructionist, knowledge-building conditions supporting development of 6-CLAs (Reynolds & Harel Caperton 2009; Reynolds & Hmelo-Silver, 2013)

This StudyBuilding evidence base for the six dimensions of the CLAs

Research Questions and Hypotheses

RQ1: What is the empirical relationship among the six proposed dimensions of contemporary learning?

RQ2: How does the introduction of Globaloria intervention in school affect frequency of engagement in CLA-based activities and practices in non-school settings?

H1: The intervention will have a larger effect on frequency of engagement for the more constructionist dimensions of contemporary learning.

INTERVENTION: Guided discovery-based game design program and curriculum offered by the World Wide

Workshop. MS, HS teachers and students gain experience and expertise in a range of agentive digital

practices.

Learning Management System as Information System

Learning Management System as Information System

MethodsNon-experimental pre/post survey design, 2011/2012 school year of Globaloria• Out of 1,356 middle and high school student participants, a

total of 1,063 completed the pre-survey (78.4%), and approximately 670 completed the post-survey (49.4%).

• Sample includes pre/post only students: 282 middle school and 781 high school students participated.

• The pre-intervention sample was 29.0% female and 71.0% male.

MethodsOperationalization of Digital Literacy:• Self-reported frequency of engagement in twenty different

Globaloria activities, mapped to the CLAs • Improvements: Definition and operationalization of digital

literacy are theoretically grounded in the social cognitivist, Constructionist and knowledge-building inspired instructional design features

• That is . . . The chosen technology activities included in our operationalization of digital literacy reflect the theory that inspired the instructional technology context’s design

About this, we are explicit.

Results• Factor analysis, reliability analysis indicate that the given survey

items for each factor hang together as 6 factors (note, all CLAs had 3 or more items, except the Publish factor (CLA 3) which had only 1 item).

• Bivariate correlational analyses indicate that at school, pre-program results indicated the strongest apparent correlations among the more constructionist dimensions of contemporary learning (CLAs 1-3)

• Repeated measures ANOVAs indicate that students increased in their frequency of engagement in all CLA dimensions @School from pre- to post-program except Surfing (CLA 6)

• Repeated measures ANOVAs indicate that students increased in their frequency of engagement in CLA dimensions 1,2, 3 and 5 (Create, Manage, Publish, Research) @Home from pre- to post-program

Regression Model Results• We ran multiple regression models investigating how

change in program engagement in all 6 CLA dimensions @School contributed to change in the 6 dimensions @Home. For instance:

• Does change in Creation @School contribute to change in Creation @Home?

• Does change in Research @School contribute to change in Creation @Home?

Regression Model Results

Regression Model Results

• Neither Gender nor proxy for SES (parent education) predicted change in engagement @Home (very interesting in light of digital divide research; Globaloria factors may mitigate digital divide factors; Reynolds & Chiu, 2013 in fact bear this out)

• Engagement @School led to an increase in engagement @Home, even though no homework.

Results• Factors predicting change in Create @ Home:

• Change in Create @ School• Change in Socialize @ School• Change in Surf @ School

• Factors predicting change in Manage @ Home:• Change in Manage @ School• Change in Socialize @ School• Change in Surf @ School

• Factors predicting change in Publish @ Home:• Change in Socialize @ School• Change in Surf @ School

• Factors predicting change in Socialize @ Home:• Change in Socialize @ School• Change in Surf @ School

• Factors predicting change in Research @ Home:• Change in Research @ School• Change in Surf @ School

• Factors predicting change in Surf @ Home:• Change in Research @ School• Change in Surf @ School

Discussion and Implications• Claim that the practices are contributing to each other in this

social cognitivist learning experience are supported by the findings

• Surfing, experimentation, play, along with social engagement in the context drive transfer of engagement to home (measure of motivation)

• For the more effortful, creative work (Creation, Publish), experiencing those practices @School is influencing students to engage at home, too.

• Contribution:• Study shows how when Digital Literacy expertise

domains, technology contexts and instructional design features are more clearly articulated and identified, it creates opportunities to explicate factors of influence

Discussion and Implications• This general learning objectives approach of digital literacy

conceptualization can support improvements in more pragmatic and executable educational policy recommendations, instructional design, measurement and assessment

Rebecca.reynolds@gmail.com

Rutgers University websitehttp://comminfo.rutgers.edu/directory/rbreynol/index.html

Thanks to IMLS!

Thanks to my partners!

Globaloria.orgWorldwideworkshop.org

Thank you!