Post on 15-Jul-2015
Patterns and Outcomes of Youth Engagement in
Collaborative Information Seeking with Varied
Resources During Guided Discovery-Based
Learning.
UT-Austin Guest LectureRebecca Reynolds, Assistant Professor
School of Communication & InformationLibrary and Information Science
Rutgers University
• Theoretical debates in the learning and information
sciences around learning, structure and agency
• Current landscape of educational technology at this
present historical moment (map technological terrain;
proliferation into K-12, without research evidence).
• Research findings from Globaloria, a game design learning
innovation that includes pilot locations in East Austin, TX
• Intersections of learning sciences, information science
theory in light of findings, and questions invited.
Agenda for Today’s Talk
Socio-technical systems / social informatics perspectives and
technology determinism (understanding reciprocal interrelationships
between humans and machines in both design and application; shaping
both the technical and the social conditions of work, such that efficiency
and humanity do not contradict each other)
Learning sciences, social constructivism, Constructionism,
design-based research (LS developed as a reaction against the
random control trial as the gold standard research methodology in
evaluation and ed policymaking, and the standardized testing paradigm
as the deterministic driver for curriculum development instead of vice
versa)
Both share roots in social-constructivist theory; both foster
more human / community-centered design of innovation
Parallels Between Socio-technical Systems Perspectives and the Goals of the “Learning Sciences”
To the extent that “constructivist” teaching pedagogies
facilitate exploration and “discovery-based” learning…
Along come Kirschner, Sweller & Clark (2006): Why Minimal Guidance During Instruction Does Not Work: An Analysis of
the Failure of Constructivist, Discovery, Problem-Based, Experiential, and
Inquiry-Based Teaching (Cited by 2759)
• Staunch objection to constructivist learning on basis of
cognitive load research
• If educators / designers have a learning objective in a
knowledge domain they are trying to meet, asking learners
to search for their own resources and make sense of the
information they find is de-motivating and frustrating;
detracts from learning, rather than contributing.
Setting Up a Straw Man Argument Against Constructivism in Education . . . . The Issue of Cognitive Load
• Objection criticized for over-simplifying “constructivism”
and minimizing role of intervention design, scaffolding and
guidance present in such interventions (e.g., Hmelo-Silver,
Duncan, Chinn, 2007, cited by 808)
• Rich research evidence for effectiveness of PBL, IPjBL &
IBL…(e.g., scholars such as Hmelo-Silver, Martin, Kapur & Kinzer,
Blumenfeld, Eccles, Kuhlthau, Eisenberg, Chu)
• We also have a history of positive results in Constructionist
contexts regarding student computational thinking,
engagement, affect, meta-cognition outcomes (e.g., Harel &
Papert, 1991; Harel, 1991; Kafai, 1995; Bruckman & Resnick, 1995; Kafai &
Resnick, 1996; Urrea, 2001, 2002; Cavallo, 2004; Kafai & Ching, 2004; Kafai,
2006; Peppler, Kafai & Chiu, 2007; Klopfer, 2008; Reynolds, 2008).
“Constructivist” learning interventions come in many shapes
and sizes. . .
Counter-arguments to Kirschner, Sweller, Clark
What About Learning in the Context of Information Seeking?
Inquiry-based learning “in the wild” tends to be less-structured, occurring more so in informal, learner-driven information search contexts without a human guide (unless one is lucky enough to have a personal librarian!).
While learning during information seeking is often the underlying goal of search (explicit or implicit), if we are to believe Kirschner, Sweller & Clark,
• Open ended, unstructured exploration and resource use without an expert guide is not optimal way to learn things
• If someone really wants to learn material, better suited to do so in guided, mentored, scaffolded, problem-based scenarios designed by experts. . . (schools/teachers)
• Informal/Formal . . . Extent of structure are factors
Any information scientist will advocate there is an appropriate role for independent research in the knowledge creation process
Key Factor in Success: Learner Expertise (Novices vs. Experts)
Educational Technologies as Information Systems. . . Mapping the Terrain
• While information search systems tend to be more generic, archival, multi-use, targeted towards experts….
• Educational technologies as information environments tend to be more specific to formal learning settings involving lessons and discrete learning objectives, and users with lesser extents of expertise/domain knowledge (e.g., youth).
What comes to mind?
Hardware (Computers/laptops/tablets/mobiles; Smartboards; robotics; other devices)
Software and web services:
MS Office and related multimedia productivity tools
Learning management systems
Games
Simulations
Mobile apps
Game design platforms (Unity; Flash; HTML5, etc.)
Educational Technologies as Information Systems. . .
Some software and web services (LMS’s, games, sims) have been designed specifically to teach a particular knowledge domain. . . The learning objective drives the design; design is emergent from the learning objective.
Others are more generic . . . Multi-purpose . . . Educator populates the content [e.g., course shells]
Still others provide “canned” curriculum . . . Publishing company products distributing existing print material e.g., text book content, in electronic format (not much of a departure in pedagogy per se; transformation of print to digital content, with multiple choice tests added on to the end).
Contexts of E-Learning Implementation: Varying… informal (e.g., after-school) vs. formal (during school time, for credit and a grade); includes line distance education, blended e-learning, fusion environments in which instructor leads in FTF classes, leveraging online environments/experiences/content. . . Role of educator varies across.
A few quick examples…
Games designed around key learning objectives (e.g., GLS games)• .
Simulations in the school subject domains. . .
• .
Learning Management System Course Shells for K-12
Generic Course Management
Generic Course Resource Sharing
Pre-Packaged, Standards-Driven Curriculum Content from Existing Publishers, Delivered in Increasingly Complex Modalities and Online Distribution Channel Models for K-12
Research Paradigms, Challenges
IES: Effects
data on research effectiveness
WhatWorks:
“Blended learning”
12 results
Clearinghouses: So useful in theory,so challenging toSUSTAIN
Research Paradigms, Challenges
Learning sciences scholarship: Smaller-N studies
• While yielding many necessary innovations, SO FAR most of these smaller N programs featuring “ideal” features don’t scale.
More generically designed LMS Solutions:
• Scaling, but relatively little research evidence base for effectiveness
• Schools are diving in, making do, based on sales claims.
The diffusion of K–12 blended learning is far outpacing the availability of useful research
So Much Variation. . .
How do we know what is effective?
• Considering Kirschner et al’s critiques Design innovations for specific knowledge domains that are refined closely to meet explicit learning objectives would be higher quality.
• And yet. . . LMS shell systems proliferate, and perhaps… a lowest common denominator of innovation.
• To what extent do less-guided systems support learning?
• Role of research . . . .
My Research Focus:
Issues of structure and agency in the context of educational technology design
and learner engagement.
Globaloria has served as a rich and complex test case in which to investigate these questions. . .
Where does Globaloria fit in the theoretical / design features landscape?
Design-based research-driven learning innovation developed by PhD researchers with many iterations of R & D. “Guided Discovery” . . .
Learning objectives: Introductory CS principles, digital literacy, 6-CLAs, through game design for MS and HS students [not in US core curriculum]
Pragmatic, semi-structured: Many teachers are not yet domain experts in CS/digital literacy, but the need for teaching children in this NOW is clear. . .[industry job market; live/livelihood skills Scholarship on CS Education, computational thinking]
Globaloria LMS has pre-populated game design tutorials, innovations in content delivery, scope & sequencing (rather than mere digitization of existing print textbook materials). Theoretically driven by Constructionism.
Guide AND autonomy-supportive: Electives; teachers w varying levels of expertise; does require self-driven learning and discovery but within a closed system that offers many rich resources in an organized sequence
INTERVENTION: Guided discovery-based game design program and curriculum offered by the World Wide Workshop. MS, HS
teachers and students gain experience and expertise in a range of agentive digital practices.
Domains of Learning and Expertise
• Constructionist digital literacy (skills needed in knowledge economy => 6-CLAs)
• Computational thinking through game design in Flash and programming in Actionscript
• Core curricular subject matter:
o When game subjects are linked to core curriculum and students deepen knowledge about topic through online research and design
• STEM career interests: Technology & Engineering; Computer Science
• Motivation, Affect, Attitudes, Life Choices, New Possibilities and Horizons
E-Learning Environment as Information System
Inquiry, Collaboration, Creation during a Game Design Course
My research investigates MS and HS student engagement in collaborative information-seeking behavior, within a pilot game design learning program involving a wiki-based LMS as a “coordinating representation” and productive social media platform.
Design affordances / constraints of the environment
Student inquiry and collaborative processes
Learning outcomes: successes, struggles / challenges
• This testbed environment and program is richly and deeply integrated into the schools in which it is being piloted (students engage daily, for credit and a grade, for a full year +)
• Findings may be generalizable to wider phenomena of shell LMS proliferation.... Which are much *less* structured, technologically.
Broad, Top-Level Research Question
Is Globaloria the best way of teaching these domains of knowledge to middle schoolers and high schoolers?
Not many existing programs yet. . . Against what can we compare?
Flexibility/Adaptability of curriculum = Some non-uniformity
Learn what we can about this implementation, share, compare when possible
Existing Effects findings
Globaloria participation increases science, social studies, and reading WESTEST standardized test scores, compared with matched case non-participants in West Virginia
Quasi-experimental research with match-case controls [Chadwick & Gore (2010) , Chadwick & Gore (2011) , Ho, Gore & Chadwick (2012) , Ho, Gore & Chadwick (2013)
Globaloria participation increases student engagement and self-efficacy in the “6 Contemporary Learning Abilities” (a framework of 6 dimensions of digital expertise specified in Reynolds & Harel Caperton (2009), Harel Caperton (2010), and tested in non-experimental pre/post design by Reynolds (2011, 2013).
Globaloria attenuates known Digital Divide effects including gender, socio-economic status, and some race categories given sample (Reynolds & Chiu,
forthcoming, JASIST)
Existing Effects findings
Conditions among factors that influence the learning, as measured quantitatively, thus far:
Intrinsic motivational disposition among students predicts successful knowledge outcomes in Globaloria (Reynolds & Chiu, 2013, ICLS)
Self-reported uses of the learning management system features in a survey (Reynolds & Baik, 2013, ASIST)
Current Study: By What Mechanisms? Globaloria
Student Learning Processes and Information Uses
DBR research agenda with multiple strands of data collection
underway (survey, observational video cases, screen capture, trace data,
content analysis of student artifacts and communications, LMS features and
natural experiments).
THIS STUDY’S RQs:(1) What tasks are central to students’ participation in this social
constructivist context?
(2) How are students engaging in inquiry and using resources to address
emergent, in situ questions and solve problems?
(3) Which resources appear to help most?
(4) In what ways might these findings contribute to learning sciences
debates on social constructivism/critiques around structure?
(5) In what ways might these findings contribute to information science
theory-building on guided inquiry, Task, CIS, learning outcomes and
sense-making?
Methods, Dataset 1: (See Slides 46-47 for details)
Google Analytics page read data for a sample of students who
participated in the Globaloria Game Design Program, during the
2012/2013 school year. NOTE: Google Analytics data available at SCHOOL level only. [Limitations]
Methods, Dataset 2:
During Spring semester 2013 (a week in March and a week in
May), we followed 2 teams per grade in grades 6-8 during site
visits at one school in AUSTIN, TX, collected observational
video data, for a total of 6 team case studies. Filmed,
qualitatively coded, categorized, analyzed, cross-tabbed the
frequencies.
Methods, Dataset 3:
Content analysis of student game quality, ranked by team,
school (inter-coder reliable analysis across 4 dimensions)
Profile
Project
Team
Google Analytics LMS Pages Reported in Findings: Curriculum (Intro, Wiki Tools, Units 1-4, Actionscript Tutorials); Social Media Pages (Profile,Project, Team)
Results, Dataset 1: Descriptive data on resource uses
[in the aggregate – all 21 schools]
Social media pages:
• Students appear to engage more so with their profile pages
than the more productivity-oriented project pages or team
pages.
• Schools varied substantially in their extent of use of these
pages.
Curriculum unit pages:
• Students appear to engage more so with earlier curriculum
topics (Intro, Wiki Tools, Unit 1, Unit 2) than later (Units 3, 4,
Actionscript Tutorials).
• A small number of schools spike in their pvs to latter curricular
topics later in the school year
Standardized PVs for Schools with Higher Avg. Game
Evaluation Scores
Standardized PVs for Schools with Lower Avg. Game
Evaluation Scores
Standardized PVs for Schools with a 0 Game Evaluation
Score (No final games to evaluate)
Results, Dataset 1: Apparent relationships between
resource uses and outcomes
• Schools with lower game design quality averages and
schools creating 0 games appear to visit profile pages more
frequently than project and team pages, whereas,
• Schools with higher game design quality averages appear to
visit the team pages to a greater extent, and also more so as
the program proceeds over time.
• Curriculum PVs: Higher performing schools also appear to
view Unit 2, Unit 3, Unit 4 and Actionscript curriculum more so
than those with lower game evaluation scores, and utilize
latter units in the sequence increasingly as the school year
progresses. Engaging in teamwork with the wiki and using its resources seem to
contribute to learning. But how are students using these resources? [lots of
hidden phenomena in school-level aggregates]
Results, Dataset 2: Variables of interest
• Task: What was student engaged in?
• Collaborative Information Seeking Modality:
Permutations of collaborative modality and information
source
• Inquiry incident outcomes: Was each incident complete
and resolved, or did it remain in limbo?
Results,
Dataset 2: Descriptive
Findings, Task, CIS
Modality, Outcome
TABLE1.CodingschemeandNofincidentsforemergentcollaborativeinformationseeking(CIS)variablesTaskDrivingInquiry CISModality:Solo,team,tchr,byresource Outcome
Graphicdesignanddrawingofdigitalimages
93
Teammateshelpingeachotherwiththeirownexpertise
109
Successfulsolutionfound
151
Advancedprogrammingfunctions
62 SoloinquiryusingwiderInternet 38 Taskstillinprogress 103
Basicprogrammingfunctions
52 Teacherhelpingstudentwiththeirownexpertise
36 Nosolutionfound,problemleftinlimbo
33
Gamesubjectdevelopment
39 Teammateshelpingeachotherwithwiderinternet
34 Notsure/researcherunabletodiscern
14
Versioncontrolandfilemgmt
15 SoloinquiryusingwikiLMS 31 Nosolutionfound,taskabandoned
8
Howtosearch 12 Teammateshelpingeachotherusingwiki 25 Problemsolutionfoundandtestedbutfailed
6
Characteranimation 10 Helpseekingfromclassmateswiththeirownexpertise
14 Problemsolutionfoundbutrejected
5
Music 10 Teacherinitiateshelpforstudent 13
Notsure 10 Teacherhelpingstudentusingwiki 11
Sharingofformalfeedback
8 Teacherhelpingstudentwithwiderinternet
4
Reviewofothergames 5 Helpseekingfromclassmatesusingwiki 2
Basiccomputerfunctions 4 Classmateshelpingeachotherwithwiderinternet
2
Results, Dataset 2: Cross-tabs, Frequency Distributions
across 2 Variables…
Task X CIS Modality:
• Appears that for easier graphic design tasks, the students
consulted each other and the internet more so, whereas for
more complex programming, students made attempts to
utilize informational wiki resources to a greater extent.
CIS Modality x Inquiry Incident Outcome
• When students relied on their peers’ expertise (most
prevalent), the results had more failed inquiry attempts
than wiki-based LMS resource uses.
Wiki can be a good source, for those who actually can/do use
it…
Nested Cross-tabs: Which Tasks and CIS Modalities
Reflect Higher Unresolved Rates? Those Highlighted N Row Labels Resolved Unresolved In Progress
186 Graphdesdraw
57% 10% 32%
2 alonewiki 0% 0% 0%
54 aloneint 59% 7% 33%
78 teamself 56% 10% 33%
6 teamwik 0% 0% 100%
38 teamint 58% 16% 26%
4 nteamself 100% 0% 0%
4 teachself 100% 0% 0%
156 Progbasic 38% 8% 46%
30 alonewiki 10% 0% 60%
69 teamself 52% 9% 39%
21 teamwik 43% 0% 43%
3 teamint 0% 0% 100%
6 nteamself 0% 50% 50%
3 nteamwik 100% 0% 0%
18 teachself 50% 17% 33%
6 teacherinitiated
0% 0% 100%
N Row Labels Resolved Unresolved In Progress
248 Progadv 26% 32% 39%
56 alonewiki 7% 21% 64%
40 teamself 60% 30% 10%
28 teamwik 14% 14% 71%
16 nteamself 0% 75% 25%
4 nteamwik 0% 0% 100%
4 nteamint 0% 100% 0%
44 teachself 36% 27% 36%
40 teachwik 30% 40% 20%
8 teachint 50% 50% 0%
8 teacherinitiated
0% 50% 50%
195 Gamesubdev 67% 10% 23%
35 aloneint 100% 0% 0%
75 teamself 67% 7% 27%
40 teamint 75% 25% 0%
15 teachself 33% 33% 33%
10 teachint 100% 0% 0%
20 teacherinitiated
0% 0% 100%
Results, Dataset 2: Cross-tabs, Frequency Distributions
across 2 Variables…
For the most complex task of Advanced Programming. . . • Higher proportions of unresolved incidents than for other tasks, across
several CIS modalities
• When you separate out information-seeking by Task, CIS Modality and
Outcomes, we see that these 3 variables appear related, such that in
guided discovery based settings like this: • For the more complex task of advanced programming, the more likely
students are to seek online information resources in the LMS than for
other tasks
• For the more complex task of advanced programming, the more likely the
inquiry incidents are to be unresolved.
• Role of educators’ expertise needs enhancement. . .
• And/or role of students’ information literacy (thus, more IL instruction) . . .
Implications
Important to note:
• Tasks and game quality variables are local to this Constructionist learning
intervention, but may be applicable to others teaching game design / digital
literacy
• CIS modality categories that emerged (collaboration type, melded with
information resource) may be applicable to many other informational and
educational settings . . .
• Inquiry outcome categories that emerged (successful / failed inquiry
attempts) also may be applicable to many other informational and
educational settings . . .
These are in-progress results!
Currently reviewing literature on Task, CIS and “sense-making” and
considering theoretical linkages.
Broad Summary of Findings
Is this an “effective” intervention?
• Digital literacy and digital divide positive effects (Reynolds & Chiu, forthcoming)
• School standardized tests positive effects (Edvantia research reports)
• Globaloria is autonomy-supportive and appears more conducive for
intrinsically motivated students (Reynolds & Chiu, 2012)
Information resource use research on Globaloria indicates:
To the extent that more advanced game programming is a program learning
objective, then it appears the program needs more structure for this. . .
Teachers, System, Information Literacy
Next Steps
Need to deploy and test improvements in the following areas:
• Teacher expertise (professional development)
• LMS system design refinements in the more advanced game
programming topics o More research needed on what programming functions pose greatest
challenges, how to design better
• Offer more information literacy instruction o Student question formulation, resource review and search, synthesis,
problem-solving, application
Future Questions invited
• What “levels of analysis” exert greater effects on learning
outcomes in complex, social constructivist “social learning
systems”?
• Design of information system/curricular features, scope,
sequencing?
• Teacher qualities / pedagogical practices;
• Student qualities / collaborative information seeking
practices (Individual; Team)
• Multi-level analysis modeling with colleague, Dr. Ming Ming Chiu
Rebecca.reynolds@gmail.com
Rutgers University website
http://comminfo.rutgers.edu/directory/rbreynol/index.html
Thanks to IMLS!
Thanks to my partners!
Globaloria.org
Worldwideworkshop.org
Thank you!
Extra Slides from Slidebank….
Methods, Dataset 1, additional information
Google Analytics page read data for a sample of students who
participated in the Globaloria Game Design Program, during the
2012/2013 school year.
• 708 students in grades 6-12
• 31 rural West Virginia public schools participated for a full
school year, in Globaloria.
• Omitting single semester and mixed group schools, and thus
the findings below represent the 21 full year schools only, for
a total of 455 students
• These students created a total of 340 individual hidden object
games, and 93 team games.
• NOTE: Google Analytics data available at SCHOOL level
only.
Methods, Dataset 1 (additional information)
Part 1. Descriptive Page Views
Aggregate page view findings by school, observe variation
Part 2. Relationship of Resource Uses (PVs) to Learning
Outcomes
Comparative analysis by school based on descriptive data on
resource uses across time
• Sort process data by outcomes:
• High, Mid, Low performing groups
• Visually observe patterns
• Build rationale for new hypotheses based on apparent
observable differences
Prevalence of E-Learning in the U.S. 1Evergreen Education Group (2013); 2NCES (2011)
25 states have state virtual schools operating in 2013-2014.1
29 states and Washington, DC have statewide full-time online schools operating in 2013-14 1
An estimated 1,816,400 enrollments seen in distance-education courses in K-12 school districts in 2009-2010, almost all of which were online courses. 74% of these enrollments were in high schools. Online courses with the highest level of enrollment fall under the categories of credit recovery (62%), dual enrollment (47%), and advanced placement (29%).2
This enrollment estimate does not include students attending most full-time online schools — approximately 200,000 full-time students in 2009-2010. As of 2012-2013, the number of students has grown to 310,000. 1
Single and multi-district blended and online programs are the largest and fastest-growing segment of online and blended learning. 1
Top reasons school districts make online learning opportunities available to their students are to offer courses not otherwise available, and provide opportunities for students to recover course credits. Credit recovery is especially important in urban environments where 81% of schools indicate this is an issue.2
Blended Learning Models
1. Traditional Schools, But With Online Options
• Online Lab Model: Instruction delivered by online teachers through a digital learning platform but within a brick-and-mortar lab environment.
• Self-Blend Model: Students take one or multiple online courses outside the school facility to supplement schooling.
2. Blended Schools
• Rotation Model: Fixed schedule rotating students between online learning and traditional classroom learning. The face-to-face teacher accountable for both the online and in-classroom work.
• Flex Model: Most of the instruction is delivered by an online platform with face-to-face teachers available for on-site support. Teachers provide tutoring sessions and small group sessions.
Blended Learning Models
3. More Virtual Than Traditional
• Online Driver Model: Students receive all primary instruction online (through an online platform and from an online teacher) with occasional face-to-face check-ins.
• On/Off-Site Rotations: Students receive most primary instruction online, but also come on-site on a scheduled part-time basis.
4. Lab Programs
• Full-time programs at a site attached to a school, including on-site teachers or paraprofessionals, online instructors, subject-area
5. Supplemental Programs:
• Serve students in a blended setting for one to several courses, while they take the rest of their curriculum in a traditional face-to-face environment.
1
2
3
4
Contemporary Learning Ability
(CLA)
Practices Representing Each CLA & How They Are Articulated/Integrated in Globaloria
Invention, creation and completion
of a digital project stemming from
an original idea
Brainstorming and developing game and simulation ideas and storylines using Web2.0 tools
Writing an original game narrative and a proposal to explain it
Generating creative ideas for designs to express the subject of the game and the user experience
Planning/programming/completing a game demo that illustrates the original game design and
functionality Developing knowledge of the game's domain or topic through game invention/creation/research
Project planning, project
management, teamwork (e.g., role-taking,
task delegation),
problem-solving
Coordinating and managing the design/creation/programming of game elements
Managing the project’s execution by creating/organizing a wiki and by sharing project assets and progress updates
Managing team work by defining and assigning team roles/coordinating tasks/executing roles
Project troubleshooting for self and others
Gaining leadership experience through the project management of all game production elements
Publishing and distribution of
self-created digital media artifacts
to an audience and/or community
of peers
Creating a wiki profile page and project pages
Integrating and publishing text/video/photos/audio/programming code/animations/digital designs
on wiki pages
Posting completed assignments/game design iteration and assets/notes and reflections about
projects to wiki Developing a blog
Giving and getting feedback
about project through social interaction,
participation, exchange
Collaborating by using Web2.0 tools such as posting to wikis/blogs/open source help
forums/instant messaging Exchanging/sharing feedback and resources by posting information/links/source code questions/
answers
Reading and commenting on others’ blogs and wiki pages
Presenting final digital projects for others both virtually in game galleries and in person in live
game demonstrations
5
6 Figure1.Globalorialearningobjectives:Promotedevelopmentofsixcontemporarylearningabilities(CLAs)
Inquiry, information-seeking,
agentive
use of resources (human and text/digital content), to support
the artifact’s topic/message,
and design/execution
Searching the Web for answers and help on specific issues related to programming games
Searching and finding resources on MyGLife.org network, website, and wiki
Searching the Web for new Flash design, animation and programming resources Searching for information in support of the game’s educational subject and storyline
Surfing, experimentation and play with existing Web
applications and tools
Surfing to MyGLife.org starter kit site and other game sites and playing games online Keeping track of and bookmarking surfing results that are relevant to projects
Browsing Web2.0 content sites such as YouTube, Flickr, blogs, Google tools
Constructionist, knowledge-building conditions supporting development
of 6-CLAs (Reynolds & Harel Caperton 2009; Reynolds & Hmelo-Silver,
2013)
Learning Management System as Information System
Learning Management System as Information System