Applying learning analytics in serious games
-
Upload
baltasar-fernandez-manjon -
Category
Education
-
view
1.400 -
download
1
Transcript of Applying learning analytics in serious games
![Page 1: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/1.jpg)
Applying Learning Analytics in Serious Games
Baltasar Fernandez-Manjon
e-UCM research group, www.e-ucm.es
@BaltaFM
École Normale Supérieure de Lyon, Novembre 2014
http://www.slideshare.net/BaltasarFernandezManjon/
![Page 2: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/2.jpg)
About me and context
• CS Professor at Complutense University• Director of e-UCM
• e-UCM research group aboutLearning technologies• 15 researchers• Serious games
• Development of technology• eAdventure, GLEANER
• Application to the medical domain• European projects
• H2020 – RAGE (call 21)• SEGAN
• www.e-ucm.es
2
RAGE -H2020
![Page 3: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/3.jpg)
Educational games in the classroom: game• The Radix Endeavor
• https://www.radixendeavor.org
• MMOG with Science, Technology, Engeneering and Math (STEM) topics• Many different quests
• Adapted to middle and high school curriculum
• Facilities for teachers• Class configuration
• Quest assignation
• Now being tested in actual settings
![Page 4: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/4.jpg)
Educational games in the classroom: settings• Master students in the e-
learning class• 13 students, 8 nationalities
• No previous info about the game• If stuck ask for help or watch the
help videos
• One hour free play• Who perform “better” at 30
minutes and at the end
• Publish screen captures in the class Moodle course forum
![Page 5: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/5.jpg)
Educational games in the classroom: outcome• Students were really engaged for the whole hour
• They enjoyed the experience
• Consider the game a commercial quality game
• No one asked for help
• All of them published the screenshots in the Moodle class forum
• Educational outcome• Only 4 of the students complete more than 2 quests
• Most of the students just spend the time exploring the environment but not doing anything “useful” or learning anything
• Actual data can be contradictory with simple observation
![Page 6: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/6.jpg)
Learning Analytics• Improving education based on
data analysis• Data driven
• Evidence-based education• Educational theories/approaches
can be contrasted with actual data
• Related with …• Educational data mining
• Business intelligence
• Visual analytics
6www.ed.gov/edblogs/technology/files/2012/03/edm-la-brief.pdf
![Page 7: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/7.jpg)
Learning analytics steps• 1. Collecting large amounts of data from a number of channels
• Interaction with the system
• 2. Translating that data into information or actionable insights. • It may be impossible to track how much a student really absorbed from one
lesson but the system CAN track his/her behaviour and use that as a signal
• 3. Use of the information for different purposes• Personalization and adaptation. Once the system gets the signal, it can then
personalize each student’s learning environment.
• Assessment. Use that information for formative or summative assessment
• Predicting the best course in the future. As students use the system for a prolonged period of time, educators will be able to track what works and what doesn’t – and adjust accordingly.
http://www.edudemic.com/grades-2-0-how-learning-analytics-are-changing-the-teachers-role/
![Page 8: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/8.jpg)
The 4 Levels of Learning Analytics
•Descriptive• What has happened?
•Diagnostic• Why did it happen?
•Predictive• What will happen?
•Prescriptive• What should I do?
http://www.edudemic.com/4-levels-learning-analytics-graphic/
![Page 9: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/9.jpg)
Game Analytics: a parallel world• Application of analytics to game
development and research
• Telemetry• Data obtained over distance• Mobile games, MMOG
• Game metrics• Interpretable measures of data
related to games• Player behavior
• User metrics• Generics metrics• Genre specific metrics• Game specific metrics
9
![Page 10: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/10.jpg)
Uses of Learning Analytics in educational games
• Game testing – game analytics• It is the game realiable?
• How many students finish the game?
• Average time to complete the game?
• Game evaluation• From pre-post test to learning analytics evaluation
• Game deployment in the class• Real-time information for supporting the teacher
• “Stealth” student evaluation
• Knowing what is happening when the game is deployed in the class
![Page 11: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/11.jpg)
The Feedback loop• Not only important the feedback but also when is provided
• Learning analytics can help in improving feedback
http://www.wired.com/2011/06/ff_feedbackloop/
http://safety.fhwa.dot.gov/speedmgt/ref_mats/fhwasa12004/
![Page 12: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/12.jpg)
Use of the educational games: Black box model
• Games as independent pieces of content
• No information about what is happening during the in-game play
• Or very simple• Completed or not completed
• Time used
12
del Blanco et al (2013). Using e-Learning standards in educational video games. Computer Standards & Interfaces 36 (1) pp. 178–187
![Page 13: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/13.jpg)
Designing for Learning Analytics in Games• Educational games need to be designed for Learning Analytics
• Games make use of in game mechanisms for the assessment of player performance and progress
• Interrelate observable in game behaviors to a competency based model related with learning outcomes rather than game performance• What are the relevant educational situations that want to be identified?
• Do we have the way to communicate the game state?• Varibles, flags, identifiers
Implications of Learning Analytics for Serious Game Design. Jannicke Baalsrud Hauge, Riccardo Berta, Giusy Fiucci, Baltasar Fernandez Manjon, Carmen Padron-Napoles, Wim Westera and Rob Nadolski. IEEE ICALT 2014
![Page 14: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/14.jpg)
Use of educational/serious games at the school
Takeuchi, L. M., & Vaala, S. (2014). Level up learning: A national survey on teaching with digital games. New York: The Joan Ganz Cooney Center at Sesame Workshop
![Page 15: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/15.jpg)
Use of games at the school
![Page 16: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/16.jpg)
Our work in Serious/EducationalGames
![Page 17: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/17.jpg)
eAdventure game platformOpen code authoring environment for the production of point-and-clickadventure games & immersive learning simulations
Easy to include Learning Analytics in eAdventuregames
http://sourceforge.net/projects/e-adventure/
![Page 18: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/18.jpg)
http://www.chermug.eu
With MGH-Harvard Universityhttp://first-aid-game.e-ucm.es
With ONT, educ@ONT
![Page 19: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/19.jpg)
eAdventure + Learning Analytics
Game Engine
CommunicationAPI
{
type: 'input',
timeStamp: some_timestamp,
device: 'some_device',
action: 'some_action',
target: 'target_id',
data: { key1: value, ...}
}
{
type: 'logic',
timeStamp: some_timestamp,
event: 'some_event',
target: 'some_id',
data: { key1: value, ...}
}
Learning Analytics Database
![Page 20: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/20.jpg)
GLEANER
• GLEANER: Game Learning Analytics for education research• Open code framework to capture game traces
20
Reference model in the EU NoE GALA, http://e-ucm.github.io/gleaner/
Ángel Serrano-Laguna, Javier Torrente, Pablo Moreno-Ger, Baltasar Fernández-Manjón (2014): Application of Learning Analytics in Educational Videogames. Entertainment Computing, Elsevier (in press, early access available).
![Page 21: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/21.jpg)
GLEANER Architecture
![Page 22: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/22.jpg)
GLEANER Architecture - Technologies
![Page 23: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/23.jpg)
GLEANER Analysis
• Reporter has access to the database, and presents its data through reports• graphics, heat-maps, relational tables…
• Evaluator has access to the database and checks the educational defined goalsin the assessment model
![Page 24: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/24.jpg)
ADL eXperience API (xAPI)
‣ Result of Project Tin Can
‣ Tracks experiences, informal learning, real-world experiences (not just completions)
‣ Allows data storage AND retrieval (ex. 3rd party reporting and analytics tools)
‣ Enables tracking mobile, games, ITS, and virtual worlds experiences
‣ Developed by open source community
24From Damon Regan (ADL) presentation at SINTICE2013
![Page 25: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/25.jpg)
25
Activity Streams• http://activitystrea.ms
• Collaboration between Google, Facebook, Microsoft and others
• Allows reporting of experiences, not just completions
• Format: <Actor> <Verb> <Object> (I did this):
Simple Statement:I (actor) watched (verb) a video on protecting employee data (object)
Complex: in the context of [information assurance certification training] with result [timestamp:2013-0618T18:30:32.360Z ].
From Damon Regan (ADL) presentation at SINTICE2013
![Page 26: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/26.jpg)
Reporting
Systems
Assessme
nt Services
Semantic
Analysis
Statistical
Services
xAPI Learning Record Store (LRS)
26From Damon Regan (ADL) presentation at SINTICE2013
![Page 27: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/27.jpg)
GLEANER traces and xAPI
![Page 28: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/28.jpg)
Example: A game to learn XML
http://gleaner.e-ucm.es/lostinspace/play/index.html
![Page 29: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/29.jpg)
About the game...
• A basic platform game to acquire familiarity with XML documents• Learn the syntax• Understand nesting and attributes• Gain agility writing and reading XML documents
• Designed as a complementary activity in a Web Programming Course• For undergraduate computer science students
• Not developed with eAdventure
![Page 30: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/30.jpg)
Understanding the game…
Goal
Main character
Power-ups (new syntax elements)
Write XML snippets here to move the main
character to the goal
![Page 31: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/31.jpg)
Educational goals
• Basic understanding of XML Syntax• Markup syntax• Basic documents• Attributes• Complex documents
• Basic programing skills• Sequencing• Loops
![Page 32: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/32.jpg)
LA perspective
• What does GLEANER trace?• Higher level events• Generic traces & game specific traces• Only those relevant for our learning objectives
• The aggregator will filter and transmit:• Level completion events• Each XML fragment submitted by the player
![Page 33: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/33.jpg)
Main dashboard (GLEANER Report)
33
![Page 34: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/34.jpg)
Progress report
34
User Session
Phases completed
Current score
Learning achievements
Achievements are updated and
highlighted in real time
![Page 35: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/35.jpg)
Detailed traces
35
Filters XML snipets
The fragments are updated in real time as submitted by the student. The instructor can filter to see only mistakes,
or all the fragments submitted by a specific students.
![Page 36: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/36.jpg)
LA simplify SG deployment in the classroom
• Experiment: Gamified experience in the classroomGoals• First to complete the game• Maximum points in the gamebut• Initially students were not informed that LA was in placeAfter some playing time LA data was showed to the students‣ Instructor knows what is happening in the class
‣ Behavior change when students know that LA was in place• More oriented to success and less exploration or “playing”
Ángel Serrano-Laguna, Baltasar Fernández-Manjón (2014): Applying learning analytics to simplify serious games deployment in the classroom. Proceedings of the 2014 IEEE Global Engineering Education Conference (EDUCON) Pages 872-877.
![Page 37: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/37.jpg)
Experimental design and Learning Analytics
• Learning Analytics should be in-line with the experimental design• Annonymization• Use of the data
• Specific regulations in several domains• Medical: ethics committees
![Page 38: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/38.jpg)
La Dama Boba : Educational game for promoting students interest in classical theater
Based on The Foolish Lady by Lope de VegaThe game is available at http://damaboba.e-ucm.es/ (in Spanish) Also in in Android Market
![Page 39: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/39.jpg)
Experiment game vs class• Teacher with an improved
standard content• 757 students
• From 8 middle and high schoolsin Madrid region
• Control group and experimental group
![Page 40: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/40.jpg)
Formal evaluation pre-post• Off-line learning analytics
• Formal evaluation of games isvery complex and expensive• Pre-test
• Post-test
• Very few games have beenformally probed to be effective
• Similar results with Learning Analytics than with pre-post test?
40
![Page 41: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/41.jpg)
learning analytics outcomes: scores, time, ...
![Page 42: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/42.jpg)
How the user play the game?
42
![Page 43: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/43.jpg)
Open issues in learning analytics in games• How to distinguish automatically game exploration from conceptual
errors?
• Traces across different types of games
• Long term learning across games
![Page 44: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/44.jpg)
Other important issues in LA
• Ethical and legal aspects
• Security model
• Ownership of information• Informing the user
• Anonymization of information
• Aspects specially relevant if you are working with kids or in the medical domain!
44
![Page 45: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/45.jpg)
Conclusions
• LA in Serious Games has a great potential from the application and research perspective• Ease the deplyment of game in the classroom• Simplify game evaluation with users
• LA in Serious Games should benefits from Games Analytics experience and work
• Still complex to implement LA in SG• Increases the (already high) cost of the games
• Frameworks and new standards specifications could greatly simplify LA implementation and adoption
45
![Page 47: Applying learning analytics in serious games](https://reader033.fdocuments.us/reader033/viewer/2022051213/55a0dcd21a28ab235a8b4784/html5/thumbnails/47.jpg)
Our current projects
RAGEH2020 ICT2014-21ª
Realising an Applied Gaming Eco-system