Monitoring, Awareness and Reflection in Blended Learning: a Systematic Review

M. J. Rodríguez-Triana, L. P. Prieto Santos, A. Vozniuk, M. Shirvani Boroujeni and B. A. Schwendimann, A. Holzer, D. Gillet

6th Workshop on Awareness and Reflection in Technology- Enhanced LearningSeptember 13th 2016, Lyon (France)

Rodríguez-Triana, M.J., Prieto, L.P., Vozniuk, A., Shirvani Boroujeni, M., Schwendimann, B.A., Holzer, A., Gillet, D. Monitoring, awareness and reflection in blended technology enhanced learning: a systematic review. International Journal on Technology Enhanced Learning. Special issue on: Awareness and Reflection (In press) http://go.epfl.ch/ARTEL

2 / 39

Motivation > Definitions > Methodology > Results > Conclusions

Motivation

Motivation Definitions Methodology Results Conclusions

3 / 39

Educational trends and challenges

Blended learning models(The New Media Consortium, 2013;

Oliver & Trigwell, 2005;Diaz & Brown, 2010)

Make use and sense of data(Sutherland et al., 2012;

Fischer et al., 2014;Eberle et al., 2014)

Motivation Definitions Methodology Results Conclusions

4 / 39

5 / 39

Making use & sense of data in blended learning

(Sclater, 2008; Ferguson, 2012; Ruiz et al., 2013)

Motivation Definitions Methodology Results Conclusions

is complex

Reviews in Blended Learning & Learning Analytics

Big data & Learning Analytics in blended learning environments

(Picciano, 2014)

Motivation Definitions Methodology Results Conclusions

Monitoring tools(Yengin et al, 2010;Conde et al., 2015)

Students' interpretation(Corrin & de Barba, 2014)

Learning Analytics dashboards(Verbert et al., 2013 & 2014;Schwendimann et al., 2016)

6 / 39

Motivation Definitions Methodology Results Conclusions

But what abouthow monitoring, awareness

& reflection may supportblended learning?

7 / 39

Definitions

Motivation Definitions Methodology Results Conclusions

8 / 39

"Any combination of learning delivery methods"

(So & Brush, 2008)

"Blended learning does not refer to a single model but comes in many different flavors and styles"

(Picciano, 2014)

Blended, Mixed & Hybrid learning

"The inclusion of multiple approaches to teaching and learning within a programme"

(Busuttil-Reynaud & Winkley, 2006)

Motivation Definitions Methodology Results Conclusions

9 / 39

"Combination of face-to-face and technology-supported learning activities"

(Koper, 2005; Graham, 2005)

"... some performed synchronously and others asynchronously"

(Diaz & Brown, 2010)

"The thoughtful integration of classroom face-to-face learning experiences with online learning experiences"

(Garrison & Kanuka, 2004)

"A blend of spaces, a blend of activity types (formal and non-formal), and a blend of technologies to integrate activities"

(Pérez-Sanagustín, 2011)

Blended, Mixed and Hybrid learning

Motivation Definitions Methodology Results Conclusions

10 / 39

Monitoring, Awareness & Reflection

Monitoring Awareness Reflection

Motivation Definitions Methodology Results Conclusions

11 / 39

Research questionHow have LA & EDM contributed to monitor learners' actions and to analyse data

in order to support awareness and reflection in blended TEL?

Questions:1. in which learning contexts have monitoring, awareness or reflection been studied? 2. what kind of research problems around monitoring, awareness & reflection have been investigated?3. what solutions have been provided for those problems?4. what is the maturity of these solutions in terms of evaluation?5. what are the open issues and future lines of work in this field?

Motivation Definitions Methodology Results Conclusions

12 / 39

Motivation Definitions Methodology Results Conclusions

Methodology

13 / 39

Methodology

Systematic literature review (Kitchenham & Charters, 2007)

Motivation Definitions Methodology Results Conclusions

14 / 39

Results1. in which learning contexts have monitoring, awareness or reflection

been studied?

2. what kind of research problems around monitoring, awareness and reflection have been investigated?

3. what solutions have been provided for those problems?

4. what is the maturity of these solutions in terms of evaluation?

5. what are the open issues and future lines of work in this field?

Motivation Definitions Methodology Results Conclusions

15 / 39

Motivation Definitions Methodology Results Conclusions

Results

16 / 39

Learning contextHow did authors interpret blended learning?

40

Motivation Definitions Methodology Results Conclusions

17 / 39

In which educational contexts have monitoring, awareness or reflection been studied?

Learning context 40

Motivation Definitions Methodology Results Conclusions

18 / 39

Learning contextIn which technological contexts have monitoring, awareness or reflection been studied?

40

Motivation Definitions Methodology Results Conclusions

19 / 39

Learning context 40

Motivation Definitions Methodology Results Conclusions

In which technological contexts have monitoring, awareness or reflection been studied?

20 / 39

Research problemhow many papers addressed monitoring, awareness and reflection?

40

Motivation Definitions Methodology Results Conclusions

21 / 39

Research problemwho were the target users?

40

Motivation Definitions Methodology Results Conclusions

22 / 39

Solutionswhat type of solutions have been provided for the aforementioned problems?

40

Motivation Definitions Methodology Results Conclusions

23 / 39

Solutionswhich LA data sources were used?

40

Motivation Definitions Methodology Results Conclusions

24 / 39

Solutionswhich indicators were used?

40

(e.g., prior education, competences, university entrance grade)

(e.g., average grade in a group)

(e.g., number of page visits, number of file downloads)

(e.g., sentiment of the forum messages, topics covered in the report)

(e.g., geographical location of learners, weather conditions during the activity)

(e.g., a graph showing communication direction in a forum)

Motivation Definitions Methodology Results Conclusions

25 / 39

Evaluationwhat is the maturity of these solutions in terms of evaluation?

● 29 (77.5%) were evaluated

40

Motivation Definitions Methodology Results Conclusions

26 / 39

Evaluationwhat was the aspect under evaluation?

29

Motivation Definitions Methodology Results Conclusions

27 / 39

Evaluationwhat was the scale & duration of the evaluation in authentic conditions?

14

Motivation Definitions Methodology Results Conclusions

● 13 gather data from students (min: 11; max: 300)

● 10 gather data from teachers (min: 1; max: 20)

● Repetitions: 1 to 3 courses● Length: from 1 week to a full

semester (per course)

28 / 39

Motivation Definitions Methodology Results Conclusions

Conclusions

29 / 39

Motivation Definitions Methodology Results Conclusions

Research questionHow have LA & EDM contributed to monitor learners' actions and to analyse data

in order to support awareness and reflection in blended TEL?

1. There's no agreed definition about monitoring, awareness, reflection & blended learning

2. Existing solutions are mainly:a) Devoted to teachersb) Applied university settingsc) Focused on computer-mediated interactions obtained from a single platformd) Coupled solutionse) Barely evaluated and adopted in authentic settings

30 / 39

Motivation Definitions Methodology Results Conclusions

What about open issues?

31 / 39

Open issues: stakeholders & purpose

- Support students in monitoring, awareness and reflection activities

- Provide feedback to teachers about their own practice

Motivation Definitions Methodology Results Conclusions

32 / 39

Open issues: educational level

- Support primary & secondary education

- Support non-formal learning

Motivation Definitions Methodology Results Conclusions

33 / 39

Open issues: proposals

- Design and build LA tools based on theoretical foundations (Gaševic et al., 2015)

- Extract technologies and (intervention) design principles for effective support of awareness and reflection

- Integrate privacy & ethical concerns

- Apply multimodal analyses

Motivation Definitions Methodology Results Conclusions

34 / 39

Open issues: infrastructure

- Decouple learning and analytics tools

- Adopt data gathering & integration architectures (e.g., Graasp & GLUE!)

- Adopt standards for learning data representation (e.g. xAPI, Caliper)

Motivation Definitions Methodology Results Conclusions

35 / 39

Open issues: data sources

- Gather evidence from face-to-face learning

- Retrieve data from sensors, mobile & wearable technologies

Motivation Definitions Methodology Results Conclusions

36 / 39

Open issues: indicators

- Consider learner, context and social related

Motivation Definitions Methodology Results Conclusions

37 / 39

Open issues: evaluation

- Look at how (or how much) awareness and/or reflection are improved

- Study the effects on learning

- Run longer-term, longitudinal studies

Motivation Definitions Methodology Results Conclusions

38 / 39

Thanks

Rodríguez-Triana, M.J., Prieto, L.P., Vozniuk, A., Shirvani Boroujeni, M., Schwendimann, B.A., Holzer, A., Gillet, D. Monitoring, awareness and reflection in blended technology enhanced learning: a systematic review. International Journal on Technology Enhanced Learning. Special issue on: Awareness and Reflection (In press) http://go.epfl.ch/ARTEL

Contact: maria.rodrigueztriana@epfl.ch

39 / 39

References● Busuttil-Reynaud, G. & Winkley, J. (2006), JISC e-assessment glossary, Technical report, Joint Information Systems Committee (JISC), Bristol.● Conde, M. A., Hérnandez-García, A., García-Peñalvo, F. J. & Séin-Echaluce, M. L. (2015), Exploring Student Interactions: Learning Analytics

Tools for Student Tracking, in ‘Learning and Collaboration Technologies’, Springer, pp. 50–61.● Corrin, L. & de Barba, P. (2014), Exploring students’ interpretation of feedback delivered through learning analytics dashboards, in ‘Proceedings

of the ascilite 2014 conference’.● Diaz, V. & Brown, M. (2010), ‘Blended Learning: a report on the ELI focus session’, EDUCAUSE Learning Initiative.● Eberle, J., Lund, K., Tchounikine, P., & Fischer, F. (2014). Challenge Problems in Technology-Enhanced Learning II: MOOCs and Beyond

Perspectives for Research, Practice, and Policy Making Developed at the Alpine Rendez-Vous in Villard- de-Lans. London: Springer. ● Ferguson, R. (2012), ‘Learning analytics: drivers, developments and challenges’, International Journal of Technology Enhanced Learning● Fischer, F., Wild, F., Sutherland, R., & Zirn, L. (2014). Grand Challenges in Technology Enhanced Learning: Outcomes of the 3rd Alpine

Rendez-Vous. London: Springer. ● Garrison, D. R. & Kanuka, H. (2004), ‘Blended learning: Uncovering its transformative potential in higher education’, The internet and higher

education 7(2), 95–105.● Gaševic, D., Dawson, S. & Siemens, G. (2015), ‘Let’s not forget: Learning analytics are about learning’, TechTrends 59(1), 64–71.● Graham, C. R. (2005), Blended learning systems: Definition, current trends, and future directions, in C. J. Bonk & C. R. Graham, eds, ‘Handbook

of blended learning: global perspectives, local designs,’, Pfeiffer, San Francisco, CA, pp. 3—-21.● Kitchenham, B. & Charters, S. (2007), Guidelines for performing systematic literature reviews in software engineering, Technical report, Keele

University (UK).● Koper, R. (2005), An introduction to Learning Design, in R. Koper & C. Tattersall, eds,‘Learning Design: A Handbook on Modelling and Delivering

Networked Education Training’, Springer Berlin Heidelberg, pp. 3–20.● Oliver, M. & Trigwell, K. (2005), ‘Can ‘blended learning’be redeemed?’, E-learning and Digital Media 2(1), 17–26.● Pérez-Sanagustín, M. (2011), Operationalization of collaborative blended learning scripts: a model, computational mechanisms and experiments,

PhD thesis, Universitat Pompeu Fabra.

References● Picciano, A. G. (2014), ‘Big Data and Learning Analytics in Blended Learning Environments: Benefits and Concerns’, International Journal of

Artificial Intelligence and Interactive Multimedia 2(7), 35–43.● Ruiz, S., Urretavizcaya, M. & Fernandez-Castro, I. (2013), Monitoring F2f interactions through attendance control, in ‘2013 IEEE Frontiers in

Education Conference’, pp. 226–232.● Schwendimann, B.A., Rodríguez-Triana, M. J., Vozniuk, A., Prieto Santos, L. P., Shirvani Boroujeni, M., Holzer, A., Gillet, D., & Dillenbourg, P.

(2016). Perceiving learning at a glance: A systematic literature review of learning dashboard research. IEE transactions on learning technologies journal. (in press)

● Sclater, N. (2008), ‘Web 2.0, personal learning environments, and the future of learning management systems’, EDUCAUSE research bulletin.● So, H.-J. & Brush, T. A. (2008), ‘Student perceptions of collaborative learning, social presence and satisfaction in a blended learning environment:

Relationships and critical factors’, Computers & Education 51(1), 318–336.● Sutherland, R., Eagle, S. & Joubert, M. (2012), A vision and strategy for Technology Enhanced Learning, Report from the STELLAR Network of

Excellence.● The New Media Consortium, N. (2013), NMC Horizon project preview: 2013 Higher Education edition, Technical report, The New Media

Consortium.● Verbert, K., Duval, E., Klerkx, J., Govaerts, S. & Santos, J. L. (2013), ‘Learning analytics dashboard applications’, American Behavioral Scientist● Verbert, K., Govaerts, S., Duval, E., Santos, J. L., Assche, F., Parra, G. & Klerkx, J. (2014), ‘Learning Dashboards: An Overview and Future

Research Opportunities’, Personal and Ubiquitous Computing. 18(6), 1499–1514.● Yengin, I., Karahoca, D., Karahoca, A. & Yücel, A. (2010), ‘Roles of teachers in e-learning: How to engage students & how to get free e-learning

and the future’, Procedia - Social and Behavioral Sciences 2(2), 5775–5787.