Multilevel analysis of collaborative activities

based on

a Mobile Learning Scenario

for real classrooms

Irene-Angelica Chounta, Adam Giemza, H. Ulrich Hoppe

Collide, University of Duisburg-Essen

{chounta, giemza, hoppe}@collide.info

Mobile learning: a special case of learning....

� Learning on the move, across space, within context

� Activities supported by mobile devices

− „location-based“ scenarios with spatially distributed learning stations (museum, outdoor trips etc)

− smartphones, tablets etc.

� The activity analysis is carried out with qualitative methods

Objectives of the study

� Introduction of a mobile learning scenario in traditional school classrooms

� Use of common learning analytics within a mobile learning context

� Study of collaborative learning activities on multiple levels of analysis

a mobile learning scenario for the classroom

� Computer Kit: „Learn how to assemble a Personal Computer with Mobilogue“

a mobile learning scenario for the classroom

� QR codes as learning station/ computer parts markers in a very localized setting

Learning Content Video Quiz Scores

Mobilogue Architecture

� Authoring environment for the creation of new activities

� Activity logs for recording and post analysis

� Online Management Environment

Method of the study

� Two case studies:

− Case A: preliminary study, 17 students

− Case B: main study, 24 students

� Task: Group of students working together to assemble a pc

� Each group was supported by one mobile device

� The activities were analyzed with respect to:

- students practice on the individual and the group level

- group characteristics

- learning outcome

13 – 15 years old

Method of the study

� For the analysis we used:

- Activity metrics from logfiles - Average time gap among consequent actions

- Average response time to quizzes

- Quiz score

- Experts observations

- Activity transcripts (recorded by students)

- Users questionnaires (with respect to user experience and motivation for collaboration)

- Pre and Post Knowledge Tests (each test consisted of a ten-items questionnaire)

Case study A - Results

� 17 students randomly grouped in 5 groups

� Activity duration ≈ 90 minutes

� Group profiles based on pre-tests:

� One „strong“ team (team A)

� One „weak“ team (team E)

� Three average teams (teams B, C, D)

Pretest Scoresavg_group max_idv

teamA 6.25 8teamB 3.75 5teamC 4.33 5teamD 2.67 4

teamE 1 1

heterogeneous

homogeneous

Case study A - Results

� Greatest improvement � teams of average performance (on group and individual level)

� The “strong” team maintained the same score on average but scored less on the individual level - homogeneity increased)

� The “weak” team slightly improved – heterogeneity increased

� Heterogeneous teams increase homogeneity and vice versaPre-test Scores

avg_group max_idv

teamA 6.25 8teamB 3.75 5teamC 4.33 5teamD 2.67 4

teamE 1 1

Post-test Scoresavg_group max_idv

6.25 7

6.25 8

5.67 7

6.00 7

2.00 3

Pre-test Scoresavg_group max_idv

teamA 6.25 8teamB 3.75 5teamC 4.33 5teamD 2.67 4

teamE 1 1

Case study A - Results

� Quiz score reflects the knowledge test performance

� The average time gap correlates negatively with quiz score

� Teams with good knowledge background and high heterogenity move faster

Activity metrics

avg_timegapavg

response time

Quiz score

27.63 32.02 120

36.56 56.96 12037.14 23.70 100

37.91 31.55 90

38.43 57.43 80

Case study B - Results

� 24 students – 6 groups of four

� Groups were created by the teacher

� Activity duration ≈ 90 minutes

Special Additions / Modifications:

� Experts monitored the activity

� Students kept transcripts of group activity

� Students filled in questionnaires regarding their experience on collaboration

Case study B - Results

� Group profiles based on pre-tests:

� One strong team (team 03)

� Two weak teams (team 00 and team 05)

� Three average teams (teams 01, 02, 04)

� The Homogeneity of teams according to pre-test confirmed teacher‘s perception

Pre-test Scoresavg_group max_idv

team 00 2 4team 01 5.75 9team 02 4.25 5team 03 6.5 8team 04 5.5 7

team 05 1 2

heterogeneous

homogeneous

Case study B - Results� Group performance improved on average (47%) and

individual (31%)

� Max knowledge gain � weak team (team 05)

� Min knowledge gain � strong team (team 03)

� Knowledge „loss“ for the „strong“ students

� Homogenity increases for heterogenous groups and vice versa (confirming case study A)

Pre-test Scoresavg_group max_idv

team 00 2 4team 01 5.75 9team 02 4.25 5team 03 6.5 8team 04 5.5 7team 05 1 2

Post-test Scoresavg_group max_idv

3.5 77.5 86 7

6.5 77.25 8

6 9

Case study B - Results

� Heterogeneous teams achieved higher Quiz scores

� The average time gap correlates negatively with quiz score (also observed case study A)

� Heterogeneous teams move faster, i.e. Individuals take the lead

Activity metrics

avg_timegapavg

response time

Quiz score

20.89 51.88 120

23.42 49.02 12024.00 45.73 11025.84 61.99 9040.44 72.51 70

27.75 82.56 70

Pre-test Scoresavg_group max_idv

team 00 2 4team 01 5.75 9team 02 4.25 5team 03 6.5 8team 04 5.5 7team 05 1 2

Case study B - Results

� The students undertook three roles throughout the scenario:

� the operator of the mobile device

� the scriber and

� the assembler

0

0.2

0.4

0.6

0.8

1

1.2

1.4

u00-00 u00-01 u00-02 u00-03

team00

Scriber

Assembler

Operator

0

0.2

0.4

0.6

0.8

1

1.2

1.4

u01-00 u01-01 u01-02 u01-03

team01

0

0.2

0.4

0.6

0.8

1

1.2

1.4

u02-00 u02-01 u02-02 u02-03

team02

0

0.2

0.4

0.6

0.8

1

1.2

1.4

u03-00 u03-01 u03-02 u03-03

team03

0

0.2

0.4

0.6

0.8

1

1.2

1.4

u04-00 u04-01 u04-02 u04-03

team04

Scriber

Assembler

Operator

0

0.2

0.4

0.6

0.8

1

1.2

1.4

u05-00 u05-01 u05-02 u05-03

team05

Scriber

Assembler

Operator

• 4 out of 6 teams followed no strict

role distribution

• Teams following strict role

distribution showed the lowest

knowledge & dissatisfaction

Discussion

� A mobile scenario to support collaborative activities for traditional classrooms

� Multilevel analysis of the activity (various methods and multiple units of analysis) to enhance the overall outcome

Learning Outcome

• The setting was more effective for students of weak and medium knowledge background

• Strong students/ teams showed little or no knowledge gain

• The students performance was increased on both individual and group level

Discussion

Group Characteristics & Learning Analytics

• Heterogeneous teams achieved higher game scores

• Game score did not correlate to knowledge gain or knowledge background

• Heterogeneous teams moved faster through the stages of the scenario

• Homogeneous teams tend to increase their heterogeneity and vice versa

Discussion

Collaboration

• Team members undertook roles with no particular plan

• In the case a strict plan was follows, it lead to frustration and low knowledge gain

• Strong students stated they would have done better on their own

• Weak/average students considered the activity as helpful

Conclusion and Future Work

� Multilevel analysis can provide valuable information and enhance mobile learning

� Extend the field of study to further investigate:

� the use of various scenarios

� varying degrees of freedom to classroom activities

� concurrent use of multiple devices within a team

� the nature of learning activities that could be supported effectively by the use of mobile devices

� ways to incorporate seamlessly mobile devices in traditional learning activities

Questions.

Thank you ☺

Thank you ☺

chounta@collide.info