Electronic Posters to SupportFormative Assessment

Salman CheemaArizona State UniversityTempe, AZ, USAsalman.cheema@asu.edu

Daniel PeadUniversity of NottinghamNottingham, UKdaniel.pead@nottingham.ac.uk

Kurt VanLehnArizona State UniversityTempe, AZ 85281, USAKurt.VanLehn@asu.edu

Alan SchoenfeldUniversity of CaliforniaBerkeley, CA, USAalans@berkeley.edu

Hugh BurkhardtUniversity of NottinghamNottingham, UKhugh.burkhardt@nottingham.ac.uk

Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.For all other uses, contact the Owner/Author.Copyright is held by the owner/author(s).CHI’16 Extended Abstracts, May 07-12, 2016, San Jose, CA, USAACM 978-1-4503-4082-3/16/05.http://dx.doi.org/10.1145/2851581.2892417

AbstractFormative Assessment is difficult to apply in real-worldclassrooms due to the requirement for extensive interac-tion between students and teachers. We have constructeda distributed system called FACT for in-class use that facil-itates the use of popular Classroom Challenges (CCs) de-veloped by the Mathematics Assessment Project. FACT letsstudents work on Android tablets equipped with styli, andenables a teacher to manage the class and to orchestratethe activities required by the CCs. In this work, we discussthe design of our system and our rationale for choosing itsinteraction metaphors.

Author KeywordsFormative Assessment; Pen-based Interaction; ComputerSupported Collaborative Learning

ACM Classification KeywordsH.5.2 [User Interfaces]: Interaction Styles; H.5.3 [Group andOrganizational Interfaces]: Computer Supported Coopera-tive Work

IntroductionA Formative Assessment requires teachers to observe stu-dents during the course of learning activities and to pro-vide useful feedback that guides students toward a bet-ter understanding of the subject material. This is differ-

Late-Breaking Work: Collaborative Technologies #chi4good, CHI 2016, San Jose, CA, USA

1159

ent from a Summative Assessment which requires test-ing and relies on scores/grades to ascertain the level ofstudent learning. The Mathematics Assessment Project(http://map.mathshell.org/) has developed 100 ClassroomChallenges (CCs) for use in middle and high school mathe-matics classrooms. These lessons cover a range of math-ematical topics and provide rich collaborative activities forstudents that are designed to expose common misconcep-tions. The Classroom Challenges and associated materi-als are freely available and have seen over 3 million down-loads.

In this work, we present the FACT system which providesa digital equivalent of the Classroom Challenges for use inclassrooms. This system has been tested multiple times inmiddle school classrooms in California and Britain. FACTuses both pen- and touch-input to support a rich set oftasks that enable students to reason about mathematicalconcepts in a natural manner. We first describe the exist-ing paper-based CCs, then the design and workflow of theFACT system, followed by a discussion of lessons learned.

Figure 1: Graph and interpretationcard sets taken from the‘Interpreting Distance-TimeGraphs’ lesson.

Paper-based Classroom ChallengesThe Classroom Challenges (CCs) developed by the Mathe-matics Assessment Project are structured as teacher man-uals, containing a suggested lesson plan and all requiredstudent and projector materials. Each CC has 3 stages: apre-assessment, a collaborative small group-based mainactivity and a post-assessment, all taking place on differentdays. The activities can be broadly split into two groups:card arrangement and handwriting-intensive problem solv-ing problems. The lesson plans also describe common er-rors and guidelines for teachers on how to address themusing formative questions. An example of a card arrange-ment CC is the ‘Interpreting Distance-Time Graphs’ lesson.In this lesson, students are given cutouts of graph cards

and corresponding interpretations (See Figure 1), alongwith poster paper and glue. Working in small groups of 2-4 people, students match each graph card to its correctinterpretation. The end product is a poster, which can beshared with the class to discuss the approach taken by thestudents.

Similarly, an example of a handwriting-intensive problemsolving CC is the ‘Sampling and Estimating: Counting Trees’lesson. This lesson requires students to estimate the num-ber of different types of trees in a tree farm without countingthem (See Figure 2). Students must choose an appropriatesampling strategy and use that to estimate the number ofeach type of tree. Students are required to write down thereasoning behind their solutions and may also draw on theprovided tree farm.

The Classroom Challenges promote several classroompractices that have been shown to improve long-term learn-ing outcomes:

• Collaborative Learning, discussion and reflection[5]• Emphasis on rich problem solving tasks[4]• Rich formative feedback, followed by re-engagement

with tasks[1]

Recently, an external evaluation [3] found measurable im-provements from using the CCs in Kentucky classrooms,reporting learning gains on state exams equivalent to anadditional 4.6 months of schooling.

The FACT SystemThe Classroom Challenges (CCs) in their original formatare entirely paper-based, allowing them to be used in anyclassroom without the need for technology. However, theteachers’ workload is high because they are required to

Late-Breaking Work: Collaborative Technologies #chi4good, CHI 2016, San Jose, CA, USA

1160

analyze complex student work and plan a productive com-ment upon it, all within seconds of walking up to a studentor group. To address these issues, we have developed theFACT (Formative Assessment using Computational Tech-nologies) system. FACT is intended to help teachers byanalyzing student work and making suggestions to teach-ers about what to say to students. To accomplish this, FACTlets students solve the assigned problems by writing andinteracting with the lesson materials on an android tablet,working either alone or in small groups as dictated by theteacher. Similarly, the teacher can use her tablet to select alesson1, assign an appropriate activity, create and managegroups of students, collect student work, view it, providefeedback on it, share it with the whole class and return it tostudents. FACT uses the Samsung Galaxy Tab 10 (2014edition) as its work platform, because this was one of thebetter options combining stylus and touch input at the timeof the project’s inception. A central server application han-dles wireless communication between student and teachertablets and acts as data storage. The FACT prototype hasthe following goals:

1. Do No Harm: FACT should not cause any loss oflearning opportunities compared to the paper version.

2. Generality: FACT should be applicable to a range oftasks and activities, not just a few selected ones.

3. Teacher Assistance: FACT should record and ana-lyze student work, identifying learning opportunitiesand points of intervention for the teacher to provideformative assessment.

4. Unconstrained Input: FACT should not restrict stu-dents’ ability to express themselves, so that it canunderstand their work easily.

Figure 2: Tree farm andassociated questions from the‘Sampling and Estimating:Counting Trees’ lesson.

1FACT lessons are the digital equivalent to paper-based ClassroomChallenges.

The fundamental unit of student work in FACT is a studentposter. Every FACT lesson is modeled as a sequence ofclassroom activities, in which students edit posters, work-ing alone or in small groups, as dictated by the lesson plan.The FACT system includes 9 lessons converted from paper-based Classroom Challenges. These 9 lessons comprisesufficient diversity to represent all the CCs and are amongthe most-downloaded materials on the Mathematics As-sessment Project’s website. For each activity in our cho-sen CCs, we have created an equivalent electronic poster.Although students can write on the poster itself, most ofthe work is done by writing on and/or arranging cards onthe poster. Figure 3 shows the poster for the ‘InterpretingDistance-Time Graphs’ lesson in FACT.

The FACT Student InterfaceThe FACT student application presents a working area witha menu bar when students log in (Shown in Figure 3). Stu-dents can log in simply by providing a username. The workarea contains two side-by-side viewports, with a left view-port for viewing prior work and a right viewport for editingthe currently assigned or opened poster. The left side ofthe menu bar displays the username for the student, thename of their current group, and a SYNC button. When theteacher assigns an activity to the whole class, the SYNCbutton on each student tablet is highlighted in red. Tappingit downloads the appropriate poster (unedited) from theFACT server. The right side of the menu bar has options forchanging the size and color of the pen, choosing an eraser,and for undoing/redoing student actions. Additionally, thereis a drop down menu for adding new cards. FACT allowsstudents to add four different types of cards: unlined, lined,graph and table cards (See Figure 4). FACT implementsaccess control at the card level to prevent simultaneousedits to the same card by group members. The first group

Late-Breaking Work: Collaborative Technologies #chi4good, CHI 2016, San Jose, CA, USA

1161

Figure 3: A screenshot of the FACT student application.

member to start editing the card2 locks it. Locked cards areshown with a padlock icon to other group members. Thelock is automatically released after 5 seconds of inactivity.

Students edit posters by using a combination of pen andtouch input. When working alone, all student actions aretransferred wirelessly to the FACT server and are saved inits database. When working in groups, all student actionsare also transmitted to all their group members. This givesgroup members the illusion of simultaneously editing thesame underlying poster. FACT automatically persists allstudent work, in order to speed recovery in case of hard-ware failure due to broken styli, and/or discharged batteries.

Figure 4: Different types of cardssupported by the FACT system.Each card includes controls forpin/unpin, changing color, andcontrol points for resizing.

The FACT Teacher InterfaceThe FACT teacher application has more functionality thatthe student application. The teacher’s user interface con-sists of four different views: Activity, Organize, Review, andMy Sheet (See Figure 5). The teacher can switch between

2Edits to a card include moving, resizing and writing.

different views by selecting a pane from the menu bar. TheActivity screen allows teachers to select an available les-son, displaying its activities in a tabular format (See Fig-ure 7). Teachers can select an activity by tapping a row.This causes the SYNC button to be highlighted for all stu-dents and groups in the class. The My Sheet screen allowsteachers to create their own solutions to any of the posters,which can be shared with students at the teacher’s discre-tion. The Review screen allows teachers to download stu-dent work from the FACT server. They can then take theteacher tablet home in order to provide comments on thestudents’ work.

The Organize pane allows teachers to manage the state ofthe classroom. They can create classes3, organizations4,and create, edit, and delete groups of students. The menubar also provides options for projecting the teacher’s screenvia screen mirroring and the ability to pause the entire

3A class is a set of student names.4An organization is a set of groups within a class.

Late-Breaking Work: Collaborative Technologies #chi4good, CHI 2016, San Jose, CA, USA

1162

Figure 5: A screenshot of the Organize screen in the FACTteacher application.

class. Students and groups are represented as cards (SeeFigure 5). The teacher can drag and drop student cardsonto group cards to group them. Alternatively, teachers canenable ‘Open Enrollment’ from the menu bar, allowing stu-dents to self-group. Disconnected or absent students areshown as gray cards. Students who have not synced withthe assigned classroom activity are highlighted in an or-ange color. At any time, the teacher can tap on a studentor group card to peek into their work. The FACT serverincludes a simple analysis system for card arrangementposters. This module works by monitoring interesting re-gions in all card arrangement posters. It can detect incor-rect pairings or placement of cards, map the mistake to anunderlying misconception and send an alert to the teachersuggesting appropriate feedback to be shared with the stu-dent. Alerts are shown on the Organize screen as numerichighlights on student and group cards (Shown in Figure 6).

Figure 6: A screenshot of theOrganize screen with an alertavailable for a group card.

Figure 7: A screenshot of theActivity screen showing theactivities in a selected lesson.

Figure 8: An alert message shownto the teacher while peeking. Theregion of the poster relevant to thealert is highlighted in a dotted redrectangle.

To view the message associated with an alert, teachersmust peek into the student/group’s poster by tapping on its

card. While peeking, FACT shows the most recent alert tothe teacher. If the student/group is still working in the sameregion as the mistake, the alert is rendered in blue color, in-dicating that it is still relevant. However, if the student/grouphas moved to a different region5, the alert is rendered ina gray color, indicating that a teacher intervention mightcause a shift in engagement from the student/group’s cur-rent task. The choice to intervene is left to the teacher, inorder to maintain flexibility. Teachers can tap the alert icon(See Figure 8) to see the detailed message to be displayedon the student/group poster. They can then choose to sendthe message, delete it or leave it on top of the queue. Whilepeeking, teachers can also directly write on the poster andadd their own cards. Teacher cards and pen utilize a shadeof red color that students cannot access, thus making thedistinction clear.

Discussion and Future WorkThe FACT system is not a traditional Intelligent TutoringSystem [6] because it does not give feedback and hintsdirectly to the student. Instead, FACT aims to be an unob-trusive helper in a classroom, helping teachers in craftingeffective learning experiences, and utilizing technology toidentify interesting points of intervention. We see this as anew design space for classroom systems. There has beensome work in this domain, with notable systems like GroupScribbles [2], but existing approaches usually do not includeanalysis of student work to highlight opportunities for inter-vention. The current version of the FACT system includessimple analysis for card arrangement posters. We are al-ready extending the range of the analysis module by in-corporating handwriting recognition and Natural LanguageProcessing.

5This is detected by examining the positions of student inputs (penand touch) with respect to the predefined regions of interest in the poster.

Late-Breaking Work: Collaborative Technologies #chi4good, CHI 2016, San Jose, CA, USA

1163

We have conducted seven classroom trials during 2015 inmiddle school classrooms in California and Britain. Thesehave enabled us to refine the teacher and student inter-faces to their current form. One of the major drawbacksof the current prototype is its pre-class setup time. Beforeevery class, the FACT application must be set up on ap-proximately 30 tablets, which must be taken to a schoolalong with a server machine, a wireless router6 and a dis-play adapter to support screen mirroring to a projector. Thisis a logistical problem which is acceptable for a researchprototype but must be overcome to increase the chancesfor large scale adoption.

Our trials indicate that the initial class setup7 takes 10-15minutes in a 50 minute lesson, which is too high. In succes-sive lessons, this time cost is negligible as the informationfrom the first session is preserved. To overcome the highinitial setup time, we have given teachers the ability to allow‘Open Enrollment’ (See Figure 5), which allows studentsto quickly choose groups themselves. The ability to pausethe entire class has also proved extremely popular in ourclassroom trials, serving as a convenient way for teacher toget students’ attention. We have also found that studentsusually discover the user interface on their own by exper-imenting for a few minutes, indicating that the interface iseasy to use. For teachers, the abilities to pause the class,projecting their screen and not having to deal with papermaterials have proven popular. Lastly, our analysis system,though an early stage prototype, is clearly helpful to theteachers in providing feedback to the students. This is themajor avenue of future work for the FACT system.

6We use our own router due to variance in school wireless facilitiesand to avoid conflicts with school IT policies.

7Initial setup requires a teacher to create a class, create an organiza-tion, and create groups of students.

AcknowledgmentsThe FACT project is supported by the Bill and MelindaGates Foundation under grant OPP10612881.

References[1] Paul Black and Dylan Wiliam. 1998. Assessment and

classroom learning. Assessment in education 5, 1(1998), 7–74.

[2] Yannis Dimitriadis, Juan Ignacio Asensio-Pérez,Davinia Hernáez-Leo, Jeremy Roschelle, John Brecht,Deborah Tatar, S Raj Chaudhury, Chris DiGiano, andCharles M Patton. 2007. From socially-mediated totechnology-mediated coordination: A study of designtensions using Group Scribbles. In Proceedings ofthe 8th international conference on Computer sup-ported collaborative learning. International Society ofthe Learning Sciences, 184–186.

[3] Joan Herman, Scott Epstein, Seth Leon, DeborahLa Torre Matrundola, Sarah Reber, and Kilchan Choi.2015. Implementation and Effects of LDC and MDC inKentucky Districts. (2015).

[4] Alan H Schoenfeld. 1992. Learning to think mathe-matically: Problem solving, metacognition, and sensemaking in mathematics. Handbook of research onmathematics teaching and learning (1992), 334–370.

[5] Malcolm Swan. 2006. Collaborative learning in mathe-matics. A Challenge to our Beliefs (2006).

[6] Kurt Vanlehn. 2006. The behavior of tutoring systems.International journal of artificial intelligence in educa-tion 16, 3 (2006), 227–265.

Late-Breaking Work: Collaborative Technologies #chi4good, CHI 2016, San Jose, CA, USA

1164