Device Comparability: Apples to Apples? - Confex · Device Comparability: Apples to Apples? ......
Transcript of Device Comparability: Apples to Apples? - Confex · Device Comparability: Apples to Apples? ......
Device Comparability: Apples to Apples?
Laurie Laughlin Davis, Ph.D. CCSSO National Conference on Student Assessment
June 26, 2014
3
Device Form Factor
o What is “Form Factor”? oSize, style and shape
o Layout and position of major functional components
oMethod of interaction (mouse, touchscreen, keyboard, etc.)
o Devices with similar form factors can be expected to have higher degrees of comparability o Example: Desktops and laptops (see Powers & Potenza, 1996)
o Example: 10” Android tablets and 10” iPad tablets
o Devices with less similar form factors can be expected to have lesser degrees of comparability o Example: Desktops and smartphones
Copyright © 2013 Pearson Education, Inc. or its affiliates. All rights reserved.
Pearson’s Digital Devices Study History
Qualitative Writing usability study (March 2012) Screen size/item type usability study (Nov 2012) Device features usability study (Feb 2013)
Quantitative Written composition study (Spring 2013)
Tablet to computer comparability study (Spring 2014) 4
6
Conclusions o No differences observed in student writing performance
across study conditions, but consider: o Essay responses relatively short (by design)
o ~275 on average for high school
o ~225 on average for 5th graders
o Essay prompts relatively short with no reference to stimuli like a reading passage
o Screen real estate taken up by onscreen keyboard less of an issue
o Low stakes for students (random drawing for a gift card) o Results may be different if any of these factors are
varied
Spring 2014 Study Design
CCSSO 2013 7
~1000 high school students
Random assignment to condition (computer or tablet)
7 schools from 5 Virginia school divisions volunteered Frederick Prince George Stafford Henrico Isle of Wight
Students were required to be enrolled in or have completed Algebra I, Biology, and English II coursework
Computers included laptops and desktops Tablets were 10” iPads 90 minute testing sessions
Study Design
9
Test Blueprint
MC Hot Spot
Drag and Drop
Fill In the
Blank
Multiple Select
MC
Inline Choice TOTAL
Reading 6 6 8 0 0 0 20 Mathematics 8 0 1 5 4 0 18
Science 12 2 2 2 0 3 21 TOTAL 26 8 11 7 4 3 59
44% 14% 19% 12% 7% 5% 100%
Study Test Form
Targeted data collection: February 2014
Actual data collection: April-June 2014
The Best Laid Plans…
Preliminary Results
Last day of data collection—June 6 Data files received—June 18 Data QC’ed and Phase I analyses completed—June 24 CCSSO presentation—June 26
CCSSO 2013 12
Phase I Results—Test Level
Preliminary Results Only—Interpret with Caution
Whole Test Tablet Computer N-count 478 471 Mean Raw Score 25.94 25.61 SD Raw Score 9.72 9.46 Mean % Correct 44% 43%
Phase I Results—By Subject Area
Preliminary Results Only—Interpret with Caution
Reading Tablet Computer N-count 478 471 Mean Raw Score 9.44 9.05 SD Raw Score 3.12 3.34 Mean % Correct 47% 45%
t=1.83; p=.0673
Phase I Results—By Subject Area
Preliminary Results Only—Interpret with Caution
Science Tablet Computer N-count 478 471 Mean Raw Score 9.11 9.12 SD Raw Score 3.69 3.38 Mean % Correct 43% 43%
Phase I Results—By Subject Area
Preliminary Results Only—Interpret with Caution
Mathematics Tablet Computer N-count 478 471 Mean Raw Score 7.38 7.43 SD Raw Score 4.52 4.47 Mean % Correct 41% 41%
Phase I Results—By Item Type
Preliminary Results Only—Interpret with Caution
Multiple Choice Tablet Computer N-count 478 471 Mean Raw Score 15.06 14.90 SD Raw Score 4.99 4.83 Mean % Correct 58% 57%
Phase I Results—By Item Type
Preliminary Results Only—Interpret with Caution
Drag & Drop/Multiple Select Tablet Computer N-count 478 471 Mean Raw Score 6.29 6.13 SD Raw Score 3.43 3.43 Mean % Correct 27% 27%
Phase I Results—By Item Type
Preliminary Results Only—Interpret with Caution
Fill In the Blank Tablet Computer N-count 477 471 Mean Raw Score 2.75 2.75 SD Raw Score 1.93 1.96 Mean % Correct 39% 39%
Phase I Results—By Item Type
Preliminary Results Only—Interpret with Caution
Inline Choice Tablet Computer N-count 477 471 Mean Raw Score 1.84 1.84 SD Raw Score 0.66 0.69 Mean % Correct 61% 61%
Motivation filtering Screens between
sections encouraged
students to do their
best, but…
Partial credit scoring Use of covariates State assessment scores Survey responses
Score distribution comparison Item-level analysis
Future Analysis Phases
Much analysis work yet to be done with 2014 data But no “show-stoppers” so far that would suggest we
should be concerned about testing on tablets No statistically significant differences between tablet and
computer conditions based on preliminary analyses By subject area or item type
Reading trending toward small significant effect favoring tablet—should monitor with additional data analysis But could also be “running away from” significance
Conclusions
With Thanks
Kathy Shannon—project manager extraordinaire!! Malena McBride Jadie Kong --Data analysis goddesses!!
CCSSO 2013 23