Using CEM Assessments to Diagnose and Remediate Language and Maths Difficulties [email protected] .
USING CEM SECONDARY DATA
description
Transcript of USING CEM SECONDARY DATA
USING CEM SECONDARY DATAPRACTICAL APPLICATIONS IN SCHOOL
APRIL 2011 Bristol Conference
Geoff Davies
CEM Secondary data includes:
• Baseline test data (developed ability)• ‘Predictive’ data including chances graphs• Value-added data• Attitudinal data• Curriculum assessments (Insight/Sosca)• PARIS software programmes
The use of this data needs to allow us to do our best to help every pupil to at least achieve if not exceed their potential.
It may challenge• The culture of ‘my’ school • Accountability policy• Expectations • Staff training in use of data and ability to cope with
data (data overload)• Integrating the data into school procedures,
storage, retrieval, distribution and access • Roles and Responsibilities
Carol Fitz-Gibbon 2001 British Psychological Society
It gradually dawned on me that providing the data to schools was the most important outcome of the effort, far more important than writing research papers…..
The provision of data to practitioners meant that they participated in the research. Indeed they were the only ones who knew the surrounding circumstances for their classrooms, their department, each pupil, each family, etc. They were the major players: the ones who could interpret and learn from the detailed data.
…..there is a need for teacher researcher posts on the Senior Management team with a brief to develop research that is useful. Given time in the timetable thousands of teachers could become active researchers….
Educational research should be a practical reality contacting scientific enlightenment not a mathematical weight lifting exercise
The sense and usefulness of what we are doing induces
creative thoughtfulness.
creative thoughtfulness
QUESTION
IS THIS STIFLED IN THE PRESENT EDUCATIONAL CLIMATE?
I keep six honest serving men
Who taught me all I know
Their names are What and Why and When
And How and Where and Who
Rudyard Kipling The Elephant’s Child
Can school leaders find the time?
Important but not urgent
Urgent but not important
Is it more like? 65%-80% or 15%
Is it more like? 15% or 60%
PRIORITISING
QUESTIONS INSPIRED BY CEM CENTRE DATA• How do we improve our professional judgement?• Do MIDYIS scores tell us more than we think?• Which ability range of pupils gets the best deal from our school?• Which ability range of pupils is served best by a particular subject department?• Can we do action research in our school using historical data?• Can standardised residuals tell us more about a department than just the + or -?• Does learning support work?• What do attitudinal surveys over time give us?• Can we compare standardised scores with comments made by teachers on reports?• Drilling down from SPC charts.• Which pupils are we succeeding with?• Which pupils are we not succeeding with?• What can be done about it?• Is there a pattern?• What are the gender issues?• What are the prior learning issues?• What can we learn from plotting standardised scores over time?• Can SOSCA help with boy/girl issues?• Is key stage 3 teacher assessment sufficiently rigorous?• How can ALIS be used to inform teaching and learning?
WHAT information do I want?WHY do I want it?WHEN do I need it?HOW do I collect it?WHERE can I find it?From WHO do I get it?
The CEM centre ticks many of these boxes for schools
A small selection of questions we will look at
1. Do Midyis scores tell us more than we think?
2. Which ability of students do most or least well in our school (using YELLIS data)?
3. Can we review post 16 pedagogy (using ALIS data)?
4. What did SOSCA teach us about Key Stage 3 assessment?
Do Midyis scores tell us more than we think?
Using Midyis baseline data
Using MidYIS IPRs Booklet.pdf
What do the sections of the test measure?
The Vocabulary component of the test is generally an important element for most subjects. For English, History and some Foreign Languages it is the best. However the Vocabulary score is perhaps the most culturally linked of all the scores. Those who have not been exposed to vocabulary-rich talk or a wide variety of reading material or whose first language is not English are unlikely to have developed as high a vocabulary score as they would have developed in a different environment.
Maths Score
The Maths score is well correlated with most subjects but is particularly important when predicting Maths, Statistics, ICT, Design Technology and Economics.
The Maths section has been designed with the emphasis on speed and fluency, rather than knowledge of Maths. Like the Vocabulary score, the Maths score is a good predictor of later academic performance.
Vocabulary Score
Non-Verbal Score
The Non-Verbal score is composed of the three sub tests: Cross-Sections, Block Counting and Pictures. The Non-verbal score is important when predicting Maths, Science, Design Technology Geography, Art and Drama. It provides a measure of the pupil’s ability in 3-D visualisation, spatial aptitude, pattern recognition and logical thinking. It can give an insight in to the developed ability for pupils for whom English is a second language
Skills Score
In the Proof Reading section pupils are asked to spot mistakes in the spelling, punctuation and grammar of a passage of text. eg mis-spelling of words like ‘there’ and ‘their’.
The PSA (Perceptual speed and accuracy) section asks pupils to look for matches between a sequence of symbols on the left and a number of possible choices on the right. Given enough time most pupils would probably get the answers correct but we are measuring how quickly pupils can find a correct match. The PSA section allows speed to be demonstrated free from the demands of memory.
The Proof Reading and PSA tests are tests for the modern world, and are designed to measure fluency and speed. They rely on a pupil’s scanning and skimming skills, skills that are desirable in examination situations.
Some pupils will display an IPR pattern with significant differences between one or two components of the MidYIS Test.
These can be the most interesting and possibly the most challenging pupils for mainstream classroom teachers.
Scenarios and anecdotal findings
It is when the IPR is placed in the hands of a teacher who knows that pupil that it becomes a powerful tool.
Confidence Limits
The pupil scored 114 on the Vocabulary section. The error bars range from about 105 to 123, about 9 points either side of the pupil’s score. If this pupil was to take this test afresh 100 times, we would expect that 95 of those times the pupil’s score would fall within the range denoted by the error bars
Are the scores significant? Relative to the Average Performance?
Performance in Vocabulary is significantly better than average performance and Maths performance is significantly below average. The error bars for the Non-verbal, Skills and Overall MidYIS scores do cross the line at 100 and hence the pupil cannot be considered to have performed significantly different to the average pupil overall.
Comparing Maths and Vocabulary Scores
The error bars for Vocabulary and Maths do not cross the line at 100 ( av. performance).
A SELECTION OF MIDYIS SCORES FOR ‘WATERLOO ROAD’ !!
Vocabulary Maths Non Verbal Skills MidYIS Score
St. Score Band St. Score Band St. Score Band St. Score Band St. Score Band
Surname Sex
A F 81 D 110 B 108 B 112 A 94 C
B F 128 A 107 B 105 B 94 C 120 A
C M 106 B 121 A 103 B 90 D 114 A
D F 107 B 84 D 96 C 107 B 96 C
E M 96 C 90 D 130 A 91 C 92 C
F F 86 D 86 D 120 A 74 D 84 D
G F 100 B 115 A 80 D 103 B 108 B
H F 121 A 96 C 114 A 86 D 111 A
I M 92 C 100 C 96 C 123 A 95 C
J M 100 C 105 B 100 C 99 C 102 B
K M 128 A 132 A 114 A 131 A 133 A
L M 76 D 70 D 74 D 73 D 71 D
What do I need to know/do to teach this (difficult) class of twelve pupils
Why would this be a very challenging class to teach?
These are real anonymous scores from a number of schools around the UK
Vocabulary scores significantly lower than other component scoresSecond language? Deprived areas? Difficulty accessing curriculum.? Targeted help does work. Seen in nearly all schools. Worth further diagnosis
Vocabulary scores significantly higher than other component scoresGood communicators. Get on. Put Maths problems in words?
Mathematics significantly higher than other scoresFrom Far East? Done entrance tests? Primary experience?
Mathematics significantly lower than other scoresPrimary experience. Use words and diagrams? Sometimes difficult to change attitude..
Low Mathematics scores with High Non-verbal ScoresUse diagrams. Confidence building often needed
Pupils with non-verbal scores different from others – High Non-verbal ScoresFrustration? Behaviour problems? Don’t do as well as good communicators or numerate pupils?
Pupils with non verbal scores different from others – Low Non-verbal ScoresPeak at GCSE? A level ?Pupils with low Skills scoresExams a difficulty after good coursework?High Skills ScoresDo well in exams compared with classwork?The Average PupilThey do exist!High scores throughoutAbove a score of 130 puts the pupil in the top 2% nationallyLow scores throughoutBelow a score of 70 puts the puil in the bottom 2% nationally
Sharing the MidYIS Information within School
Once you have received your MidYIS feedback you need to decide who will be privy to which information. Some schools decide to keep the data within the senior management team, others with Heads of Department and/or Heads of Year, some share with all staff and what about pupils and their parents?
Use you MIS systems to put the data where it matters
MidYIS data can be useful:
•to indicate reasons for student learning difficulties and may go some way to explain lack of progress, flag up causes for underachievement and even behaviour problems.
•for all teachers and support staff. It can help to support professional judgement and give a better understanding of the progress students make at school and their potential later performance.
•to refer to for pupil reviews, writing reports, meeting parents, monitoring progress and interim assessments.
2. Which ability of students do most or least well in our school (using YELLIS data)
Which ability range of pupils gets the best deal from our school?Which ability range of pupils is served best by a particular subject department?Can standardised residuals tell us more about a department than just the + or -?
Using standardised residuals in a different way
0.2
-0.6 -0.5-0.2 -0.3
-1.3
0.1
-0.1
0.1
-1.1
0.1
-0.1-0.3
0.4
-0.5 -0.5-0.9
-0.5
0.3
-4.0
-3.0
-2.0
-1.0
0.0
1.0
2.0
3.0
4.0
Art
& D
esig
n
Des
ign
& T
echn
olog
y
Eng
lish
Eng
lish
Lite
ratu
re
Fre
nch
Geo
grap
hy
His
tory
Mat
hem
atic
s
Med
ia S
tudi
es
Mus
ic
Rel
igio
us S
tudi
es
Sci
ence
: G
CS
E
Sci
ence
: G
CS
E A
dditi
onal
SC
Art
& D
esig
n
SC
Geo
grap
hy
SC
His
tory
SC
Rel
igio
us S
tudi
es
Voc
App
lied
Sci
ence
Voc
Hea
lth &
Soc
ial C
are
Ave
rag
e S
tan
dar
dis
ed R
esid
ual
CONTRAST THIS
0.0 0.0
0.4
0.9
0.10.2
1.4
0.4 0.3
-0.1
0.5 0.6
0.2
0.8
-0.1 -0.2
0.3 0.4
0.9
-4.0
-3.0
-2.0
-1.0
0.0
1.0
2.0
3.0
4.0
Art
& D
esig
n
Bus
ines
s S
tudi
es
Des
ign
& T
echn
olog
y
Dra
ma
Eng
lish
Eng
lish
Lite
ratu
re
Fre
nch
Geo
grap
hy
Ger
man
His
tory
Hom
e E
cono
mic
s
ICT
Mat
hs
Mus
ic
Phy
sica
l Edu
catio
n
Rel
igio
us S
tudi
es
Dou
ble
Sci
ence
Wel
sh
SC
IC
T
Ave
rag
e S
tan
dar
dis
ed R
esid
ual
WITH THIS
yellis exercise.doc
As on the last two slides CEM provides value added charts using average standardised residuals for departments
We are going to show how standardised residuals can be used in a different way in your school
1. Review of post 16 pedagogy (using ALIS data)
SUBJECT A 2008
Why the improvement?
PEDAGOGY….ALIS surveys
SUBJECT A 2005
SUBJECT A 2007
SUBJECT A 2008
DO NOT GET TOO EXCITED!
We have made a comparison of perceived teaching methods as analysed by ALIS in 2004-5 with those in 2007-8. Some subject areas have appeared to change their methods radically. Others have not. Though the samples are small it is an interesting exercise to try to correlate it with the departments statistical process charts over that period. One would like to say that changes in the variety of teaching methods result in improvement but the evidence is a little tenuous so far.
SUBJECT
Exercises
exampapers
essays
Reading
discussions
notes dictated
own notes
handouts
Practical work
Using AV
IT
Researching
Presenting
helping students
original work 2005 to 2008
SIGNIFICANTCHANGE
ART YES L L M L LBIOLOGY NO L LBUSINESS YES M L M LCHEMISTRY NO LCOMPUTING YES L L M M L MDRAMA NO M L MFOOD YES L L M M M MPRODUCT DESIGN YES L L L M LENGLISH NO LFRENCH YES M M M L MGEOGRAPHY YES M M M MHISTORY NO L LICT YES L L L L M M MMATHS NO M M MMUSIC YES M M M M M M MPE YES M M M M M M MPHYSICS YES M M M M M M LRE NO L MWELSH YES M M M M M
YES 11NO 7MORE (M) 54 5 4 1 3 4 5 6 5 2 3 2 3 2 6 3LESS (L) 29 3 3 2 6 3 1 2 1 2 1 0 0 2 3 0
4. What have we learned from SOSCA?
GENDER ISSUES DYFFRYN TAF KEY STAGE 3
BOYS
BOYS
BOYS
GIRLS
GIRLS
GIRLS
OVERALL
OVERALL
-0.4
-0.2
0
0.2
0.4
0.6
0.8
English Maths Science
SUBJECT
AV
ER
AG
E S
TA
ND
AR
DIS
ED
RE
SID
UA
LS
MID
YIS
TO
KS
3
BOYS
95%CONFIDENCE
95% CONFIDENCE
GIRLS
95% CONFIDENCE
95% CONFIDENCE
OVERALL
95% CONFIDENCE
95% CONFIDENCE
GRAPH 1
AVERAGE STANDARDISED RESIDUALS BOYS/GIRLS MIDYIS TO KEY STAGE THREE TEACHER ASSESSMENT
GENDER ISSUES DYFFRYN TAF SOSCA
BOYS
BOYS
BOYS
GIRLS
GIRLS
GIRLS
OVERALL
OVERALL
OVERALL
95% CONFIDENCE
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
Reading Maths Science
SUBJECT
AV
ER
AG
E S
TA
ND
AR
DIS
ED
RE
SID
UA
LS
MID
YIS
TO
SO
SC
A
BOYS
95%CONFIDENCE
95% CONFIDENCE
GIRLS
95% CONFIDENCE
95% CONFIDENCE
OVERALL
95% CONFIDENCE
95% CONFIDENCE
GRAPH 2AVERAGE STANDARDISED RESIDUALS BOYS/GIRLS MIDYIS TO SOSCA
GRAPH 1 AVERAGE STANDARDISED RESIDUALS BOYS/GIRLS MIDYIS TO KEY STAGE THREE TEACHER ASSESSMENT
GENDER ISSUES DYFFRYN TAF KEY STAGE 3
BOYS
BOYS
BOYS
GIRLS
GIRLS
GIRLS
OVERALL
OVERALL
-0.4
-0.2
0
0.2
0.4
0.6
0.8
English Maths Science
SUBJECT
AV
ER
AG
E S
TA
ND
AR
DIS
ED
RE
SID
UA
LS
MID
YIS
TO
KS
3
BOYS
95%CONFIDENCE
95% CONFIDENCE
GIRLS
95% CONFIDENCE
95% CONFIDENCE
OVERALL
95% CONFIDENCE
95% CONFIDENCE
GRAPH 2AVERAGE STANDARDISED RESIDUALS BOYS/GIRLS MIDYIS TO SOSCA
GENDER ISSUES DYFFRYN TAF SOSCA
BOYS
BOYS
BOYS
GIRLS
GIRLS
GIRLS
OVERALL
OVERALL
OVERALL
95% CONFIDENCE
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
Reading Maths Science
SUBJECT
AV
ER
AG
E S
TA
ND
AR
DIS
ED
RE
SID
UA
LS
MID
YIS
TO
SO
SC
A
BOYS
95%CONFIDENCE
95% CONFIDENCE
GIRLS
95% CONFIDENCE
95% CONFIDENCE
OVERALL
95% CONFIDENCE
95% CONFIDENCE
Using MIDYIS and SOSCA puts the school in a strong position to improve its professional judgment of teacher assessments at Key stage 3. Statutory testing disappeared in Wales some five years ago.Comparing the value added for MIDYIS to SOSCA and MIDYIS to KS 3 teacher assessment shows up some interesting data
Schools who depend on teacher assessment data only to measure value added from Key stage 3 to Key stage 4 need to be aware of the pitfalls. The use of SOSCA data in this exercise highlights that
http://www.tes.co.uk/article.aspx?storycode=6062337see
Subjective views of pupils - as well as pressure from parents - make model unreliable, warns Professor Tymms.
The differences appear to relate to the types of assessment used in the various subject areas. English and Welsh use extended writing for teacher assessment which is more likely to have subjective judgments. What we have learnt from this is that despite a moderation process built on portfolios of work for teacher assessment, it is not sufficient in isolation.Computer adaptive tests such as SOSCA and the resulting value added information from MIDYIS are more informative in a diagnostic sense than levels produced by teachers for statutory assessment.SOSCA has also been used to investigate any changes in reading from baseline testing. A high correlation was found between the London Reading score given to pupils on entry the MIDYIS score and the SOSCA reading test.
PITFALLS
1. Tracking developed ability measures over time.
2. Looking at average standardised residuals for teaching sets.
3. Effect of one result in a small group of students
REGRESSION TOWARDS THE MEAN
Pupils with high MidYIS scores tend to have high SOSCA scores but not quite as high. Similarly pupils with low MidYIS scores tend to have low SOSCA scores, but not quite as low. It is a phenomenon seen in any matched dataset of correlated and normally-distributed scores, the classic example is a comparison of fathers' and sons' heights. Regression lines reflect this phenomenon - if you look at the predictions used in the SOSCA value-added you can see that for pupils with high MidYIS scores their predicted SOSCA scores are lower than their MidYIS scores, whereas for pupils with low MidYIS scores their predicted SOSCA scores are higher than their MidYIS scores.
DIFFERENCE SOSCA-MIDYIS MATHS
-50
-40
-30
-20
-10
0
10
20
50 70 90 110 130 150 170
MIDYIS MATHS STANDARDISED
SO
SC
A-M
IDY
IS D
IFF
ER
EN
CE
DIFFERENCE
Stu
den
ts
Ad
mis
sio
n N
o.
Mat
hs
Tes
t K
3 W
a
Yel
lis
Sco
re
Yel
lis
Ban
d
YE
LL
IS M
AT
HS
BA
ND
PR
ED
ICT
ION
MA
TH
S
YE
LL
IS G
CS
E P
RE
DIC
TIO
N M
A
WJE
C/G
CS
E 0
1840
1 R
esG
F
WJE
C/G
CS
E 0
1840
2 R
esG
F
WJE
C/G
CS
E 0
1840
3 R
esG
F
GC
SE
Sta
nd
ard
Res
idu
al M
a
a 90019 6 62 A A B B A 1.20
b 90090 7 B B
c 90045 6 63 A B B B B 0.10
d 90063 7 64 A A B B B 0.10
e 90166 6 48 B B B C C 0.40
f 90123 7 70 A A A A B -0.40
g 90129 6 47 C C B C C 0.50
h 90146 6 59 B B A B A 1.40
I 90047 7 62 A A B B B 0.20
j 90115 7 67 A A * A A* 1.70
k 90004 6 46 C B B C B 1.50
l 90164 7 65 A A A B A* 1.90
m 90099 7 70 A A A A A* 1.50
n 90011 7 61 A A A B A 1.30
o 90112 7 66 A A B A A 0.80
p 90058 6 70 A A B A A 0.50
q 90150 7 72 A A A A A 0.40
r 90127 6 52 B B B C B 1.00
s 90030 6 58 B B B B B 0.50
t 90050 7 71 A A A A A 0.40
u 90016 6 69 A A B A B -0.40
v 90174 7 74 A A A A A 0.20
w 91165 6 62 A B B B B 0.20
x 90109 7 63 A B B B B 0.10
y 90138 7 47 C B B C B 1.40
z 90122 7 60 A A * B A 1.30
ab 90009 7 60 A A A B A 1.30
ac 90169 7 79 A A * A* A -0.20
ad 90153 6 56 B B B B B 0.70
ae 90010 7 64 A B B B A 1.00
af 90154 7 61 A C B B B 0.30
Total 1323 1868 109 105 201 156 12 190 20.90
Number of Results 31 30 30 30 31 30 2 29 30
Mean 42.68 62.27 3.63 3.5 6.48 5.2 6 6.55 0.70
Mean Grade 6.00 B B B B B B
Marksheet Name : SUBJECT REVIEW
Marksheet Group : 11S1
Export Date : 04/10/2005
CLASS REVIEW
BEWARE PITFALLS
INTERPRETATION
SexScore (Band)
Raw Residual
Standardised
Residual REVISEDM 53 (B) 5.4 (B/C) 6 (B) 0.6 0.5 0.5M 38 (C) 4.5 (C/D) 3 (E) -1.5 -1.1 -1.1F 36 (D) 4.4 (C/D) 3 (E) -1.4 -1.0 -1.0M 48 (C) 5.1 (C) 5 (C) -0.1 -0.1 -0.1F 52 (B) 5.3 (B/C) 6 (B) 0.7 0.5 0.5F 65 (A) 6.1 (B) 7 (A) 0.9 0.7 0.7M 70 (A) 6.4 (A/B) 3 (E) -3.4 -2.5 M 38 (C) 4.5 (C/D) 4 (D) -0.5 -0.4 -0.4F 40 (C) 4.6 (C/D) 5 (C) 0.4 0.3 0.3F 70 (A) 6.4 (A/B) 7 (A) 0.6 0.4 0.4F 44 (C) 4.8 (C) 6 (B) 1.2 0.9 0.9M 56 (B) 5.6 (B/C) 5 (C) -0.6 -0.4 -0.4
5.3 (B/C) 5.0 (C) -0.3 -0.2 0.0
Predicted Grade Achieved Grade
SUBJECT M
creative thoughtfulness
PLEA
DON’T LET THE SYSTEM DESTROY THIS
USING CEM SECONDARY DATAPRACTICAL APPLICATIONS IN SCHOOL
APRIL 2011 Bristol Conference
Geoff Davies