1

Inter-Subject Comparability

Helen Myers and David Blow

Based on ASCL June ppt with Joint Letter etc

+ updated graphs + survey29 nov 17

• A brief history of Inter-Subject Comparability (ISC)

• Ofqual consultation (Dec ‘15)

• Ofqual position (April ‘17)

• Opportunity for A/L Bi, Ch, Ph and A/L Fr, Gn, Sp

• Action needed for GCSE Fr, Gn, Sp

• Evidence

2

• Official position for a long time was that all subjects were of equal difficulty and that a grade C (say) in one subject was the same standard as a grade C in another

• However, long-standing concern that this was not the case in practice - research by Nuttall in 1974; Cockcroft Report on Maths in 1982 highlighted that 25% of cohort passed O-level Maths whereas 33% passed O-level English

• ALIS (Carol Fitzgibbon) in 1992 showed clearly large variations in grading at A/L

A Brief History of ISC - 1

3

• 2004: Peter Tymms and Robert Coe from CEM in Durham coined term “severe grading” particularly with reference to A/L

• In 2008, QCA / Ofqual seminar on Inter-subject Comparability (see ISC Paper on official views at the time) – Helen Myers and David Blow presented on behalf of ML community (together with STEM and English reps)

• Robert Coe presents to ASCL Conference• Then pause until Ofqual launch formal

consultation on ISC in Feb 2016

A Brief History of ISC - 2

Helen Myers (Past President, ALL & Asst Head, The Ashcombe School) website: www.all-london.org.uk

4

• “New research published today (18 December) examines whether different GCSE and A level subjects are comparable to each other and whether a better alignment could be achieved. The working papers identify the policy options that Ofqual could adopt in relation to the comparability of different subjects.”

Ofqual consultation

(Dec ‘15)

https://www.gov.uk/government/news/can-different-gcse-and-a-level-subjects-be-compared-accurately

5

• 6 working papers, 1. Comparability of Different GCSE and A level Subjects in England: An introduction.2. Inter-Subject Comparability: A Review of the Technical Literature.3. Inter-Subject Comparability of Examination Standards in GCSE and GCE.4. Inter-Subject Comparability: An International Review.5. A Recent History of Regulatory Perspectives on Inter-Subject Comparability in England.6. Exploring Implications of Policy Options Concerning Inter-Subject Comparability.

• conference in Feb ’16 with presentations incl from Charles Tracy (Institute of Physics) and Nick Mair (Independent Schools Modern Language Association)

Ofqual consultation (Dec ‘15)

6

Q. Should Ofqual have an approach to ISC?1. “No action should be taken” – this would perpetuate all the existing problems2. “Action to achieve inter-subject comparability” – probably too far for ALL subjects; need bands/groupings3. “Post hoc action to achieve inter-subject comparability” –too remote and does not address the reality in the classroom4.“Action to achieve an alternative to inter-subject comparability” – same norm-referencing for all subjects does not take account of cohort ability.

Ofqual: Question + 4 options

Need a pragmatic approach to tackling the anomalies. Vital to take action, and so need to be flexible

?

7

• “In April 2016, leaders of a number of high-profile science organisations wrote to Ofqual expressing disagreement with the finding in some of the working papers Ofqual had produced on inter-subject comparability. Ofqual then undertook research based on the letter to look at the progression of students from GCSE to A level.”

Ofqual website

Letter from science organisations

https://www.gov.uk/government/publications/inter-subject-comparability-2015-to-2016

8

• Written by Dennis Opposs, and presented to Ofqual Board Nov 2016; published April 2017

• Ofqual moved quickly at point of publication with purdah imminent and combined with announcement on A/L ML native speaker research

Ofqual policy position

https://www.gov.uk/government/publications/inter-subject-comparability-2015-to-2016

https://www.gov.uk/government/news/setting-grade-standards-in-a-level-modern-foreign-languages

9

“Ofqual has decided that the challenges apparent in conceptualising and measuring how subjects may compare means that it will not try to take coordinated action to align grade standards across the full range of GCSE or A level subjects. This decision follows more than a year of public discussion and debate about a number of potential policy options.” “However, the regulator will consider making one-off adjustments to standards where there is a compelling case. Ofqual has already begun conversations with subject communities in relation to A levels in physics, chemistry and biology, as well as MFL.”

Ofqual policy position - 1

10

“Sally Collier, Chief Regulator, Ofqual, said: ‘Everybody expects that grade standards between exam boards in any particular subject should be comparable. However, inter-subject comparability is more difficult to consider. We are extremely grateful to the stakeholders who have offered their opinions and the many researchers who have helped us reach our policy decision. It, combined with the actions we are taking this summer specifically in relation to A level languages, will mean we are better placed than ever to ensure standards in GCSEs, AS and A levels are appropriate.’”

Ofqual policy position - 2

11

• Suggest active support for science organisations and ML organisations in making “compelling case” for one-off adjustments to A/L Bi, Ch, Ph and A/L Fr, Gn, Sp to take up the specific invitation offered by Ofqual

• Urge Ofqual to also allow “compelling case” for one-off adjustments to GCSE Fr, Gn, Sp

• What is the evidence?

Reflection

12

• Peter Downes raised concerns over time (from O-level), leading to recommendation in the Nuffield Report

• Series of presentations from 2005 onwards by Helen Myers and David Blow using official data (DfES/DCSF/DfE, Ofsted, QCA/Ofqual etc)

• Continuing concern in ML community, esp from 2004 when Languages no longer compulsory at KS4 and GCSE numbers go into freefall

• Series of meetings by ALL, ISMLA, ASCL etc with ministers, QCA, exam boards

• ISMLA, ALL, ASCL and HMC have worked closely together on severe grading in MFL for over 10 years now – state & indept; teachers & Heads

Severe Grading in MFL- brief history

Peter Downes (former Chief ML examiner, former

Head, former President ALL & SHA now ASCL)

.

13

• Initially, 2005 – 2008 during Dearing Review etc, phrase was “perception”

• 2008 Ofqual seminar on Inter-subject Comparability (see ISC Paper on official views at the time) – Helen Myers and David Blow presented on behalf of ML community (together with STEM and English reps) – see later slides

• So focus then was on convincing about the reality; that has now been accepted and evidenced in the Ofqual ISC papers

Severe Grading in MFL- brief history (2)

14

• National outcry at drop in numbers GCSE Aug 06 - Alan Johnson sets up Dearing Review

• “Mr Johnson wants to see what more can be done to encourage14-16 year olds to study GCSE or other language courses leading to a recognised qualification.”

DfES Press Notice 2006/0144 Oct 06

• Meeting Nov 06 with Lord Dearing–Idea of broad comparability with Maths grading–Awareness of public and political sensitivity

Dearing Review

15

• Led to recommendation in Consultation Report (Jan 07) confirmed in Final Report (Mar 07) to have definitive study published on “perception” of severe grading (with implicit corollary that there should be action...) - the consultation had “found strong confirmation of the view that award of grades is more demanding than for most other subjects”

Dearing Review

16

• Dr Coe, of Durham's curriculum, evaluation and management centre, believed such trends were repeated in other years, but insisted "the question of difficulty is not about content of the subject ... It is purely about the examination and grading process."

• David Willetts, Conservative education spokesman, said: "If there is evidence modern languages is tougher than other GCSEs, then that is something that has to be corrected. They should be the same level of challenge as traditional academic GCSEs."http://education.guardian.co.uk/gcses/story/0,,2031769,00.htmlJames Meikle, education correspondent Monday March 12, 2007

Dearing Review – press - 2007

17

• Grade standards in GCSE modern foreign languages• February 2008• QCA/08/3570

12. To test its practical application we asked AQA, Edexcel and OCR to calculate what changes there would be to their 2007 GCSE French results if the grades were based only on the relationship between candidates’ key stage 3 test scores and their results in GCSE mathematics. The analyses indicated that the changes would be marked at the higher grades. About half the candidates presently awarded a grade B would gain a grade A as the threshold mark or performance standard for a grade A would have to move down by about half a grade width. There would be a similar effect at grade C.

QCA Report – Feb 2008

18

• Dr John Dunford, General Secretary of ASCL said: “The main problem is that modern language GCSEs are graded more severely than other subjects and it is extremely disappointing that the QCA, while recognising the problem, does not intend to bring the grading of GCSE languages into line with mathematics, English and other similar subjects.”

• “This is not an issue of ‘dumbing-down’ language GCSEs, it is levelling the playing field with other similar subjects .....not to maintain a grading standard that was unfair to begin with.

Joint ASCL, ALL, ISMLA statement re QCA report 2008 in response to Dearing

20 Feb 2008

19

• Inter-subject comparability: research documents

www.all-london.org.uk/severe_grading.htm

Citations in Ofqual docs

• "Inter-Subject Comparability: A Review of the Technical Literature- ISC Working Paper 2“

• Page 10: "In more recent years, the debate on inter-subject comparability has been driven largely by subject groups. Those with an interest in languages (Myers, 2006; Dearing and King, 2007) and mathematics and sciences (Coe et al., 2008; Royal Society, 2008) have been particularly vocal.“

• Page 45: Myers, H. (2006) The ‘severe grading’ of MFL grades at GCSE and A level. London, Association for Language Learning.

20

• JCQ Report into M.L. (July ‘14) – GCSE and A/L• Paper 1: The decline in uptake of A-level Modern Foreign Languages: literature review

(Dr Debra Malpass)• Paper 2: Attitudes towards Modern Foreign Languages (Ipsos MORI)• Paper 3: A-level Modern Foreign Languages: why so few A*s? (Ben Jones, Beth Black

& Christine Wong)• Paper 4: A-level Modern Foreign Languages: assessment instruments and script

analysis (Beth Black)

• Ofqual Report (Sep’14) – A/L esp A*• JCQ report and Ofqual report into ML looked in detail at all

the factors from content through to assessment (questions) to marking to grading that contribute to variability (some typical, others undesirable)

Official Reports

http://www.jcq.org.uk/media-centre/news-releases/mfl-review-press-noticeOfqual Report: http://ofqual.gov.uk/news/improvements-made-level-foreign-languages/

21

• Subtle distinction• NOT looking at the judgemental boundary A/B• BUT looking at the proportion of A* relative to A + A* • We are comfortable that the overall % of A+A* will

vary from one subject to another because cohort profiles vary, but argue that the RELATIVE proportion of A* should be comparable esp amongst “facilitating subjects”

A* in A/L ML

22

• The highly technical 2014 Ofqual report showed that the “difficulty” of the exam was out of kilter with the ability profile of the cohort, as the latter had changed significantly in last 10 years becoming almost all very high prior attainment

• Because of the way A* was calculated, increasing the proportion of challenging questions to bring the two back into line would affect the proportion of A* to A + A* without affecting the A/B judgemental boundary – highly technical reasons

• Put into practice in Summer 2016 succcessfully• And will continue into reformed with “comparable”

A* in A/L ML

23

• “The decision stems from new research, published by the regulator today, which suggests that awarding should take into account the fact that native language speakers take these subjects. The adjustment to grade standards will be decided in early summer. “

• “If the ability of the cohorts is similar to previous years we would anticipate small increases in the proportion of students getting top grades in each subject this August.” Ofqual

A/L ML native speaker

https://www.gov.uk/government/news/setting-grade-standards-in-a-level-modern-foreign-languages

24

A/L native speaker - Fr

320 = A

200 = D

Note relatively high proportion of native to non-

native at lower grades

25

Outcome in Aug 17: Ofqual Key Points:• 3. In A level French, German and Spanish, outcomes at

grades A* and A are up following an agreement with exam boards to make an adjustment to take account of native speakers in these languages.

• So the percentage of candidates gaining grades A and A* has risen from 2016 to 2017 in all 3 of those languages: French 37.3% to 39.0%; German 39.6% to 41.4% and Spanish from 34.4% to 36.9%

• The proportion of A* grades relative to the number of A and A* grades in total is now around 26% meaning that ML candidates are being treated comparably to other "facilitating" arts subjects such as History.

A/L ML native speaker

https://www.gov.uk/government/news/guide-to-as-and-a-level-results-for-england-2017

26

• Sadly this has become almost “horse has bolted” as much of the damage has been done. In 2004 clearly, severe grading was a major issue especially at AS level. Look at the stark drop in candidates in the last 10 years at E grade; no. of E grades in 2015 was 29% of the number in 2002

Severe grading at

A/L

27

• Which has led to a major mismatch at A/L between the “difficulty” of the exam and the cohort profile – Ofqual report

• And since 2010 we have had the A* issue as a final nail in the coffin

• A&A* stable until 2012, when a drop of 1,000 candidates to 2015

•

A/L nos

28

GCSE MFL grading• Our outline principle is that typical students

studying a mainstream GCSE ML should have a reasonable expectation that they will get similar grades across EBacc subjects, without any systematic variation.

• So, looking at the many different ways of analysing cohort performance, ML should be brought broadly into line with the other EBacc subjects e.g. Maths

29

Issues for pupils• Perceive that they are performing worse

in languages in other subjects

• Reduces the take-up at KS4 and A-level

• In each case, pupils are doing relative comparisons on grades (note key issue at AS in French as compared with other subjects - at A2 Univs can make subject by subject adjustments)

30

Issues for staff & managers• Heads and Governors may think that

languages are “under-performing” so issues re Performance Management etc

• External people e.g. Ofsted may think same or that the school as a whole is “under-performing” against current performance measures

31

• It says: "The exam system is seen as one of the principal barriers to the successful development of language teaching.

• "The comparative difficulty of exams in languages in relation to other subjects, and widely reported harsh and inconsistent marking, are deeply de-motivating for both pupils and teachers."

• The report says the EBacc, where pupils have to study English, a language, maths, science and history or geography to GCSE, "appears to be having very little impact on the numbers of pupils taking languages post-16".

• Uptake after GCSE is found to be a particular concern, with some state schools suggesting the small numbers of students opting to take languages at A-level means the subject is becoming "financially unviable".

Language Trends survey - 2016

32

• Under half compared with 2004;

Impact on nos. at GCSE ML

Slight rise in 2013 after announcement

of EBacc in 2011

Drop in 2016 and further in 2017 as consequence of

Progress 8 EBacc3 definition (next slide)

33

Progress 8 EBacc3

bucket issue

Slight rise in 2013 after announcement

of EBacc in 2011

Drop in 2016 & more in 2017 as consequence of Progress 8 EBacc3

As part of Progress 8, best 3 scores in EBacc (Sciences, Computing, Hi/Geo, ML) count as EBacc3 bucket score.

So typically schools will use 2 Sciences + Hi or Geo rather than ML as pupils get higher scores on average

Nearly 500,000 sitting in 2002

34

• Qual. Reform and funding cuts hitting ML at A/L

• Hitting indept sector as well

Impact on nos. at A/L

http://educationdatalab.org.uk/2017/05/young-people-are-taking-fewer-a-

levels-as-qualification-reforms-kick-in-and-per-student-spending-falls/

35

• We believe it is unrealistic to move to comparable grading for all subjects

• Currently bands ML - His, Geo, Maths, Sci - English -Art, Drama, PE,

• We are proposing simply that ML moves into the 3rd band -tiny changes in grade boundaries involved, similar to those which have taken place - “one-off step change”

Way forward - 2008

data from CEM

SCORE 2008 paper

Slide from Ofqual Oct ‘08 ISC seminar

36

We believe it is unrealistic to move to comparable grading for allsubjects

Way forward -

2017

We are proposing simply that ML moves up to Hi / Ge (tiny changes in grade boundaries involved), similar to those which have taken place - “one-off step change”

data from Ofqual ISC paper 3

37

• ..

By KS2 prior

Grade C

Grade B

Grade A

Grade A*

half a grade difference

38• .

Original Ofqual graph in ISC Paper 3: 2010 - 13

Top row: grade AMiddle row: grade CBottom row: grade E

Sp, Gn, FrGeMa + SciEn

39

• Daily Mail Jan 2016 article about the Ofqual ISC papers "At GCSE the hardest subjects – modern foreign languages, Latin and statistics, are about half a grade harder than the average."

Media

http://www.dailymail.co.uk/news/article-3398347/Official-Exams-soft-subjects-really-easier-Board-admits-pupils-unfairly-marked-courses-maths-discuss-overhaul-grading-system.html

40

Importance of language

Terminology is important. Words like “hard”, “standards”, “demand” carry multiple connotations.

"An earlier version of this article used the expression 'marking down'. Ofqual would like us to clarify that the issue not about the quality of the marking but rather about how likely a student is to get a good grade depending on the exam subject. We have amended the article to avoid confusion.“

note at end of Daily Mail article on ISC (Jan ‘16)

thanks to Peter Tymms and Robert Coe for the phrase “severe grading” from Sept ‘04. The Times: “Student offers “should recognise easy A-levels”

41

• Need to avoid invalid comparisons / deductions re severe grading, numbers etc STEM v ML etcOur line on numbers has been: “Making changes here [grading] will not in itself solve the overall problem [pupil nos], but, to use a metaphor, will remove having a hand tied behind one's back.” ASCL / ISMLA / ALL to Dearing

• Similarly with pedagogy, time, etc etc• Need to look at each subject, Govt incentives, etc

Distractors

42

• We do not underestimate the sensitivity of this topic

• Through all of our involvement, have sought to make politically (and media) realistic proposals - e.g Lord Dearing in 2006 liking the idea of comparability with Maths:

“no-one will say that Maths is easy”• So he would have been pleased with

headline in Jan 2016 Daily Mail article on ISC !

Sensitivity

43

• June 17: ASCL adopt Position Statement “ASCL supports Ofqual in tackling severe grading in GCSE MFL so that students learning mainstream GCSE MFL should have a reasonable expectation that they will get similar grades across EBacc subjects, without any systematic variation”

Latest situation

https://www.ascl.org.uk/policy/position-statements/curriculum-and-assessment.html

44

• We cautiously welcome the action they [Ofqual] propose to take this year regarding grading at AL ML this year linked to the issue of native speakers, as we did to the action taken effective in June 2016 to address the anomalies in the proportion of A* grades in A/L ML

• We welcome the offer of further work on A level ML • We are deeply disappointed that Ofqual's

announcement about inter-subject comparability (ISC) does not address the issue of severe grading at GCSE.

Joint letter to Ofqual from ALL, ASLC, HMC, ISMLA

45

Joint letter -

1

46

Joint letter -

2

47

Joint letter -

3

48

• Representatives of the 4 organisations met with Ofqual in November to discuss the points raised, including outcome of ASCL hosted survey – see separate ppt

Next steps