The quality of GCSE exam papers in braille and modified large print

Author: Kate Flynn, Sue Keil and Rory CobbPublisher: RNIB; Year of Publication: 2016

1. Summary

1.1 IntroductionOver many years RNIB has received anecdotal information from qualified teachers of children and young people with vision impairment (QTVI) about problems with the provision of GCSE exam papers in accessible formats for blind and partially sighted candidates. To get a better understanding of the scale and nature of the problem, in 2015, RNIB with the support of Ofqual and the four Awarding Organisations (AOs) carried out research on the accessibility of GCSE exam papers in braille and modified large print (MLP). The project comprised two stages: a survey of 94 educators and a quality assessment of 101 GCSE papers provided by four AOs.

1.2 Key findings 87 per cent of the QTVIs, teachers and teaching assistants

surveyed had concerns about the provision of GCSE exam papers in accessible formats.

Concerns included the poor availability of practice papers, confusions in the ordering process, and the quality of diagrams, maps, and graphs in both braille and MLP.

75 per cent were aware of the ‘Best practice guidance for the modification and production of examination papers for candidates with a visual impairment’, nearly all of whom professed to follow it in the production of practice materials.

A quality assessment of 101 papers indicated an average, per braille paper, of 2.3 problems which were likely to have a high impact on the candidate. The average per MLP paper was 1.75

Some of the individual papers assessed had a far higher incidence of problems than the averages that we have reported here.

Among the braille papers, science was of particular concern, with a high number of problems relating to layout and design.

Among the MLP papers, mathematics was of particular concern, with layout and production problems predominating.

The quality of tactile and MLP diagrams, maps and graphs are a particular cause for concern, with a lack of consistency across the AOs in how they are presented.

2

1.2 ConclusionThis research set out to investigate whether anecdotal reports about provision of GCSE exam papers in accessible formats for blind and partially sighted candidates were isolated accounts or indicative of a more widespread problem. The findings from the two separate strands of the research provide evidence that these are not isolated incidents.

While this report highlights issues of concern about the provision of accessible papers to candidates with VI it is important to recognise that there is also much good practice which can be built upon.

The report concludes with recommendations for awarding organisations, modifiers, and educators, with a view to improving the current provision of accessible GCSE exam papers.

RNIB would like to thank Ofqual and the AOs for their support in carrying out this research and we look forward to working collaboratively to address the issues that it has identified.

3

2. IntroductionOver many years RNIB has received anecdotal information from qualified teachers of children and young people with vision impairment (QTVI) about problems with the provision of GCSE exam papers in accessible formats for blind and partially sighted candidates. Issues and concerns about exam papers are a regular topic on VI-Forum, which is an online forum for QTVIs.

In an effort to get a better understanding of the key issues, in July 2014 RNIB hosted a workshop of QTVIs and exams officers who have been involved in preparing blind and partially sighted candidates for examinations. Quality of exam papers was one of the topics discussed, and several delegates gave examples of problems they had experienced in braille and MLP papers. These included a braille paper in which the candidate was mistakenly required to produce a sketch, several examples of poor quality graphics, and part of a question missing from a paper.

As these were anecdotal accounts we had no way of knowing if they were isolated incidents or indicative of a more widespread problem. It was also unclear whether the concerns raised related mainly to problems in production, or problems in modification of papers, or both. We therefore believed that research was needed to get a better understanding of the scale and nature of the problem.

In 2015 Ofqual reported on issues in standard papers that might have a negative impact on GCSE and A level students. Their findings provide some context for this research. 1,302 exam papers and supporting materials were developed for GCSEs, and 1,672 were developed for AS and A Level. Ofqual reported that 10 errors occurred in a total of 2,974 papers. This figure included errors that would make a question impossible to answer, such as pages in the wrong order or smudged print, but also minor issues such as grammatical mistakes and typos that would not affect a student’s ability to answer the question (Ofqual, 2015). If candidates with a vision impairment are taking examinations on a level playing field with their sighted peers, we would expect the overall number of problems we recorded to be comparable to Ofqual’s findings for standard papers.

This report covers two stages of the research process. The first stage comprised a questionnaire survey of QTVIs, teaching

4

assistants and exam officers on their experiences relating to the provision of braille and MLP exam papers, and to what extent they take account of GCSE exam formats when modifying curriculum materials for their pupils. The second stage comprised a quality assessment of a sample of braille (including tactile diagrams) and MLP GCSE exam papers for the summer 2015 examinations, by QTVIs who are also subject specialists.

2.1 Aims and objectivesThe project aimed to: Ascertain the frequency of problems in a large sample of braille

and MLP GCSE Maths, Science and humanities exam papers. Identify the types of problem and whether they are more likely

to arise as part of the modification or production process. Identify the types of papers within the subjects sampled in

which the problems are most likely to occur (i.e. braille or MLP, particular subjects).

Identify any other issues of concern for teachers relating to the provision of GCSE papers for candidates with vision impairment.

Ascertain the extent to which QTVIs pay attention to the available exam formats when modifying curriculum materials for their pupils.

3. Methodology3.1 Questionnaire surveyWe designed a short written questionnaire which focused on the steps respondents take to prepare blind and partially sighted learners for GCSE examinations, and their views on the braille and MLP papers provided by awarding organisations (AOs). The survey was particularly focused on the practices and experiences of classroom practitioners, such as QTVIs, mainstream teachers and teaching assistants, although other parties involved in the exam process such as exam officers could answer the questions applicable to their role.

Respondents were able to complete the questionnaire either as a word document, which they could email directly to us; or they could provide their responses via SurveyMonkey, which is an online platform for questionnaires.

5

The questionnaire was live throughout June and July 2015. It was sent to local authority education VI services, and special schools and colleges for learners with VI, and was also available to download from the RNIB website. The survey was publicised via various networks of QTVIs such as VI-Forum, HOSS forum, VIEW, NatSIP, the RNIB education e-newsletter (which has a circulation of 1500 teachers and teaching assistants), and the Exam Officers’ Association.

3.2 Quality assessmentThe second part of the project was endorsed by Ofqual, and carried out in collaboration with four AOs. The support of the AOs was essential to reviewing the current system for the provision of accessible exam papers.

RNIB compiled a request list of braille and MLP papers before the examinations took place. Papers were randomly selected for inclusion. The sample size was determined using Acceptance Sampling tables (Ryan, 2011), which are a tool for checking quality in a given lot.

Biology, Chemistry, Physics, Combined Science, English Literature, English Language, Mathematics, History and Geography were all included. The number of papers we requested was proportionate to the total number of papers offered for a particular subject. This means that the samples differed in size from subject to subject; for instance more science papers than English papers were requested.

In practice not all the requested papers were available, as not all papers are routinely produced in braille and MLP. Accordingly we accepted substitutes when these were offered. Time constraints also placed a limit on the number of papers that could be reviewed. A total of 101 papers were assessed. (See Appendix for a list of the papers that were requested, and assessed).

The available papers were reviewed by three exam paper assessors, all of whom were qualified teachers of children and young people with vision impairment (QTVI) and subject specialists in science, mathematics and/or humanities. Their aim was to review the papers against the “Best practice guidance for the modification and production of examination papers for candidates with a visual impairment” (JCQ, 2012). Departures from

6

the guidance would be considered a potential problem. Prior to receiving the papers the assessors and researchers grouped potential problems into the following categories:

Proofing problems e.g. misspellings, missing braille cells/content/pages, incorrect mathematical coding, pages in the wrong order.

Layout problems. Presentation that could confuse or disadvantage the candidate e.g. starting a question at the bottom of a page.

Design problems. Graphics that don’t convey the intended information, or that can’t be used as intended e.g. angles that are obscured beneath a tactile protractor.

Production problems e.g. poor print contrast, grid lines that aren’t tactually discernible, poorly discriminated tactile “shading.”

Modification problems. Questions that are inappropriate for a candidate with vision impairment and irrelevant to the assessment objectives e.g. asking the candidate to draw sketches.

Other problems. Any other causes for concern e.g. style differences in approaches to modification.

The assessors also rated each problem according to their perception of likely impact on the candidate—either low, medium or high impact. It is important to note that these ratings are not directly interchangeable with the categories Ofqual uses for errors in standard papers. For Ofqual, the highest level of error, “Category 1”, makes a question impossible to answer; while the next highest, “Category 2”, may cause unintentional difficulties for the candidate when answering the question (Ofqual 2015). In this project, some problems which might be classified as “Category 2” errors under the Ofqual rating were, using the RNIB definition, deemed likely to have a high impact on blind or partially sighted candidates. For instance candidates may have been able to attempt a response but not achieve full marks; or they may have been able to answer the question by giving it disproportionate time, at the expense of the rest of the paper.

7

Where the same problem recurred throughout a paper—for instance, if paragraph indentations were incorrectly formatted in several questions—it was counted as only one problem in that paper’s score. The cumulative impact of problems on the candidate is therefore likely to be greater than the scores indicate. The individual reports for AOs will indicate which problems recurred in which papers.

During the review the assessors and researchers remained in contact to aid consistency. To provide an additional layer of scrutiny, an adult with a science degree who is an expert braille user offered a user’s perspective on five of the braille papers.

4. Results4.1 Questionnaire SurveyQualified teachers of children and young people with vision impairment comprised just over a third of respondents, and Exam Officers/Managers comprised just under a third. Respondents in the “other” category included head teachers, invigilators, and librarians amongst others.

Respondent’s role Percentage of respondentsn=94

Qualified teacher of children and young people with vision impairment (QTVI)

34

Teaching assistant (TA) 15Mainstream (non VI specialist) class teacher

2

Exams Officer/Exams Manager 30Other 17

We asked respondents what type of setting they worked in. Nearly half worked in a mainstream school. (However, if we exclude exam officers, the percentage of QTVIs, teaching assistants and teachers working in a mainstream school is lower at 35 per cent, while 54 per cent work within a local authority VI/Sensory Advisory Service).

Setting/context of respondent’s work

Percentage of respondentsn=94

8

Mainstream school 48Mainstream FE College 6Special school for pupils with VI 2Special college for pupils with VI 4Local authority VI/Sensory Advisory Service

29

Other type of special school, college or setting

11

We asked respondents whether they had any concerns about the provision of exam papers in accessible formats. The majority said yes. QTVIs, teachers and teaching assistants were more likely to say yes (87 per cent).

Concerns about the provision of exam papers in accessible formats

Percentage of all respondents n=92

Percentage of respondents (excluding exam officers) n=46

Yes 68 87No 27 13Don’t know 4 0

In an open ended question, we asked those respondents with concerns to provide the three main ones. 56 people replied, 36 of whom were practitioners with direct classroom experience such as a QTVI, mainstream teacher or teaching assistant. Comments broadly focused on the preparation of learners, shortcomings in the process of applying for braille and MLP papers, and the quality of the papers provided.

Many commenters expressed that they wanted to prepare learners appropriately and did not have the materials to do so. A recurrent concern was the difficulty of accessing past papers with which learners could prepare for their exams:

“My biggest issue about the provision of exam papers in accessible formats including braille and modified large print is the lack of papers that are available for schools for VI pupils to use to practise with. This does not allow the VI young people that we support to have equal access to resources that their sighted friends

9

have and suggests that exam boards do not have high expectations for these pupils.” Respondent 18 (Teaching Assistant)

“MLP and Braille - availability of past papers and advice re the cost of them. Relevant particularly to GCSE science, maths and geography, where the students NEED experience for the style of diagramused. After all, research has shown that familiarity with the format is a contributing factor to exam success.” Respondent 13 (QTVI)

“There were no examples of what a papermight look like before my student sat his first exam, causing him stress and anxiety. I could not therefore give him practice papers in the correct format.” Respondent 93 (Teaching Assistant)

Some respondents expressed concerns about the application process for braille and MLP papers:

“We tried to arrange for him to have access to a paper copy of the exam with an appropriate font size and an electronic version which he could load on his laptop. We found it confusing trying to order these formats. Ticking the box for an electronic copy generated a different response from each board, which we did not expect.” Respondent 93 (Teaching Assistant)

“The deadline for ordering the papers is too early - it is very difficult to determine so far ahead of the examination which tier a student will need to sit, for example. This applies to all formats.” Respondent 59 (Exams Officer)

Alarmingly, some papers never arrived or were incomplete:

“MLP papers being ordered but then not arriving.” Respondent 15 (QTVI)

“MLP: last minute dispatch of exam papers or, in some cases, NO dispatch at all.” Respondent 19 (Exams Officer)

10

“Not enough braille papers - 6 ordered, 3 turn up.” Respondent 43 (QTVI in training)

“Two came with things missing but we didn't know because we can't open the papers in advance to check them, so we had to apply for spec. con.” Respondent 22 (Exams Officer)

However, the most common issue raised by respondents related to the quality of diagrams, maps, and graphs, in both braille and MLP papers:

“The lines used on graphs tend to be very narrow and often grey. The result of this is that the student I work with struggles to see graphs in Science and Mathematics papers, even when using a CCTV to further magnify it. Discussion with (the board we use for those subjects) resulted in logic on the lines of ‘we have modified it, we aren’t going to modify it further and you cannot modify a modified paper’. A comment which seemed a little unhelpful.” Respondent 93 (Teaching Assistant)

“Production of graphs in tactile diagrams in Braille - The labels on the side axis are sideways meaning that in order to read them you have to rotate the graph which proves difficult when trying to read information off of the graph. Tactile diagrams Science - Some of the diagrams have been difficult to understand especially the size of the periodic table.” Respondent 79 (Teaching Assistant)

“Braille/Tactile diagrams - these are not consistent - some are extremely good, but others have clearly not followed basic guidelines to make them accessible to a blind person (example this year have been cluttered diagrams, diagrams and tables that extend over more than one sheet of paper and are difficult to track, angles to measure with lines that are shorter than the radius of the RNIB standard tactile protractor).” Respondent 71 (QTVI)

11

Less commonly mentioned accessibility issues included poor contrast and colour of papers, mistakes in transcription including question numbering, the difficulty of handling A3 papers, and minimal or inadequate modification. These issues will be picked up later in the report when we consider the findings of the quality assessment.

Reassuringly, only three respondents mentioned schools not taking the available formats into account when they prepared learners for exams (one QTVI, and two Teaching Assistants).

Only two respondents—both QTVIs--expressed dissatisfaction that learners’ preferred format wasn’t available at GCSE (but note two more respondents, an Exams Officer and a Teaching Assistant, raised similar concerns in reply to the survey’s final question).

As the following questions relate to classroom practice, the findings exclude exam officers and apply only to QTVIs, teachers, and teaching assistants, i.e. up to 48 of the respondents.

We asked if they were aware of the “Best Practice Guidance for the Modification and Production of Examination Papers for Candidates with a Visual Impairment.” The majority were.

Aware of ‘Best practice guidance for the modification and production of examination papers for candidates with a visual impairment’

Percentage of respondentsn=40

Yes 75No 25

Among the respondents who were aware of the guidance, all but one said they followed it when producing, or advising others on producing, braille, large print, modified large print and/or tactile diagrams for pupils with VI.

The majority of respondents also said they had the formats of GCSE exam papers in mind “a lot” when they produce or

12

advise on producing curriculum materials. There was less difference than we might expect between respondents who were aware and unaware of the Best Practice Guidance. This may be due to small sample sizes and “ceiling effects” (that is, the question’s highest available answer or “ceiling” doesn’t allow us to discriminate between the respondents). It may also be that respondents’ perceptions of “a lot” differ.

The following tables show to what extent respondents had GCSE formats in mind for modified large print, braille and tactile diagrams. The sub-samples are small, so we’ve provided the number—rather than percentage—of respondents who gave a particular answer.

People who were aware of the Best Practice Guidance were more likely to report taking GCSE formats into account “a lot” when producing or advising on curriculum materials in modified large print.

Similarly, people who were aware of the Best Practice Guidance were more likely to report taking GCSE formats into account for producing braille curriculum materials.

13

Whether has formats of GCSE papers in mind when producing/advising on curriculum materials (Modified large print)

Number of respondents (all)(n=36)

Number of respondents (Aware of Best Practice Guidance)(n=27)

Number of respondents (Unaware of Best Practice Guidance)(n=9)

A lot 28 22 6A little 4 1 3Not at all 2 2 0Not applicable 2 2 0

Whether has formats of GCSE papers in mind when producing/advising on curriculum materials (Tactile graphics/diagrams)

Number of respondents (All)

Number of respondents (Aware of Best Practice Guidance)

Number of respondents (Unaware of Best Practice Guidance)

  (n=34 ) (n=26) (n=8 )A lot 21 18 3A little 1 0 1Not at all 2 1 1Not applicable 10 7 3

Finally, people who were aware of the Best Practice Guidance were also more likely to take GCSE formats into account for producing tactile graphics/diagrams in curriculum materials.

Whether has formats of GCSE papers in mind when producing/advising on curriculum materials (braille)

Number of respondents (all)(n=36)

Number of respondents (Aware of Best Practice Guidance)(n=27)

Number of respondents (Unaware of Best Practice Guidance)(n=9)

A lot 19 15 4A little 4 3 1Not at all 3 2 1Not applicable 10 7 3

We asked respondents about the types of past papers they used for exam preparation. Three quarters of respondents prepared learners with past papers produced by the exam boards, and about the same proportion did so with papers produced locally:

Practice exams with past papers produced by the exam boards

Percentage of respondentsn=38

Yes 79No 21

14

Practice exams with past papers produced locally (on a bespoke basis) in pupil’s preferred format

Percentage of respondentsn=38

Yes 76No 24

As can be seen from the tables, quite a few respondents did both, which is not surprising given that so many had experienced difficulties in obtaining copies of past papers.

The majority of respondents who prepared their students for exams began doing so before Year 10:

School year that preparations usually begin

Percentage of respondentsn=34

Year 7 32Year 8 9Year 9 29Year 10 26Year 11 0Not applicable 3

Over three quarters of respondents had experienced difficulties in obtaining past exam papers:

Difficulties in obtaining past papers in accessible formats for practice purposes in the past two years

Percentage of respondentsn=38

Yes 76No 16Not applicable 8

We asked respondents what type of difficulties they had experienced. Low availability of papers from AOs, and the cost associated with braille copies, were the main issues:

“When we contact exam boards we are told that the Braille exam papers are a 'one off' and they cannot give us a copy. Purchasing exam papers in braille from other areas, rather than exam boards (which say

15

they cannot provide the past exam paper) proves expensive for schools depending on how many papers are required per subject.” Respondent 84 (QTVI)

“If no VI student had taken that paper/ subject, there were no modified papers available, so I had to modify my own using previous years as a guideline.” Respondent 70 (Teaching Assistant)

“It took several months to be e-mailed an example of a large print modified History paper.” Respondent 73 (Teaching Assistant)

Finally we asked all 94 respondents whether they had any other comments they would like to make. 29 replied. Some took the opportunity to re-emphasise a concern raised earlier in the questionnaire, such as GCSE papers never arriving, or being inappropriately modified. Others chose to highlight the circumstances of learners who are not well served by the current options:

“As the student I work with has a degenerative condition, it is likely that 36pt will not be sufficient for him in the future. As this is the current largest font used in MLP papers, I wonder what will happen to him next. If we cannot modify a modified paper and 36pt is too small what happens?” Respondent 93 (Teaching Assistant)

“[The current range of formats] is applicable to the current cohort, however, we will have a student starting in Year 7 for whom modified papers will not be enough as they will require size 48 font.” Respondent 28 (Exams Officer)

Both these points have implications for respondents’ own professional practice; and demonstrate the inherent difficulties of creating a system that matches the needs of individual candidates, given the range of the population. Furthermore, the comment on the availability of 36pt papers indicates there may be some confusion about what options are offered as standard (MLP is

16

available in 18pt and 24pt only) as opposed to being available only on special request.

4.2 Quality Assessment101 papers were checked by the assessors – 20 science papers, 32 mathematics papers, 19 English Literature and English Language papers, 17 History papers and 13 Geography papers.

As some variation between the assessors could be caused by differences in interpretation, this section foregrounds the highest impact problems. Our aim is to minimise the focus on more marginal, subjective issues and discuss the issues that assessors would agree are of greatest concern.

Each of the participating AOs has received an individual report with problem counts for their particular papers. Although there was some variation between the mean high impact problems for each AO, please note that the difference was not statistically significant. This means that the discussion and recommendations in this report are relevant to all four AOs.

Across the entire sample of 101 papers, the mean number of high impact problems per paper was 2, the median was 0, the mode was 0, and the range was 12.

The tables below show the number of high impact problems by format, subject, and type of problem.

17

Science (including Biology, Chemistry, Physics, foundation

and higher tiers) Mathematics

English Language and English Literature History Geography All papers

Number of papers checked 8 12 9 8 4 41Proofing problems 3 3 0 0 2 8Layout problems 23 3 0 0 0 26Design problems 14 3 0 0 0 17Production problems 3 10 0 0 2 15Modification problems 5 8 1 1 8 23Other problems 5 0 0 0 0 5Total number of high impact problems 53 27 1 1 12 94Mean number of high impact problems per paper 6.6 2.3 0.1 0.1 3 2.3

Table 1: High impact problems in braille papers by subject and type of problem

18

Table 2: High impact problems in MLP papers by subject and type of problemScience (including

Biology, Chemistry, Physics, foundation

and higher tiers) Mathematics

English Language and English Literature History Geography All papers

Number of papers checked 12 20 10 9 9 60Proofing problems 1 6 0 0 0 7Layout problems 7 14 0 0 0 21Design problems 13 4 0 0 1 18Production problems 0 35 2 0 0 37Modification problems 1 8 1 1 11 22Other problems 0 0 0 0 0 0Total number of high impact problems 22 67 3 1 12 105Mean number of high impact problems per paper 1.8 3.35 0.3 0.1 1.3 1.75

19

4.3 Braille user’s commentsThe expert braille user read five of the sampled exam papers and broadly agreed with the assessors’ identification of high impact problems in those cases.

From a user’s perspective, he encountered a range of issues which could confuse or hinder candidates. For instance, he noted that in some cases print had been inappropriately or over-literally transcribed:

“‘Do all your rough work on your answer sheets. Cross through any work you do not want to be marked.’ This would amuse candidate – not very helpful. Would have expected modifier to have removed this.”

Note this problem could be attributed to producers, rather than modifiers, as modifiers are instructed not to make recommendations for cover sheets.

In some papers the braille user observed a failure to use standard braille practice for paragraph indentations, equation indentations and bullet lists, which sometimes made it harder to understand questions – for example:

“Throughout the paper the paragraphs within questions are not shown as braille paras. The first line should be indicated by indenting by 2 braille cells (standard braille practice) but all is left justified.”

Occasionally the braille was too small, and in some cases contained inaccuracies:

“Should say ‘respiration’ but says’ respirsion’”

“Sulphur is spelt with an ‘f’ in part a, then ‘ph’ in part b and again, ‘f’ in part b.”

20

The expert braille user commented particularly on the poor readability of a periodic table (note a similar comment was made by one of the survey respondents):

“The periodic table is very confusing – unless the candidate has seen something similar before. Asterisk against element 57 on the table – on the key page it is written the wrong way round.”

Elsewhere the readability of the braille was affected by production problems, such as where the braille ran into the hole punched for binding. Some papers were poorly bound and others comprised excess amounts of paper, making them difficult to handle. The expert braille user remarked on one diagram booklet:

“Problem is there is so much stuff – desk management would be an issue! Do some of these diagrams have to be so big?”

In combination, these issues could undermine a candidate’s confidence. The expert braille user commented that “these problems would increase anxiety” if he were a candidate.

5. DiscussionBoth stages of this research project indicate that GCSE candidates with vision impairment face avoidable barriers to exam success. These barriers do not apply to their sighted peers. One of the biggest concerns for the survey respondents was the lack of practice papers available in braille and MLP. While some past papers can be obtained from AOs, this fact is not well publicised. In addition, the fact that past braille papers are not free is a disincentive for schools to obtain them. In contrast, copies of past papers in standard print are widely available and at no cost indicating a lack of parity for learners with vision impairment. Respondents’ frustration at the lack of practice papers suggests that the inequality begins well in advance of the exam itself. Given the importance of practice effects to success, poor availability of

21

accessible papers places VI candidates at a substantial disadvantage compared to someone who is not disabled under the terms of the Equality Act.

The ordering process also contributes to disadvantaging some VI candidates. This is because the requirement to meet an early deadline for ordering papers may mean that tiers for VI candidates have to be decided earlier than for sighted candidates.

VI candidates may also be placed under additional stress if the process for ordering the actual exam papers is too confusing to deliver the formats requested on their behalf; or if the administrative burden on AOs means that accessible papers don’t arrive in time. The differences between AOs in the process for ordering electronic versions of papers, which was noted by one respondent, illustrates the complexity of the process.

An aim of this research was to explore whether quality concerns about accessible GCSE papers reflected problems in provision, or an expectation on the part of educators for a more bespoke service. It is notable that very few respondents requested a broader range of formats than those currently provided as standard. Nevertheless, from a small number of comments there appears to be some confusion over what actually constitutes standard provision. RNIB is aware that the AOs differ in the range of standard MLP formats that they provide; however, this information is not readily available to exam centres.

A number of survey respondents commented negatively on the quality of some exam papers, with poor quality of diagrams, maps and graphs in both braille and MLP being the most common issue raised. The scrutiny of sampled papers suggests that there are indeed quality issues in the provision of both braille and MLP papers, particularly—as reported by survey respondents—in the design, layout and production of diagrams and graphs. While the English and History papers contained few high impact problems, the sample of Geography, Mathematics, and especially Science papers had worryingly large problem counts. The feedback provided by the expert braillist, giving a user perspective, supports the findings of the assessors and the concerns raised by survey

22

respondents. In general, it would appear that there is insufficient regard given by AOs to the Best Practice Guidance for the Modification and Production of Exam Papers for Visually Impaired Candidates. At the same time, the Best Practice Guidance needs to be regularly reviewed to make sure that it is as comprehensive and up to date as possible.

The number of problems in the sampled papers is particularly striking because it greatly exceeds the number of problems Ofqual reported in standard papers. While Ofqual’s categorisation of errors differs slightly from the approach we adopted for this project, that fact alone cannot account for so large a difference. While only 10 errors in total were reported in 2,974 standard GCSE and AS/A-level papers (Ofqual, 2015), our assessors found 199 high impact problems in 101 accessible GCSE papers. This indicates that candidates with a vision impairment face barriers that sighted candidates do not.

One outcome of the project is the provision, to AOs, of a detailed breakdown of the problems identified in their papers. This will allow them to take appropriate actions for improvement in a way that acknowledges the differing pressures on them. However, we would like to emphasise that the difference between their mean high impact problem scores was not statistically significant. The issues raised above apply to all the participating AOs.

6. Conclusion This research set out to investigate whether anecdotal reports about provision of GCSE exam papers in accessible formats for blind and partially sighted candidates were isolated accounts or indicative of a more widespread problem. The findings from the two separate strands of the research provide evidence that these are not isolated incidents. Concerns raised in the survey of teachers and TAs about the quality of braille and MLP exam papers and modified and tactile diagrams were supported by the separate scrutiny of papers undertaken by the specialist assessors and the expert braille user. Problems were found in the modification, layout and production of papers, indicating problems at all stages of the process.

23

These and other issues raised in this research, such as difficulties in obtaining past papers in braille and MLP indicate that there is a real lack of parity for blind and partially sighted exam candidates’, placing them at a substantial disadvantage in comparison with their fully sighted peers.

While this report highlights issues of concern about the provision of accessible papers to candidates with VI it is important to recognise that there is also much good practice which can be built upon. While problems were found in papers produced by all four AOs, we also identified strengths. For example, the quality of braille papers from one AO was generally high but was let down by poor proof reading.

7. RecommendationsRNIB would like to thank Ofqual and the AOs for their support in carrying out this research and we look forward to working collaboratively with them to consider and act upon the following recommendations.

1. The Best Practice Guidance should be reviewed and updated to ensure that it remains fit for purpose.

2. A proper system for recruiting and training modifiers should be established. Funding will be needed to ensure that this system is sustainable.

3. Producers should be required to demonstrate competence in producing MLP and braille papers in line with the Best Practice Guidance.

4. Producers should have effective systems in place for proof reading braille and MLP papers.

5. An independent quality assurance system should be established to check a sample of modified papers from all AOs on an annual basis.

6. Options should be explored to achieve greater consistency in the way that graphics (and tactile diagrams in particular) are presented in modified papers. A good starting point for this would be the single production of standard resources (e.g. the periodic table) which could be shared by all AOs.

24

7. Copies of past papers that have been modified should be made more readily available to schools and colleges. This should include past braille papers available as electronic downloads.

8. QTVIs and TAs working with children and young people with vision impairment should receive more training in access arrangements to ensure they prepare their students appropriately.

9. Specific training should be provided within the VI sector for teachers and TAs in how to design and produce tactile diagrams.

10. There should be a review of the range of accessible formats which are available as standard and how best to communicate information about these to exam centres and QTVIs.

25

8. AppendixRequested papersAQAScience A Route 1 4405: BL1FP, CH1HP and PH1FSScience A Route 2: SCA2HPScience B (Science in Context) 4502: SCB3FPEnglish Language 4704: ENG1FEnglish Literature 9717: 97151H and 97154FMathematics 4360: 43601F and 43603FMathematics (Linear) B 4365: 43652FMethods in Mathematics (Linked Pair) 9365: 93652FApplications of Mathematics (Linked Pair) 9370: 93702FGeography A 9032: 90301FGeography B 9037: 90351HHistory A 9142: 91401AHistory B 9147: 91451

OCRScience A (J241): A161/1, A171/1 and A181/1Science B (J261): B711/1English (J350) and English Language (J355): A680/1English Literature (J360): A662/2Mathematics A (J562): A501/1 and A503/1Mathematics B (J567): J567/3Geography A (J382): A732/1, 2Geography B (J385): B563/1, 2History A (Schools History Project) (J415): A955History B (Modern World) (J418): A021 – A022

WJECScience A: Chemistry 1 Foundation 4462/01Science A: Biology 1 Higher 4461/02Science A: Physics 1 Higher 4463/02Science B: Unit 2 Foundation 4782/01English/English Language Unit 1 Reading Foundation: 4171/01English/English Language Unit 2 Writing Higher: 4172/02English Literature Unit 2b Foundation: 4202/03Mathematics (Unitised) Unit 1 Foundation: 4351/01Mathematics (Unitised) Unit 2 Higher: 4352/02

26

Mathematics (Linear) Paper 1 Foundation: 4370/03Geography (Spec A) Unit 1 Foundation Tier: 4231/01Geography (Spec B) Unit 2 Higher Tier: 4242/03-04History Unit 1: 4271/01-06History Unit 3: 4373/01-04, 4383/01-02

Pearson EdexcelScience: 5BI1F, 5CH2F, 5PH3F, 5CH1H and 5PH2HEnglish: 5ETIF, 5ET2H and 5EN2FMathematics: IMA0 1F, IMA0 SH and 5STIFGeography: 5GA1F, 5GB2H, GGA3HHistory: 5HA01, 5HB01, 5HA02, 5HB02, 5HA03

Papers checked by assessorsAQAScience A/Biology Unit Biology B1 Foundation Tier BL1FPScience A/Chemistry Unit Chemistry C1 HigherTier CH1HPScience A2 Unit6 Higher Tier SCA2HP with Chemistry Data Sheet & Physics Equations SheetMathematics 43603FMathematics 43601HMathematics 43601FMathematics 4365/1HHistory SHP Medicine 91401History B Modern World 91451English Literature Poetry 97152HEnglish Literature 97151HEnglish Language ENG1FGeography A Unit 1 90301F

OCR21st C Science Chemistry A/Science A A171/01 Modules C1 C2 C321st c Science Biology A/Science A Modules B1 B2 B3 Foundation Tier A161/01Mathematics J567/01Mathematics B (Higher) J567/02Mathematics B (Higher) J567/03English/English Language Information and Ideas A680/01English Literature Mod Drama

27

History A955/22History B Mod World Bsh Society A021/01Geography B563/01Geography A732/01/02WJECScience A/Biology 1 Higher tier 4461/02Science A/Chemistry 1 Foundation 4462/01Science A/Physics 1 Higher Science B Unit 2: Science and Life in the Modern World Foundation Tier 4782/01Mathematics 18pt 4352/02Mathematics 18pt 4730/03English Literature 4202/03English Language 4171/01History 4271/02Geography A F Unit 1Geography H 4242/04-A

Pearson EdexcelBiology/Science Unit B1: Influences on Life Foundation Tier 5BI1F/01Chemistry/Science Unit C1: Chemistry in Our World Higher Tier 5CH1H/01Physics/Additional Science Unit P2: Physics for your future Higher Tier 5PH2H/01Statistics (Foundation) 5ST1F/01Mathematics 1MA0/1FMathematics 1MA0/1HEnglish Literature Understanding Poetry 43110English Language F 44694 5EN2F/01English Literature Understanding Prose 43107AHistory Cold War 44749History Germany 2A 44750History SHP Germany 44772History War and transformation 1903-28 44753

ReferencesJCQ (2012) General and vocational examinations best practice guidance for the modification and production of examination papers for candidates with a visual impairment. Standing

28

Committee for Examination Candidates with Special Requirements for Joint Council for Qualifications (JCA)

Ofqual (2015). Summer report 2015. Office of Qualifications and Examinations Regulation. https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/489745/Summer_2015_Report_final_23_Dec.pdf

Ryan, T. P. (2011). Statistical Methods for Quality Improvement. Hoboken, New Jersey: Wiley.

29