The National Student Survey (NSS)

Penny Jones, Strategic Planning Office

Tracy Goslar and Miles Willey , Academic Standards & Partnership

Wednesday 16 March 2011 (Moulsecoomb)Friday 18 March 2011 (Eastbourne)

• NSS began in 2005 and is a key element in the Quality Assurance Framework

• The QAF, which also includes institutional audit, is the means by which HEFCE meets its legal obligations for ensuring the quality of publicly funded higher education.

• NSS was and remains a key element in the commitment to provide public information about quality and standards of publicly funded higher education (www.unistats.com)

The National Student Survey (NSS)

• NSS is becoming increasingly integrated into the national quality assurance system, as evidenced by plans to further develop and enhance the survey

• Student feedback from the NSS is considered alongside other student evaluation as a central aspect of the University’s annual Academic Health process

• Nationally, the student experience is viewed as a central pillar of quality assurance systems

Integration of the NSS into quality assurance systems

• 22 questions • 5 point scale – Definitely agree to definitely disagree • 6 main sections, Questions 1-21

1. The teaching on my course

2. Assessment and feedback

3. Academic support

4. Organisation and Management

5. Learning resources

6. Personal Development • Question 22 – overall satisfaction • Additional questions • Comments

•

The survey

• The survey is anonymous

• To receive published results for the NSS, the University must receive a response rate of at least 50%

• Results at course level are available for internal use only where 10 or more respondents complete the survey

• Responses below the threshold of 10 are published at the JACS subject level on the Unistats website

Publication of response rates

• Samples not representative

• Comparisons or ranking of whole institutions in league tables

institutions fall within narrow range covered by sampling error

Subject mix in different institutions

• Students expectations different

• Opinion not truth

• Easy to manipulate

• UNISTATS

Why we don’t like the NSS - survey criticisms

• Free market research! Important tool to improve student experience and highlight broad areas where there are issues, rather than ranking tool

• Use free text comments to contextualise figures

• Work in conjunction with SU and course reps to check what is real

• Use alongside internal evaluation - modules etc.

• Benchmarking against sector

• Issues discussed in report to HEFCE ‘Enhancing and Developing the NSS’ http://www.hefce.ac.uk/pubs/rdreports/2010/rd12_10/

Why we do like the NSS...

• Staffcentral link: http://staffcentral.brighton.ac.uk/nss

• NSS website: www.thestudentsurvey.com

• General enquiries: Tracy Goslar (Academic Standards & Partnership) – t.goslar@brighton.ac.uk

• Analysis of response rates and results: Penny Jones (Strategic Planning Office) – ext 2052

• Marketing materials: Susan Lightfoot (Marketing & Communications) – ext 2221

Useful contacts

• Response Rates 2011

• Trends and demographics 2006 to 2010

• Further Analysis

Numbers!

2010 First time UoB response rate higher

than sector!

Response Rates

50%

52%

54%

56%58%

60%

62%

64%

66%

68%

2006 2007 2008 2009 2010

University of Brighton % Response Rate Versus the Sector 2006 to 2010

UoB %

Sector %

50%

55%

60%

65%

70%

0

500

1000

1500

2000

2500

3000

2006 2007 2008 2009 2010

University of Brighton % Response Rate and Actual Number of Respondents

UoB No

UoB %

74% more respondents than 2006

Response Rates 2011 – versus previous years

Course Name: Course Code: School Delivering:

H0051 Brighton (University of) H0051 Brighton (University of) H0051 Brighton (University of)

Sample Eligible

2011

3907

Sample Eligible

2010

4022

Sample Eligible

2009

424845.2

65.6

57.1

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

1M 1F 2W 3M 3F 4W 5M 5F 6W 7M 7F 8W 9M 9F 10W 11M 11F 12W 13M 13F 14W 15M 15F 16W 17M 17F 18W

% R

espo

nse

Rate

Week Number & Day

2011

2010

2009

Response Rates 2011 – versus the sector

45.2

45.3

0

5

10

15

20

25

30

35

40

45

50

1M 1W 1F 2M 2W 2F 3M 3W 3F 4M 4W 4F 5M 5W 5F 6M 6W 6F 7M 7W 7F 8M 8W 8F 9M 9W 9F

10M

10W 10

F

11M

11W 11

F

12M

12W 12

F

13M

13W 13

F

14M

14W 14

F

15M

15W 15

F

16M

16W 16

F

17M

17W 17

F

18M

18W 18

F

% R

espo

nse

Rate

Week & Day

University of Brighton National Student Survey % Response Rate Versus the Sector

UoB inc NHS 2011

Sector inc NHS 2011

• Trends

Mode Gender Domicile Age Ethnicity Level Disability NHS

• Look at the optional set of questions

Presentation of Trends

National Student Survey Findings and Trends 2006 to 2009 (HEFCE 2010/18)

Mode – Full time and Part Time

2006 2007 2008 2009 2010

Brighton Full Time 78 77 80 79 78

Brighton Part Time 80 83 88 89 86

Sector Full Time 80 81 82 81 81

Sector Part Time 86 86 85 84 87

70

72

74

76

78

80

82

84

86

88

90

% Agree Overall Satisfaction in Full time and Part time Students at UoB and in the Sector

Brighton Full Time

Brighton Part Time

Sector Full Time

Sector Part Time

‘...the profile of satisfaction scores for those on part-time courses was significantly different to that of those studying on full-time courses. The question categories with the largest differences were Assessment and feedback, where part-time students were more satisfied, and Learning resources and Personal development, where part-time students were less satisfied.’ (HEFCE 2010)

Gender

2006 2007 2008 2009 2010

Brighton Female 78 76 80 81 78

Brighton Male 77 79 81 79 80

Sector Female 81 82 82 82 82

Sector Male 80 81 82 81 82

75

76

77

78

79

80

81

82

83

% Agree Overall Satisfaction, by Year, by Gender, at the University of Brighton and in the Sector

Brighton Female

Brighton Male

Sector Female

Sector Male

‘While Overall satisfaction of female and male students was similar to the global score significant differences were observed in other categories, most notably for Academic support and Learning resources.’ (HEFCE 2010)

Domicile

0

10

20

30

40

50

60

70

80

90

100

2006 2007 2008 2009 2010

% Agree Overall Satisfaction by Domicile at University of Brighton

EU (excl UK)

Other

UK

0

10

20

30

40

50

60

70

80

90

100

2006 2007 2008 2009 2010

% Agree Overall Satisfaction by Domicile in the Sector

EU (excl UK)

Other

UK

‘Students from the EU and international students were significantly less satisfied with Teaching and learning than students from the UK. For Assessment and feedback, students from the EU were significantly less satisfied and international students were significantly more satisfied than the global score.’(HEFCE 2010)

Age

70

72

74

76

78

80

82

84

2006 2007 2008 2009 2010

% Agree Overall Satisfaction by Age, at Brighton and in the Sector.

UoB Mature

UoB Young

Sector Mature

Sector Young

‘...students in the age groups 21-24 and over 25 were significantly less satisfied than students aged under 21, in the questions on Academic support, Organisation and management, Learning resources and Overall satisfaction. However, students aged over 25 were significantly more satisfied than the global score for the category Assessment and feedback.’ (HEFCE 2010)

Ethnicity

50

55

60

65

70

75

80

85

90

2006 2007 2008 2009 2010

% Agree Overall Satisfaction, by Year, and Ethnicity, at University of Brighton.

Asian

Black

Not Known

Other

White

50

55

60

65

70

75

80

85

90

2006 2007 2008 2009 2010

% Agree Overall Satisfaction, by Year, and Ethnicity, in the Sector.

Asian

Black

Not Known

Other

White

University of BrightonEthnicity 2006 2007 2008 2009 2010Asian 99 104 53 106 122Black 55 59 24 78 94Not Known 87 107 124 217 247Other 55 65 49 95 127White 1194 1418 1033 1792 2009

‘Satisfaction profiles varied significantly depending on students’ ethnic background. However, the only question category where all the differences in satisfaction score were significant was Teaching and learning; where students from a White ethnic background were more satisfied than the global score and students from all minority ethnic backgrounds were significantly less satisfied than the global score. Similar trends were seen in the question categories Academic support, Learning resources and Overall satisfaction. However, the results were not all of statistical significance, and the analysis does not take into account additional factors such as subject area or institution which may also affect scores Minority ethnic groups in this context are defined to be students from Asian, Black, Chinese, Mixed or Other ethnic backgrounds.

Level of Study

65

67

69

71

73

75

77

79

81

83

85

2006 2007 2008 2009 2010

% Agree Overall Satisfaction by Year, by Level, at UoB and the Sector

UoB First Degree

UoB Other Undergraduate

Sector First Degree

Sector Other Undergraduate

‘The largest differences from the global score for respondents studying for foundation degrees and other qualifications are for question categories Assessment and feedback, where they are more satisfied than first degree students, and Organisation and management, where they are less satisfied.’ (HEFCE 2010)

Disability

60

65

70

75

80

85

2006 2007 2008 2009 2010

% Agree Overall Satisfaction, by Year and by Disability Status at University of Brighton.

UoB Disability (excl Dyslexia)

UoB Dyslexia

UoB No known Disability

60

65

70

75

80

85

2006 2007 2008 2009 2010

% Agree Overall Satisfaction, by Year and by Disability Status in the Sector.

Sector Disability (excl Dyslexia)

Sector Dyslexia

Sector No known Disability

‘Disabled students were significantly less satisfied than the global scores in five of the seven question categories (see Figure 12). The question categories with the lowest satisfaction scores were Organisation and management and Overall satisfaction. ‘ (HEFCE 2010)

NHS Funded Students

60

65

70

75

80

85

2006 2007 2008 2009 2010

% Agree Overall Satisfaction, by Year, by NHS, at Brighton and the Sector

UoB NHS student

UoB Non-NHS student

Sector NHS student

Sector Non-NHS student

‘the profile of satisfaction scores for NHS-funded students was significantly different to that of non-NHS funded students. The question categories with the largest differences were Organisation and management, where NHS-funded students were less satisfied, and Personal development, where NHS-funded students were more satisfied.’ (HEFCE 2010)

Optional Questions

40

45

50

55

60

65

70

75

80

85

90

2007 2008 2009 2010

% Agree satisfied with Work Placements, by Year, UoB compared to the Sector

Brighton (University of)

England

HEI

Top Quartile

40

45

50

55

60

65

70

75

80

85

90

2007 2008 2009 2010

% Agree satisfied with Feedback from Students, by Year, UoB compared to the Sector

Brighton (University of)

England

HEI

Top Quartile

40

45

50

55

60

65

70

75

80

85

90

2007 2008 2009 2010

% Agree satisfied with Intellectual Motivation, by Year, UoB compared to the Sector

Brighton (University of)

England

HEI

Top Quartile

40

45

50

55

60

65

70

75

80

85

90

2007 2008 2009 2010

% Agree satisfied with Social Opportunities, by Year, UoB compared to the Sector

Brighton (University of)

England

HEI

Top Quartile

• Turn the presentation into a more detailed paper – or wait to next year and do a copy of the HEFCE

• Satisfaction by Campus

• Partner Colleges – report available from SPO

Further Analysis

HEFCE Publication - National Student Survey Findings and Trends 2006 to 2009. http://www.hefce.ac.uk/pubs/hefce/2010/10_18/

http://www.heacademy.ac.uk/ourwork/supportingresearch/nss

Literature

Comparative review of national surveys of undergraduate students

Degree Attainment, ethnicity and gender

Dimensions of Quality

Enhancing and Developing the National Student Survey

EvidenceNet Wiki pages on the NSS

Exploratory evaluation of the use of the National Student Survey Results Dissemination Website

Exploring the NSS: Assessment and Feedback Issues

Interpreting National Student Survey (NSS) data using multi-level modelling

Multilevel analysis of discipline differences - Herb Marsh (August 2008)

National Student Survey (NSS) institutional case studies

National Student Survey: Findings and trends 2006 to 2009

NSS 2007 optional items - Institution Guide and Statistical Report

NSS Resources on EvidenceNet

Paula Surridge's report: NSS Three years on

Why we shouldn’t use student surveys of teaching as satisfaction ratings