DLHE Good Practice

17 September 2009

London

Objectives

• Introduce good practice within DLHE record

• Demonstrate tools designed to aid good practice

• Provide institutional perspectives on completion of survey and data collection

• Highlight data quality issues and checking

Target response rates

Target response rates

• 80% for FT UK domiciled students (including Research Council students)

• 70% for PT UK domiciled students

• 50% all other EU

Difficulties in meeting targets

• Increasing student mobility reduces the reliability of contact information

• Public scepticism about what personal data is held against them at a governmental level

• Survey saturation

• High targets set

Current response rates

• 78.5% for FT UK domiciled (-0.4% when compared with 2006/07)

• 70.7% for PT UK domiciled (-0.4% when compared with 2006/07)

• 54.7% for all other EU (+0.7% when compared with 2006/07)

Preparing for DLHE 2008/09

April target list system

• Uses a student file from that year

• Reduced validation

• Generates an accurate population file to survey against

• System currently underused

Population

• Important to check the POPDLHE file generated from the Student Record submission…

• …make sure you are not missing students…

• …or that no one is going to be surveyed that shouldn’t be

• The Match Report will show you how successful you were at this

Know your graduates

• Up-to-date contact information

• Some graduates will like to complete forms online, others will not, so offer as many methods of completing the survey as possible

• Call at the right times

• Identify your part-time and ITT graduates

Sections D and E

• Ensure part-time and/or ITT students complete the sections relevant to them

• Failure to do so will result in validation warnings

• Be aware of students who changed to a PT mode of study at the end of their course but studied the majority as a full-time student

Telephone interview

• Ask the respondent the core questions first if graduate is in a hurry…

• …without them answered you do not have a valid response

• Clarify contradictions or information that would appear incorrect

Location information

• Location of employment information is very important to statutory users of data

• Allows analysis of student’s geographical movement from entry to HE to the labour market

• Try to code to full postcode, however at least outward postcode level or town postcode should be reported

Audit

• The data should be kept only as long as is necessary

• The completed questionnaires should be kept (as either a hardcopy or electronic version) for each individual leaver for one year after the data is returned to HESA

• If the purpose is research then data may be kept for a longer period of time

• There is no need to anonymise the data

Aardvark system changes

• From now on, in addition to the Access Codes, users will also need a PIN code to create new accounts and/or add permissions to existing accounts. The PIN codes will be distributed by letter a few days in advance of the Access Codes

• Both the Access Codes email and the PIN letters are sent to the nominated record contact at institutions

• The system will now send an email to the transaction owner when a transaction has completed processing

DLHE online service

Why use it?

• Help to improve response rates by offering an alternative to paper or phone based questionnaires

• Students can complete it at their own accord and at their own pace

• Intuitive and free to register and use

• Built in validation

System Usage

Percentage of HEIs using system

31

43

53

0

10

20

30

40

50

60

S05018J S06018J S07018J

Online DLHE Questionnairefor both January and April census

Graduates complete form at www.dlhe.ac.ukor www.dlhe.ac.uk?INST=nnnn

to bypass institution selection

Institutions download data from submit.hesa.ac.uk

DLHE Online Statistics 1

DLHE online completed surveys

0

5000

10000

15000

20000

25000

30000

35000

40000

45000

S05018J S06018J S07018J

Completed Surveys

DLHE Online Statistics 2

% Response rates of HEIs using S07018 for first time

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

A B C D E F G H I J

C06018

C07018

DLHE Online Statistics 3

Percentage response rates

78.478.6

76.2

76.7

75.0

75.5

76.0

76.5

77.0

77.5

78.0

78.5

79.0

C06018 C07018

Institutions usingDLHE online

Institutions notusing DLHE online

Uses of online system

• Existing institutions have used the system in different ways…

• …either as an additional tool to the mailings and telephone follow-ups…

• …or as part of a full integration policy

• Many institutions also use the system as a data inputting tool for staff

Registering for system

• Ahead of each DLHE survey you must pre-register for use of the system by the date we publish…

• …and as such requires incorporation into planning

The Aardvark system

• Register to use DLHE online on Aardvark

• Control panel allows you to specify your requirements for system e.g. inclusion of institution's student number or including a PIN

• Download the data of completed surveys from Aardvark

CASCOT

CASCOT (IER)

• CASCOT is designed to assign a code to a piece of text. e.g. a SOC code to a job title from the DLHE survey

• When CASCOT assigns a code to a piece of text it also calculates a score from 1 to 100 which represents the degree of certainty that the given code is correct

• Available to download from: https://submit.hesa.ac.uk/aardvark/CASCOT.ASP

Benefits of using CASCOT

• Ensures consistency within institution and also across sector

• Reduces difficulty of interpreting text

• Improves efficiency and reduces burden of data inputting

Potential problems

• The quality of coding depends on the quality of the input text:

"advertising & marketing" 

- These are the area of work. Are they an advertising executive, an advert designer or an advertising salesman?

"answering phone enquiries“

- This is a job description, not a title.

Working with batches

• CASCOT can be used to code on a record-by-record basis or can work in batches

• Input files allow user to code as many records as they require

• Setting up an output file will then store the results in a spreadsheet for you

Interpreting results

• Always interpret the information accurately…

• …however you do not have to use the recommendation made by CASCOT (regardless of the score it is given)

• Use your judgement to assess whether it is the most accurate code and reflects the true nature of the job

For example

• Student employed as an architectural model maker for which CASCOT gives the following code…

• ‘Modeller (Architectural) 5315’

• …however that is classified as a non-graduate job…

• …whereas ‘Modeller, Artistic 3411’ is a graduate job   

DLHE good practice

The School of Pharmacy

17th September 2009

Mark Gittoes

Head of Quantitative Analysis for Policy, HEFCE

m.gittoes@hefce.ac.uk

Areas of use

• Institutional/subject area reporting

• Research

• Understanding of provision

• Overview of HE

• Other bodies

Institutional/subject area reporting

• HEI Performance indicators

• Teaching quality information and UniStats

• Regional profiles

HEI Performance indicators (1)

• Proportion of graduates who are working or studying (or both) by institution

• Currently published for full-time first degree graduates

• Extending to other cohorts

• Interest in the development of other indicators

HEI Performance indicators (2)

Unistats (1)

• “Employment prospects” broken down by institution and subject area

• Aimed at potential students

• Reports:

– Employment status

– Top 10 profession types

– Job category (graduate, non-graduate)

Unistats (2)

Research

• Student ethnicity

• Job types

Student ethnicity (1)

• Relationship between a student’s ethnicity and experience in HE:

– Entrant profiling

– Progression through HE

– Attainment in HE

– Satisfaction

– Graduate destinations

Student ethnicity (2)

50%

55%

60%

65%

70%

75%

80%

85%

90%

95%

100%

White Black Pakistani orBangladeshi

Indian, Chineseor Other Asian

Mixed or Other

Per

cen

tag

e em

plo

yed

or

stu

dyi

ng Young Mature

Research: Job types (1)

• Examination of Standard Occupational Classification (SOC)

• Study of relationship with student perception of criteria needed for job?

• Makes use of responses to DLHE question:

– “Would you have been able to get the job you were

doing on 14 April 2008/12 January 2009 without the

qualification you recently obtained (the actual

qualification, not the subject of study)?”

Research: Job types (2)

Understanding of provision

• Strategically important and vulnerable subjects

• Foundation degrees

• Erasmus and placement courses

• Further Education Initial Teaching Training

SIVS: Early careers of graduates

“To address the lack of information on the supply and demand of STEM skills, HEFCE should … publish an annual report describing: undergraduate subject trends; recent graduate jobs and salaries; and the subjects where employers and government departments believe that there are, or are likely shortly to be, shortages of graduates with key skills.”

SIVS: Early careers of graduates

Destination No. %

Studying (not employed) 1,850 47%

Studying and employed 825 21%

Employed 1,015 26%

Total employed 1,840 47%

Unemployed 105 3%

Other 110 3%

Total DLHE respondents

3,900 100%

Note: Full-time foundation degree qualifiers registered at English HEIs, 2005-06

Foundation degrees: destinations

Erasmus and placement courses:Salary six months after qualifying

£-

£5,000

£10,000

£15,000

£20,000

£25,000

Erasmus Placement year Other 3 yeardegree courses

Other 4 yeardegree courses

Type of degree undertaken

Mea

n s

alar

y o

f g

rad

uat

es

Other bodies• Office of National Statistics

– UK migration mapping

• Department for Business, Innovation and Skills

– Parliamentary questions

– Research

• Research Councils UK

• Higher Education Policy Institute

– Male and female participation in HE

• Other UK HE funding bodies

Quotes

• “It could be that males are more likely to be rejected for the jobs that they apply for, …, and females are experiencing discrimination in the salaries that they are paid.” Male and female participation and progression in HE, HEPI June 2009

• “Graduate employment data makes grim reading” The Times Higher, July 2009

• “More graduates finding work” The Times Higher, August 2008

The future?

• Move from DIUS to DBIS

• Increased focus of employability

• Development of additional indicators

• HE framework and capturing institutional strength

DLHE Good Practice Seminar: 17th September 2009

Nathalie ReesSenior Careers Information

Officer

Background & current role

• Survey experience

• Review of First Implementation& good practice document

• Survey co-ordinator/manager at the University of Chester

• Departmental contact for data

Who makes DLHE work @ Chester?

But also:

• MIS colleague

• Temporary telephone team

• Career Link Tutors

• Departmental Administrators

Procedures & training• Preparation: procedures checklist

• Training for new staff members: coding and SITS

• Regular updates at team meetings; keeping the team informed of progress

Mailing & processing questionnaires

• Preparation for first mailing & online survey

• First mailing

• Processing incoming questionnaires

• Coding

Telephone follow-up

• Recruitment of phoners

• Training

• Documentation/resources

• Quality checking

Liaison with departments

• Contact details

• Obtaining information

• Recruitment of students for telephone follow-up

• Raising awareness of DLHE survey

Inputting & data quality control

• Tools used

• Coding checking, inputting & validation

• Spelling & postcodes

• Excel spreadsheets – MIS partnership

Challenges

• Up-to-date contact details

• Staffing/administration of the survey

• Survey overload

• Meeting targets for specific cohorts etc…

Room for improvement

• Updating procedures/documentation

• Increasing online replies

• Using DLHE data

• Survey awareness raising with final years & academic departments

Internal use of DLHE data

• Academic departments

• Careers Advisers

• Publications & website

Contact Details

Email address: n.rees@chester.ac.uk

Telephone: 01244 513066

Any questions

DLHE Good PracticeUniversity College Falmouth

Contents of Presentation

• Profile of University College Falmouth• DLHE outcomes at Falmouth over the last 10 years• Outline of the DLHE process at Falmouth• The role of Alumni• Lessons to be learnt• Tricks of the trade• Wish list

Profile of University College Falmouth

• Small HEI – just under 3000 students• Specialist – Art, Design, Media and now Performance• Practice based – most courses are studio based with

students spending 80% of their studies doing hands-on practical work & 20% of their time on written work.

• No exams – continual assessment• High levels of dyslexia and mental health issues

DLHE at Falmouth over the last decade

• Admissions staff at Falmouth have responsibility for HESES, HESA, DLHE, Enrolment, Assessment Processing, Student Transcripts, Open Days etc.

• Average response rate is just above 50%, Falmouth’s employability rate is normally between 91%-94%.

• The worst employability rate came the year we

employed an ex-BBC researcher to work on the DLHE for 6 months. Response rate went up to 97%. We came bottom of the employment tables with 43% of graduates obtaining jobs. Luton University came top with a mere 42% response rate.

Outline of the DLHE process at Falmouth

Inclusion of a letter in their graduation pack

Letter & form to their home address

Letter & form again to their home address

Using ‘text tools’ to send text reminders to mobile ‘phones

A postcard ‘where are you’ to their parents asking them to forward on any envelopes bearing the College logo.

Emailed reminders

Phoning graduates during the day

Phoning graduates between 6 & 7:30 pm

Utilising tutors private Alumni

Use other contacts and other methods

The Role of Alumni

Tutors tend to operate a private Alumni, whilst often effective for DLHE purposes, transgress the boundaries of the Data Protection Act.

Falmouth have established an official Alumni in the last 18 months which it is hoped will improve communication with graduates by maintaining up-to-date contact details, networking, course cameraderie and another conduit for the DLHE.

Lessons we have learnt

• Visual thinkers such as art and design students do not engage well with the DLHE official questionnaire

• 31 questions are 27 questions too many for our graduates• Students are suffering from questionnaire fatigue• Postcards rather than letters have a better response rate• Mothers, house-mates, student ambassadors are invaluable in

nudging graduates and/or providing new contact details• Graduates with dyslexia or mental health issues do not engage

well with the DLHE• Cheap tricks like ipods or book tokens don’t work• Don’t try too hard to contact 100% of your graduate cohort• Graduates tell lies or bend the truth.

Tricks of the trade (1)

• Use student ambassadors, they know who knows who and they talk the same language (doh..)

• Utilise technology such as social networking sites e.g. face book, friendsreunited, twitter.

• Parents and mature graduates prefer printed post e.g. reminder postcards and stamped addressed envelopes.

• Graduates prefer electronic communication – invest in ‘text tools’ to perform mass SMS texting, download spreadsheets of private email addresses to upload into Outlook and b.c.c DLHE messages to graduates

• Ask course teams to feed back on references for graduates.• Set up an Alumni; use case studies in Alumni newsletters.

Tricks of the trade (2)

• In ‘phone and text chase-ups, there are 4 key questions to ask• Have you got a job, where, who with, doing what.• You can use the web to find out post codes (Q10), what the

organisation does (Q9), size of organisation (Q11), the type of school (Q26, Q27) etc.

• The kind of work the graduate is in, can often provide missing answers – e.g. you don’t need a degree in Graphic Design to work as a shelf stacker in Asda (Q12, Q13).

• Don’t offer gifts, extend the use of workshops, studios, library to Alumni instead.

• Don’t use careers to give DLHE talks, invite successful Alumni to give DLHE talks to 3rd years.

Our wish list

• Severely cut the number of DLHE questions• Create a visual mind map version of the DLHE• Produce a summarised parent’s version of the special

issue ‘statistics focus’ viewable via http://www.direct.gov.uk

• A DLHE user group on JISCMAIL

79

University of LondonGood Practice

Oliver Gardham &

Julie Cheung-Inhin

80

Introduction

• Who we are• Where we are• How many?• The Careers Group

81

Who We Are

• A ‘family’ of universities and institutes affiliated with or bound to The University of London

• Some colleges constitute one HESA return, others are combined

82

Who We Actually AreRoyal Holloway,

UoLKings College

LondonUCL

Goldsmiths Courtauld IoA Queen Mary, UoL

SOAS School of Pharmacy LSHTM

Cancer Research Heythrop Institute of Education

Birkbeck Royal Veterinary Coll.

St George’s HMS

Others…. UoL Institutes (below)

Warburg Institute UoL in Paris Inst. Advanced Legal Studies

Inst. Classical Studies Inst. Commonwealth Studies

Inst. English Studies

Inst. Germanic & Romance Stud

Inst. Historical Research Inst. Musical Research

Inst. Philosophy Inst. Study of the Americas Millport Marine Biological Station

83

Where We Are

Royal Holloway

84University Marine

Biological Station,Millport

85

Millport

86

University of London Institute in Paris

87

Senate House andStewart House

88

How Many?

• Between 45,000 and 50,000 total graduates

• Around 30,000 of which are U of L• Most colleges request internationals,

due to diversity of London graduates• It increases every year

89

The Careers Group• Central Careers Administration• Products and services designed to help

individuals achieve and maintain job satisfaction throughout their working lives

• Customers include academic institutions, students, graduates, graduate recruiters, and other careers services providers

• Products and services include careers advice, graduate fairs, online services, courses, professional training

• Graduate Research

90

Graduate Research

• Full-time DLHE team of 4 all year round, one of which is the Call Centre Manager

• On-site call centre• Department with a collective 20

years’ experience, including manager with 10 years of experience

91

DLHE Problems

• Spotting trends and problems in time– Bad numbers– Unemployment

• Quality control• Language barriers• Promoting/engaging with the data

92

Bad Numbers

Near-instant reporting on:

• Bad numbers• Unemployment rates• Response rates

93

Quality Control

• Invest in a good capture system• Call afternoons and evenings, week

and weekends if necessary)• Experienced callers• Employee retention is valuable• International codes and time zones• Language diversity

94

Colleges with Language Schools

• UCL (Dutch, French, German, Greek, Hebrew, Italian, Spanish, the BA Modern Languages course allows over 250 combinations of modern languages)

• SOAS (over 50 languages, including Arabic, Amharic, Bengali, Chinese, Hausa, Hebrew, Hindi, Japanese, Korean, Nepali, Swahili, Thai, Tibetan, Urdu, Vietnamese and Zulu)

• King’s (Arabic, Bengali, Catalan, Dari, Farsi, French, German, Greek, Gujarati, Hebrew, Hindi, Italian, Japanese, Mandarin, Punjabi, Polish, Portuguese, Russian, Sanskrit, Spanish, Swedish, Turkish, Urdu)

• Royal Holloway (French, German, Hispanic Studies, Italian)• Birkbeck (French, German, Japanese, Spanish, Portuguese)• Goldsmiths (French, German, Spanish, Portuguese)• Queen Mary (French, German, Hispanic, Russian)• Institute of Germanic and Romance Studies (French, German,

Hispanic, Italian)• University of London Institute in Paris (French)

95

Engaging with the Data

• Familiarise yourself with both survey and registry data. Over 100 fields of information are available for each graduate allowing simple or complex cross-referencing

• Prepare a menu of available reports for academics or other interest groups who may not be aware of DLHE’s potential, for example…– Graduate level work and/or post-graduate study– Destination summaries– Work sectors– Geographical reporting– Top ten employers/jobs

• Allow for various breakdowns/versions for each report, for example– PT/FT, UG/PG, UK/EU, Department, Course, Gender, Ethnicity,

Class

96

SIC/SOC & SectorsSIC CODE SIC Description The Careers Group Sector

6010 Radio broadcasting Motion Picture Production/Sound Recording/Broadcasting

6020 Television programming and broadcasting activities Motion Picture Production/Sound Recording/Broadcasting

6100 Telecommunications Telecommunications

6110 Wired telecommunications activities Telecommunications

6120 Wireless telecommunications activities Telecommunications

6130 Satellite telecommunications activities Telecommunications

6190 Other telecommunications activities Telecommunications

6200Computer programming, consultancy and related

activities Computer Programming, Consultancy & Related

6201 Computer programming activities Computer Programming, Consultancy & Related

6202 Computer consultancy activities Computer Programming, Consultancy & Related

6203 Computer facilities management activities Computer Programming, Consultancy & Related

97

A Few of Our Sectors

The Careers Group Sector

Architectural/Engineering Activities

Arts, Entertainment & Recreation

Building & Landscape Services

Computer Programming, Consultancy & Related

Construction

Motion Picture Production/Sound Recording/Broadcasting

Telecommunications

Transportation & Storage

98

“Helping Graduates in Work” Website

• Mayor of London website• http://

www.london.gov.uk/graduates/Default.aspx

• “What can I do with my XXXX degree?”

• Jobs, salaries, employers• “What does a XXXX do?”

99

Destinations Website

• Online access to wide range of reports for all DLHE years with different levels of access for Head of Careers Services, Careers Advisers, academics

• Destinations Breakdown, Top Work Locations (using Google Maps), Average Salary, Employment Index, Graduate-Level Employers, and Graduate-Level Jobs

• Institutional breakdowns at UoL, college, department and course level

• Different levels of log-in

100

Summary

• Monitor and report collection data asap

• Retain staff where possible• Be prepared for international calling• Make the data interesting and

useable for academics/CAs/principles• Give people reporting options

101

Thank you

102

Q & A?

DLHE Data Quality

Exception checks

• Designed to pick up peculiarities within the data

• Can be errors or warnings

• Feedback to user given at point of ‘COMMIT’ transaction

• Significant exceptions will be raised by Data Quality Assurance at HESA

Common issues

• Students working and studying in full-time modes

• Incorrect salaries being reported – salary information should be reported as FTE

• Don’t return ‘0’s for salary if information not collected - use Xs instead

• Part-time and/or ITT students not completing the relevant sections

• Incomplete postcodes – unknown regions

Teaching Quality Information

• Teaching Quality Information (TQI) is displayed in three tables:

- Table 5: Destinations information

- Table 6: Job categories information

- Table 7: Common job types information

• Ensure you check each table as the information is widely used…

UNISTATS

• In 2008 – 565,250 unique visits

• In 2009 so far – 409,157 unique visits (comparable time in 2008 there had been 299,982 unique visits)

• International readership

• Significant advertising campaign placing links to UNISTATS on popular websties

Table 6: Job categories information

Check documentation…

Future developments

Catherine Benfield

Review of Early DLHE

• Implementation 2011/12 (Apr 2012/Jan 2013)

• Review will take place from Autumn 2009 and be completed by February 2011

• Usual HESA review procedures including review group with institutional representation

The review will include…

• Revisions to the questionnaire and telephone script

• Review of supporting documentation

• Development of online systems and support

• Technical update to use XML

• Other aspects of the survey

Review items – questionnaire and telephone script

• As well as the usual DLHE review items which include…– Reassess core and non-core

questions– Ensure routing works (issues with

sections D and E)– Any design improvements– Explore any problems that have

been reported

Review items – questionnaire and telephone script (cont.)• …there are some specific items for

review/consideration this time round:– Remove NHS question (Q8)– Update method categories to reflect the

current options– Capture of free text information– Q1 and Q2 options – Use of JACS for Q21 (subject area of

study or training)– Section E (part-time leavers)– How the survey works for self-employed

leavers

Review items – supporting documentation

• Review letter and email texts

• Review the methodology document

• Possible SOC 2010 implementation

SOC10 implementation

• SOC 2010– Will be finalised by ONS soon– There are significant changes planned,

particularly to Major Group 1 (Managers)– Some of the disaggregation currently done

at 5-digit level will be available at 4-digit (e.g. separate codes for barristers and solicitors)

– No decision yet on the need for a 5th digit– “Graduate jobs” will need to be re-specified

Review items – technical update to XML

• eXtensible Markup Language

• The HESA Student Record has been translated to XML

• Data standards - Information Standards Board

• Hierarchical data structures

Review items – other issues

• Coverage– BIS is commissioning research to

investigate feasibility of including international students in DLHE. This will feed into DLHE development

– There is ongoing discussion about the exclusion from DLHE of those who receive their awards from dormant status

– Multiple qualifications for a student - HIN

Review items – development of online systems and support

• Investigate the development of an input-only version of the online questionnaire

• Remove the HTML provision in order to focus on improving the DLHE Survey System

• Assess the use of the April target List System and whether or not it can be removed

• Look at how the DLHE Survey System is hosted

The future for the Longitudinal Survey?

• Results of second survey published earlier this month

• The funders of the survey (Funding Councils, DELNI, TDA and RCUK) will take a decision on the future of the survey in the next few weeks

• Quite separately, RCUK intend to undertake further follow-up with the 2004/05 cohort of PGR graduates

Operational Training

• DLHE Introductory seminar

• 9 February 2010 in Cheltenham

• www.hesa.ac.uk/seminars

Institutional Liaison Helpdesk

• Telephone 01242 211144

• Email Liaison@hesa.ac.uk