The business case for academic development

A symposium hosted by:Linda Creanor, Glasgow CaledonianCathy Gunn, AucklandNeil Lent, EdinburghKeith Smythe, Napier

Does evidence align with stakeholder priorities?

• Why ask the question?

• Experience shows it’s hard to answer• Different forms serve different purposes

Report on activity

Evidence to inform practice

Report on activity

• Accountability

• Progress on strategic initiatives

• Show return on investment

• Raise awareness

Inform practice

• Educational research described by one author as ‘alchemy’ (Sloman 2012)

• Literature notes poor links between theory and practice in learning technology research

• Scholarship of teaching / learning design research is fine in context.. not generally

Evaluation: Understanding cause and effect

Dr Neil Lent Institute for Academic Development,

University of Edinburgh

Not always easy, for example:

The Quality Enhancement Framework for learning and teaching in Scottish

higher education (QEF)

Principles underlying the QEF

• Collegiality• Consensual development = greater

ownership• Alignment of aspirations to specific

actions not general exhortation• A relatively ‘light touch’ is most likely to

yield improvement

‘Impact’

• Problem of certainty • Difficulty of establishing lines of determination• Diagnostic function: does it work, does it fit our

values and vision of enhancement• Alignment of values and practices not

measurement of cause and effect

Culture(s) of enhancement

• Culture as practice and discourse• Away from what is towards an aspiration

towards what could be• Review and reflection embedded and

internalised way of life• Multiple stakeholders • Risk-taking• Sharing good practice

How can we evaluate cultural change in a system?

• Build a model of cultural values for a system:who is there?what they do?what they use?

• Look for alignment of practices and values observed with those associated with the desired culture

• Think of activity in terms of loose frameworks and relationships between people and subsystems

10

The case for Blended Learning: shaping the evidence

ContextGlasgow Caledonian University:• Post-1992 with a strong widening

participation agenda• Blended learning integral to vision• Restructured in 2011 (again…)

Academic Development & Blended Learning• Recent separation of development and

research• Blended Learning team of 3 in Centre for

Learning Enhancement & Academic Development (GCU LEAD)

• 9 Learning Technologists in 3 Schools

Academic Development & Distributive Leadership

CPD framework influenced by Australian Faculty Scholars Distributive Leadership Model (Lefoe, 2010)

“... a distribution of power within the sociocultural context of universities, and a sharing of knowledge, of practice and reflection through collegiality.” (Lefoe et al, 2007:5)

Reporting on Blended Learning

BLENDED LEARNING REPORT

to Academic

Policy Committee & Senate

School comparisons

Strategic developments over previous year

VLE usage stats

Average no. accesses per student

%-age of modules with ≤ 12 accesses per student

Usage trends for blogs, wikis, podcasting etc.

Analytics to capture broader access trends

“The ‘headline’ data should be interpreted with some degree of caution.”

“… [they] do not present a comprehensive picture of blended learning activity at GCU ....”

“They do not tell us what kind of learning activities students are engaged in online, or how interactive

these might be.”

(Blended Learning Report, 2012)

Caveats….

Includes UK Sector ComparisonUCISA Biennial Survey of Technology Enhanced Learning• For example: web dependent, web enabled, online

– BUT how accurate are these figures?

NSS and ISB scores for teaching and technology use• ‘teaching on my course’, ‘access to resources’ (NSS) • ‘virtual learning’ (ISB)• Useful info mainly from students’ open comments• Most recently the focus was on staff engagement (or

lack thereof) and availability of online resources rather than technical issues.

Positive Outcomes

Reinforces the importance of strategic direction and central co-ordination

Blended Learning Roadmap updated annually

Recommendations made & priorities agreed

Agreed actions shared among - Blended Learning Team School Learning Technologists Information Services Library

IssuesWhat? An over-reliance on quantitative rather than

qualitative evidenceWho? Collaboration, shared responsibility and local

ownership – embedding distributive leadership (Creanor, 2012 – in press)

Why? Highlighting that central co-ordination and expertise remain key to strategic impact

How? Balancing facts & figures with scholarly activity, robust evaluation, case studies ……???

Making the case for taught staff provision: evaluating the personal and strategic impact of

two institutional programmes

Dr Keith SmythOffice of the Vice Principal (Academic)

Edinburgh Napier University

Three framing propositions• Pg Certificate Learning and Teaching (or equivalent) occupy a

central position within many of our institutions, often linked to appointment, promotion and LTA strategy

• Our approach to ‘evaluating’ taught provision for staff is often limited to standard post-module and post-course measures (standard surveys, numbers of completers) that provide little insight into the impact on the individual or their wider, developing role within the institution

• The current debate on the worth of Pg Certs is demanding a richer evaluation of their role and value (Stefani, 2011)

Current context at Edinburgh Napier• Post 1992 teaching-led institution. Office of the Vice Principal

(Academic) is responsible for institutional LTA strategy, and runs two taught programmes for staff.

• Pg Cert Teaching and Learning in Higher Education. HEA/SEDA/NMC accredited. Running for 15 years. All new lecturers must complete within two years. Blended delivery.

• Pg Cert/Dip/MSc Blended and Online Education. Fully online, applied and practice-based. SEDA accredited, and linked to new institutional TEL strategy and an initiative to establish accredited online educators in every School.

Key challenges• The programme teams know, anecdotally and through standard

methods of evaluation, that the programmes are well received and seen as relevant to practice and wider institutional aims. We also see some evidence of impact through other means, e.g. Teaching Fellow applications.

• However, we do not really know about the range of ways in which the programmes are impacting practice on a personal and institutional level and do not have a strong qualitative evidence base from which to illustrate impact and effectiveness over the medium and longer terms…

• ..and justify the institutions resourcing of the programmes

How we propose to address this• Institutional investigation ‘A study of the effectiveness of

Edinburgh Napier’s Academic Practice Programmes’

• Three months, mixed-methods with a strong emphasis on qualitative dimensions of impact or ‘Illuminating the link between developing staff as individuals and the strategic LTA development of the institution’ (Gosling, 2008)

• Current and former students over last ten years for Pg Cert and last 5 years for the MSc BOE

General issues to be addressed• Impact on LTA practice and student experience

• Recognition of LTA practice internally and externally (e.g. internal LTA Awards, discipline-specific LTA awards)

• Leadership and promotion (supporting colleagues, pathways to Teaching Fellow/STF, programme and subject leader, leading school or university LTA initiatives)

• ‘Stories’ which evidence individual playing central role in LTA strategy implementation locally or across institution

• Publication and scholarship relating to Pg Cert/MSc projects

Issues in TEL practice to be addressed• Nature and impact of the work being undertaken as a ‘local

champion’ through sponsored completion of Pg Cert BOE

• Evidencing own engagement with new TEL strategy in design and delivery of own course provision

• Scholarship and dissemination in TEL

• Recognition of good practice (including through shortlisting for recent TEL related LTA awards, or external recognition)

What we propose to do with this• Produce ‘impact evaluation’ report for University LTA committee

to include institutional and local analyses

• Create case studies to contribute to LTA Resource Bank

• Identify enhancements to the programmes

• Disseminate findings of the evaluation, including the approach taken and data collection instruments used, to allow others to undertake similar evaluations

• Forward plan periodic re-running of evaluation on smaller scale (perhaps tied into programme review cycle)

Tracking the invisible: impact evaluation for elearning development

Dr Cathy GunnCentre for Academic Development, The University of Auckland

Evidence to inform practice

• A case study exploring evaluation challenges

Overt elements Covert elements

Formative = input to design

Summative = impact on practice

Evaluation framework and exampleScoping and planning How to achieve specific learning objectives in

context – literature, theory and practiceDesign specification Accessible / easy to use for target groups,

dynamic and engaging, course focused, authentic tasks with self-testing

Implementation Use in large classes, essential steps to complete assignment, initial support, topic coverage, meaningful activities and feedback

Summative review Learner feedback and grades, lecturer & librarians’ reflections, student interviews, publication & presentation

Further action Model deemed successful, disseminated to other subjects and disciplines, web elements reused, experience supports skill sharing

Outcomes for staff and students

• ALL students able to acquire necessary skills

• Design concept & web elements reused across subjects led by ‘new experts’

• Time saved, autonomy promoted…

• But…impact of elearning development role is hard to quantify

If critical factors are largely invisible…

Online Modules

Find learning edges

Ground & craft design

Build trust / gain buy in

Easy to use tools

… how can we report them?

Summary and discussion

• Neil – cause and effect or align activity with values and vision?

• Linda – over-reliance on quantitative data which paints a partial picture

• Keith – impact evaluation and case studies

• Cathy – invisibility & risk when all runs smoothly AD that is deemed unnecessary

Discuss…

• Are we:

• Asking the right questions?

• Gathering the right information?

• Talking the language of our audiences?

We’d like to share your views

• In an ideal world, what kind of culture and practice would we want in this area?

What constraints & opportunities exist in the less than ideal world we currently inhabit?

Most importantly – what positive moves can we make towards the ideal?

Creanor, L. (2012 – in press), Raising the profile: an institutional case study of embedding scholarship and innovation through distributive leadership, Innovations in Learning & Teaching International

Gosling, D.(2008) Educational Development in the United Kingdom http://www.hedg.ac.uk/documents/HEDG_Report_final.pdf

Gunn, C., & Donald, C. (2010). Tracking the Invisible: An eLearning Group’s Approach to Evaluation In L. Stefani (Ed.), The Effectiveness of Academic Development (pp. 133-142). Routledge.

Gunn, C., & Steel, C. (2012). Linking Theory to Practice Learning Technology Research. Research in Learning Technology, 20(2)

Lefoe G., Smigiel H., Parrish D. (2007) Enhancing HE through leadership capacity development, in Enhancing Theory and Scholarship, Proceedings of the 30th HERDSA Annual Conference, Adelaide, Australia, 8-11 July http://ro.uow.edu.au/asdpapers/56/

Lefoe, G. (2010) Creating the Future: Changing Culture Through Leadership Capacity Development, in Ehlers, U-D & Schneckenberg, D. (Eds), Changing Cultures in Higher Education, part 1, 189-204, Springer Verlag, Berlin, Heidelberg

Stefani, L. (Ed) (2011) Evaluating the Effectiveness of Academic Development: principles and practice. Routledge