Interim Reporting Template - REAP · Interim Reporting Template ... Pharmacy, Psychology ... AUT...

SFC e-Learning Transformation Programme

Interim Reporting Template

Project Name Re-Engineering Assessment Practices [REAP] in Higher Education, University of Strathclyde.

Report compiled by David Nicol and Catherine Owen

With contributions from Gillian Roberts, Linda Creanor, Steve Draper

Reporting period Second Interim Report 1st February 2006 - 31st July 2006

Section One: Summary

During this period each of the five departments participating at the University of Strathclyde (Mechanical Engineering, Pharmacy, Psychology, Childhood and Primary Studies and Marketing) has implemented one or more pilot projects and most have tested a variety of activities and technologies in support of new assessment practices in (mainly) large first year classes. At Glasgow Caledonian University, re-engineering is taking place across a whole faculty, the Caledonian Business School, with specific course re-engineering activities occurring in at least six core modules covering a range of business disciplines including finance, economics, management, marketing, human resources and business research methods. At the University of Glasgow, plans for course re-design to support new assessment practices are under development in the departments of Psychology and Computing Science for implementation during the 2006/07 academic session.

Technologies introduced to support enhanced assessment and feedback processes include electronic voting systems, simulations and virtual environments, message boards and emails, data banks of feedback comments, multiple choice question tests and e-portfolios. Evaluation of these initial re-engineering pilots, using a range of approaches, commenced in January and is due for completion during July 2006. A template for collecting interim case study data has been designed and circulated to participating departments. The REAP team is collaborating with departments/modules to produce the case studies and these will be published on the REAP website in July and August 2006. The pilots have already demonstrated important early results which are informing plans for more ambitious re-engineering activities during the session commencing in October 2006.

The project has developed a robust evaluation strategy which has informed evaluation activities at department level, at project level and across participating institutions. At department or class level, case studies are assessing progress in re-engineering against the drivers initially identified locally, which included improvements in the quality of teaching and learning, efficiency and workload gains and improvements in retention and progression rates. REAP team members are working with departmental teams and class leaders to assess their progress against the pedagogical framework developed by the project and to examine the practical issues associated with changes to assessment and with the introduction of new technologies. The impact of project activities on class design, course delivery, departmental planning and institutional planning is being examined.

The REAP team continue to participate in a number of institutional and community initiatives including the Strathclyde University First Year Initiative, the HEA National E-Learning Benchmarking Exercise and in the Scottish Collaborative e-Learning Benchmarking Club. These activities are informing approaches to evaluating the transformational impact of REAP at institutional level across all three participating institutions.


Section Two: Activities and Progress

Progress against work-packages in the project implementation plan:

Work-package 1: Project Initiation

• REAP project plan and implementation plan

• Consortium agreement

• Recruitment

Complete

Work-package 2: Evaluation Planning

• Baseline data collection/departmental ‘position statements’

• Develop evaluation strategy

• Develop evaluation instruments

• Collect baseline data

Complete

Work-package 3: Prepare for Re-engineering

• Departmental implementation plans

• Refinement of implementation plans

• Establish internal funding agreements

• Preparation for re-engineering

Complete

Work-package 4: First Implementation

• Implement new teaching and assessment practices in participating departments

Five departments have implemented re-engineered learning and assessment practices within first year classes at SU and at least six core modules have re-engineered assessment at Glasgow Caledonian Business School. Considerable work has taken place at Caledonian Business School to address the strategic implications of new assessment approaches, and within various strategic committees at the University of Strathclyde to address the roll-out of new practices across the institution. At Glasgow University two departments are currently drawing up course re-design plans (see Section Three: Outputs, Outcomes and Deliverables for details).

• Evaluation instruments

[See section six: Evaluation]

• First interim report on transformational change

[See section four: Progress towards Strategic Transformation]

Work-package 5: First evaluation and review

• Gather evaluation data

• Refine evaluation instruments

Complete [See section six: Evaluation]

• Produce first interim report on transformational change

In production for publication end September 2006 [See section four: Progress towards Strategic Transformation]



Work-package 6: Second Implementation

• Refined re-engineering plans

Refined project plans for second implementation are currently in development in all participating departments.

Work-package 7: Internal Dissemination

[See section five: Dissemination for details of internal dissemination activities]

Work-package 8: External Dissemination

[See section five: Dissemination for details of external dissemination activities]

Work-package 9: Second Evaluation and Review

• Finalise and publicise evaluation instruments

A paper detailing the REAP project’s approach to evaluation was accepted in June 2006 for publication to the British Journal of Educational Technology (BJET)

• Gather evaluation data on transformational change

• Develop senior management briefing paper(s)

• Develop papers on transformational change at institutional level

A paper on transformational change has been accepted for presentation at ALT-C in September 2006.

• Produce a book on technology-supported assessment

In production for provisional publication Autumn 2006.

Work-package 10: Project Management

• Local Management Groups

Local Management Group established at SU to support project management activities and to facilitate links to relevant institutional initiatives, support services and departments. This group meets monthly and is also a forum for communication from project team leaders at GCU and GU and from participating departments. In CBS, there have been frequent meetings, generally involving all participants in the project (module leaders and e-champions) and monthly opportunities for e-champions and the project team to plan activities.

• Project Steering Group

This group includes senior representation from consortium partners and from key external bodies including the SFC, QAA, HEA and others. This group met in November 2005 and June 2006 and convenes on a six-monthly basis to provide guidance on the strategic development of the project.



• Project Websites

[See section five: Dissemination for more details of internal and external dissemination]

The REAP website (www.reap.ac.uk) has undergone a significant expansion and re-design and includes a growing number of project documents including case studies, reports and research papers. The site includes tools for intra-project communication and document-sharing.

Two project mailing lists has been established: one to support communication between the 70-80 project team members at three universities ([email protected]), and one to support external dissemination about project activities and events (reap-public @jiscmail.ac.uk).

CBS also maintains its own project website to enable e-champions and module leaders to access and share information with the REAP team.


Section Three: Outputs, Outcomes and Deliverables

Main project deliverables to date include the pilot implementations including case studies, evaluations and dissemination events (the latter are dealt with in Section Five).

University of Strathclyde

University of Strathclyde: Department of Mechanical Engineering

First interim pilot January – June 2006:

REAP-supported developments in the department include:

• The use of online MCQ tests before an interactive EVS classroom session. This enables in-class teaching to be targeted to the students’ needs and level of understanding (called just-in-time teaching) and helps integrate out-of-class and in-class learning.

• Piloting Confidence Based Marking (CBM) during EVS sessions in which students rate their level of confidence in their answer and are marked accordingly (encourages reflection on answer)

• Piloting the use of the Mastering Physics Intelligent Homework System to support reflection and self-assessment during problem-solving activities.

Interim results and future plans: The use of the Mastering Physics Intelligent Homework System has shown that significant efficiency gains are possible. The department expects to reduce staff marking time from 50 hours to 15 hours per tutor (or 75%) during the planned two-semester implementation in 2006/07. Mastering Physics also provides staff with accessible and timely diagnostic data on student performance which supports ‘just-in-time’ teaching activities. Immediate feedback from the homework system and online MCQ tests allows students to identify and address gaps in their knowledge without recourse to tutors and to make better use of face-to-face opportunities. In-class learning will be reinforced by the use of confidence-based marking (CBM) during EVS class tests. The first evaluation of the student experience of CBM identified that students require time to adapt to the new marking regime (CBM involves negative marking). CBM encourages students to engage in extra cognitive processing by reflecting on the level of certainty they have in their own knowledge and understanding. This extra processing should help develop self-regulation of learning. The department intends to further trial this process during 2006/07 with support from Dr Steve Draper from the University of Glasgow. Pilot deliverables and dissemination to date:

• REAP interim case study available from: www.reap.ac.uk

• Paper presented by Professor Jim Boyle at the Institute of Mathematics and its Applications (IMA) conference on ‘The Mathematical Education of Engineers’, Loughborough, March, 2006

• Paper presented by Dr David Nicol at: CAA Conference, Loughborough, July 2006

• Presentation at University of Strathclyde REAP/ Learning Enhancement Network (LEN) event, April 2006


Section Three: Outputs, Outcomes and Deliverables (cont…/)


University of Strathclyde: Department of Marketing



• The use of WebCT as a repository and distribution hub for re-designed class notes and new supporting course materials which were made available to students prior to lectures. This encourages better use of face-to-face teaching contact time in that it can focus on debate and dialogue rather than on information transmission.

• The introduction of online multiple choice question tests to support both formative and summative assessment activities and enhance student feedback opportunities.

• The use of an electronic bank of comments to provide feedback on students’ essays and other assignments and an online pro-forma for marking essays and distributing feedback to students.

Interim results and future plans:

The introduction of these three mutually-supporting technologies, in conjunction with a significant re-design of course content and re-allocation of teaching responsibilities in the first year class has contributed to an increase in progression rates (students electing to continue studying the subject into second year) from 40% to 60%. The department has identified a number of practical barriers that have had a negative impact on tutors’ engagement with the online comment bank and marking pro-forma and which will require a re-design prior to the second round of implementation. Also, student engagement with MCQ tests was variable (90% of students took practice tests in preparation for a summative exam, but formative testing opportunities linked to course materials were less popular, attracting 55% of the class). The department plans to engage in further re-design to improve alignment between tests, lectures and tutorials in order to address this problem. AUT industrial action had a negative impact on the department’s ability to implement and evaluate some activities during this period.

Pilot deliverables and dissemination to date:



University of Strathclyde: Department of Primary Education



• The introduction of e-PDP software (PebblePad) to support personal development planning in a core first year module Skills for Learning

• The preparation and support of students and staff in the use of PebblePad in teaching and learning


After initial testing with a small group of second year students and training and surgery sessions hosted by the REAP project team, the e-PDP tool was made available to first year students on placement in schools. A number of structured tasks were set to encourage student engagement with the software, but during this first pilot phase students were able to choose whether to create an electronic portfolio in PebblePad or maintain a traditional paper-based portfolio. Early outcomes from the pilot suggest that students have welcomed the availability of communication tools within PebblePad including blogs, document sharing and email alerts. The software has been shown to support peer and tutor feedback process both on campus and when students are away from campus on school placements. The department has been successful in securing a long-term commitment from the university’s IT services to support PebblePad. The Faculty of Education is also keen to roll out the e-portfolio to support reflection and PDP across the faculty. A Faculty Steering Group has been formed to look into this and already there are plans to support other first and second year modules using PebblePad during session 2006/07.

(cont…/)





• Workshop at University of Strathclyde Learning Enhancement Network (LEN) event, March 2006


University of Strathclyde: School of Pharmacy



• Integrated e-PDP implementation and activity alignment in two first year classes Personal Skills Development and Foundation Pharmacy

• Development of online prescription simulation tutorial to support third year class Pharmacy Practice


In previous years, first year students engaged in personal development planning as part of a discrete module, Personal Skills Development using an e-PDP tool within the Pharmacy VLE. This raised issues about the compartmentalisation of reflection within the context of this module rather than seeing reflection as a process occurring within all first year modules. Changes have been piloted with the first year Foundation Pharmacy class where tutor feedback and student reflection on a written assignment were integrated with the record of learning deposited in the e-PDP. Evaluation activities have been delayed due to the AUT industrial action. The Pharmacy School plans further e-PDP integration of additional first year modules and further assessment re-design in the coming academic year.

An online tutorial has been developed and piloted to support the third year Pharmacy Practice class. This enables students to test and refine their skills in identifying errors in prescriptions and to receive feedback without requiring time-consuming face-to-face feedback from staff. Initial student reactions to the pilot were positive. Staff members anticipate significant efficiency and quality gains during a full implementation in session 2006/07. Funding from the REAP project is supporting the development of a databank of 200 prescriptions which will be available from September 2006 to support the scaling up of this re-design of class assessment practices.



• Abstract accepted for The International Journal of Pharmacy Practice.

• Poster accepted for British Pharmaceutical Conference, September 2006

• Submission in development for The Pharmaceutical Journal




University of Strathclyde: Department of Psychology



• Use of WebCT discussion board to structure and scaffold self and peer feedback in the first year class Basic Psychology


Prior to the pilot, the department was concerned about the lack of feedback opportunities available to this large class comprising 550 students. During the pilot, the basic class was re-designed to provide opportunities for constructive self-assessment (scaffolding) linked to supportive peer discussion using the discussion board in the virtual learning environment. The students were introduced to a task that required them to submit individual, then group, responses to progressively more complex questions on psychology topic (e.g. human memory). The group response required them to agree a written answer to the question (forcing them to reach a consensus through peer-group discussions). The students also self-assessed their responses against tutor provided model answers. Evaluation of the pilot demonstrated that students respond extremely positively to the group discussions and that the task helped build a deep understanding of the topic. These findings have given the department the confidence to propose a radical redesign of the first year class commencing in 2006/7, abolishing half the scheduled lectures and replacing these with similar online group exercises across all topic areas and making self and peer feedback core components of the class. These changes will result in both improvements in student learning and efficiency savings in staff time.




• Paper presented by Dr David Nicol at: QAA Integrative Assessment and First Year Experience Initiatives Joint Meeting, Edinburgh, May 2006

• Paper presented by Dr David Nicol at: CAA Conference, Loughborough, July 2006

• Paper by Jim Baxter and Dr Andy Tolmie in production for publication in September 2006



Glasgow Caledonian Business School Subject Groups/Divisions

Glasgow Caledonian Business School: Level 1 Core Module: Marketing Fundamentals


REAP-supported developments in the module include:

• Use of online MCQ tests to support summative assessment

This module is delivered to approximately 900 students taught in two cohorts (one in each semester). 60% of the current summative assessment is based on a multiple choice test taken in examination conditions (the remaining 40% of the mark is assessed via coursework). Although previous paper-based tests had been electronically scanned, a high level of inaccuracy meant that additional marking was also required which was time-consuming and inefficient. Online testing on this scale has not been previously attempted by any other Scottish university and the team addressed a large number of practical issues including identifying accommodation for simultaneous testing of a cohort of 450 students; security; invigilation and technical support needs.

Interim results and future plans: The Semester ‘A’ exam was delivered via the university’s VLE, Blackboard, and held in 23 computing labs across the university with technical support available to each floor. Technical difficulties during the initial pilot in January meant that 258 students (60%) undertook the exam online and the remaining 40% completed their exam on paper. No students were penalised for time. The mean exam score for students in semester A was 53%, 5% higher than the Semester A cohort in session 2004/05, which suggests that overall performance was not adversely affected by the technical difficulties experienced. During Semester ‘B’ the exam was split into two exams of 150 students, morning and afternoon. A total of six labs were used with a corresponding reduction in invigilation needs. This second online exam proceeded without any technical problems. Student feedback was gathered by survey (55% response) and two focus groups. Despite early technical difficulties the vast majority of students responded very favourably to being assessed online and welcomed receiving instant feedback of their mark. The team are now considering the feasibility and desirability of providing enhanced feedback to students beyond a score mark.



• Presentation at Teaching and Learning Showcase, Glasgow Caledonian University, May 2006

• Guidelines for staff on establishing a group policy to ensure security, reliability for hosting ‘high stakes’ online assessments (available from REAP website during August 2006)

• Guidelines for invigilation where ‘high stakes’ online assessments are taken under exam conditions (available from REAP website during August 2006)

• Case study (available from REAP website during August 2006)



Glasgow Caledonian Business School: Level 2 Core Module: Consumer Buyer Behaviour First interim pilot January – June 2006: REAP-supported developments in the module include:

• Introduction of weekly online MCQ tests (25%)

• Introduction of three inter-related, progressively demanding, group-based written pieces of course work (learning papers) with staged submission and feedback delivered via Blackboard (75%)

This is level two core module delivered to approximately 300 students. Drivers for change included concerns about staff workload, poor timing of feedback to students and poor take-up of feedback available and concerns about pass rates (85%) failing to meet university targets. The introduction of weekly online tests based on publishers resources and delivered via Blackboard and progressive, group-based coursework encourages students to remain actively engaged with the module and their own learning on a regular basis. Students receive multiple opportunities for feedback from tests, peer group members and tutors. Interim results and future plans: MCQ tests: Early technical problems associated with publisher’s security mechanisms caused some difficulties for students accessing online MCQ tests early in the pilot. These were resolved relatively quickly and an ‘amnesty’ was provided in week 12 to ensure all students had the opportunity to complete the tests. Students received online qualitative feedback on their learning via question answers and guidance where incorrect; quantitative feedback on performance in the form of marks issued in ‘My Grades’ via Blackboard and tutors used class results to inform the content of seminars in a ‘just-in-time’ teaching format. Key features of implementing the online tests contributing to their success included randomising questions, imposing fortnightly deadlines, promoting open book approach, answering one question at a time. Results from the evaluation of this pilot suggest that students are very enthusiastic about the tests although some staff members were concerned about early technical problems that created stress for both staff and students.

“On-going assessment throughout the year keeps us motivated to learn, concentrate and

continuously read-up on what we are learning. Overall (it) helps us learn more and better.” Student quote

Re-designed group coursework: The concept of three inter-related group coursework was welcomed by most students though some indicated a preference for some individual element. The scheduling of submission dates and the timeliness of feedback to some students was compromised by unanticipated illness affecting all three tutors simultaneously. Tutors reported that, when feedback was received in time, it did influence student performance in the next stage of the coursework and that some students appeared to be better prepared for seminar discussions. There has been some evidence of increased student demand for tutor feedback. The introduction of group work has reduced the overall number of items to be marked and the use of a new feedback pro-forma drawing on a databank of standardised comments encouraged staff to give more detailed feedback which had workload implications. Despite the operational problems outlined in this first iteration, the teaching team believe the changes made are moving in the right direction. This belief is confirmed by student feedback such as the one which follows which was typical:

‘I think the assessment methods in this module were very successful. I believe this is the case as it prompts you to constantly learn and study the whole way through, instead of only studying before the exam. Therefore I felt I was constantly learning and taking what I was learning in a lot better and more productively. Definitely a successful way of assessment.’ Student quote.


• Presentation at Teaching and Learning Showcase, Glasgow Caledonian University, May 2006 (Available now on REAP website)




Glasgow Caledonian Business School: Level Three Core Module: Strategic Management

First interim pilot January – June 2006: REAP-supported developments in the module include:

• Introduction of electronic feedback software to improve the quality and timeliness of feedback to students

Strategic Management is delivered to approximately 700 students by a teaching team of eight, three of whom are full-time and five of whom are part-time members of staff. While the module team is continually re-engineering its learning, teaching and assessment strategy (LTAS) through an annual review process, a central focus is improving the timeliness, quality and consistency of feedback provided to students. Despite team focus of assessment criteria, staff members have identified variability in the quality of feedback provided to students and a tendency for students to focus on marks rather than written feedback provided (many students failing to collect feedback at all). The team have tested the use of electronic feedback software (EFS) to improve the quality and timeliness of feedback to students. The software was designed by Phil Denton of Liverpool John Moores University (www.ljmu.ac.uk/cis/software/feedback.asp). It is based on a Microsoft Office application and can be used by tutors to design, generate and email feedback to students. As well as generating feedback, the software can also be used to analyse the distribution of both marks and feedback comments allocated by markers. It can also be used to detect instances of plagiarism between pairs of students. During the initial pilot five members of the teaching team tested the software on fifty scripts from a previous assessment. Each staff member was asked to complete an evaluation questionnaire designed to capture their perception of the ease of use of the software, the potential for improvements in quality and consistency of feedback and implications for staff workload. Interim results and future plans: The module team found that establishing agreed assessment criteria to load into the software was essential to secure staff ‘buy-in’ to the system but also time-consuming. In large teaching teams individual tacit assumptions and understandings of marking criteria need to be made explicit on a group basis. Use of electronic software feedback facilitates this process and is likely to result in significant improvements in the quality and consistency of feedback but set-up costs are likely to be substantial as a result. Technologies such as EFS might be usefully viewed more as a driver for change, rather than as a panacea for re-engineering assessment processes. A full implementation involving students is planned for 2006/07. Staff comments included:

“The quality of the feedback is far superior to what the students are given at the moment (excluding one to one meetings) ”. “If feedback can be returned to the student electronically I believe that this significantly improves the chances of them reading it and therefore making use of it”


• Presentation at Teaching and Learning Showcase, Glasgow Caledonian University, May 2006 (available from REAP website)




Glasgow Caledonian Business School: Level 1 Core Module: Managerial Finance



• The introduction of online weekly tests (10 tests at 1% each) to improve feedback opportunities during the course and to support weaker students

Managerial Finance is a foundation module for non-accountancy students who have an accounting component in their programme. Prior to the pilot summative assessment included group-based coursework project (20%), two class tests comprising case studies (20%) and a two hour multiple choice question examination (60%). All of these summative assessment activities took place after week nine of the course. Formative assessment opportunities were available during tutorials and students were offered the opportunity to take online quizzes based on publisher’s materials to self-assess their performance during the module. Uptake of these voluntary tests was disappointing and tutors were concerned that student attendance at lectures declined during the course. Tutors suspected that the end-loaded nature of assessments had a negative impact on student performance. Interim results and future plans:

During the pilot, the module team altered the overall weighting of coursework and added one new instrument of assessment. The team awarded 10% of module marks for ten on-line MCQ assessments delivered via Blackboard. Students were required to sign a declaration and to record their first attempt score for each quiz. Thereafter, students could take the test as often as they wished to close their ‘learning loop’. Two independently facilitated evaluation focus groups held with students reported that students felt the online tests were worthwhile and valuable as a study aid and to encourage reading of the course textbook. Students would have preferred greater weighting of the tests and better scheduling with tutorials and lectures with regular deadlines for test completion. Students identified a need for more and better quality feedback on performance in tests. During the second phase of implementation the course team plan to enhance feedback content, to revise weighting and content of all class assessments and to introduce electronic voting into classroom activities to enhance formative assessment opportunities. Pilot deliverables and dissemination to date:

• Case study (available from REAP website during August 2006) Glasgow Caledonian Business School: Level 1 Core Module: Understanding Management



• The introduction of electronic feedback software to improve the quality and timeliness of feedback to students

Understanding Management is a level one core module delivered to approximately 110 accountancy students. Prior to the pilot, assessment included a group presentation (10%), individual report (20%), a multiple choice question test (30%) and a two hour final exam (40%). Assessments have been designed to scaffold student learning across the course. Tutors have identified variations in the quantity and quality of feedback given to students, difficulties associated with a slow turnaround of feedback provision and staff workload issues. Students have demonstrated a tendency to focus on marks rather than improve learning based on feedback comments. (cont…/)




Electronic feedback software (EFS) was used to improve the quality and timeliness of feedback to students based on a 1500 words individual report. The software was designed by Phil Denton of Liverpool John Moores University (see Strategic Management module for link). Feedback comments used were based on prior work from Oxford Brookes University and locally-generated variations. Formal evaluation results are pending. Seventeen staff members from the Strategy Operations and Leadership subject group have attended a workshop on EFS and this software is now being trialled in a number of other modules for implementation during the 2006/07 session.



Glasgow Caledonian Business School: Level 3 Core Module: Business Research Methods



• Introduction of evidence-based portfolio (electronic version implementation 2006/07)

Business Research Methods is level three module delivered to approximately 800 students (including a cohort of Singaporean students with local tutors). The module provides the underpinning theory in preparation for student dissertations in level four and supports early facilitation of dissertation supervision. The teaching/learning strategy for this module is based on a constructivist philosophy of learning, utilising Mayes’ learning cycle (1997) of ‘conceptualisation, construction and dialogue’ All students follow the core lectures but participate in tailored seminar programmes relating to students’ main areas of interest.

Prior to the pilot, assessment of this module comprised a written research proposal submitted in week ten (50%) and a final examination (50%). Tutors recognised that the timing of the coursework submission and the final exam meant students had limited feedback opportunities before commencing work on their fourth year dissertation and as a result students often demonstrated unrealistic expectations of converting their third year research topic into a successful dissertation.


During the pilot existing assessment instruments were replaced with a portfolio submission comprising three pieces of coursework, giving students opportunities for earlier feedback during the course. Initial evaluation suggests that both students and staff welcomed opportunities for earlier feedback and a move away from a final exam which did not support integration with fourth year work. However, staff members have identified changes to workload associated with an increase in coursework (although they are no longer required to mark examination scripts) and are likely to require additional support in providing consistent feedback to a large number of students. The REAP team are helping staff members to identify and implement appropriate software to support electronic portfolio development during the academic session 2006/07.


• Case study (available from REAP website during August 2006) Glasgow Caledonian Business School: Piloting and evaluating electronic voting systems in lectures


This pilot was not aimed at course re-design in the first instance but rather to trial the potential of two alternative EVS products to introduce formative feedback in large lectures. Assisted by colleagues from Strathclyde and Glasgow Universities, this is a collaborative exercise involving staff development and technical advice. Evaluation of EVS products in being continued with a view to selecting one for use in core modules with all CBS first year students in session 2006-07.



University of Glasgow Course redesign to support improved assessment practices in first year classes is currently taking place with the support of Dr Steve Draper and Dr David Nicol in two departments (Computing Science and Psychology) for full implementation in October 2006. Project plans will be available from the REAP website in August/September 2006. Dr Draper has worked closely with Professor Jim Boyle at the Department of Mechanical Engineering to support implementation of electronic certainty-based voting in the classroom. Initial plans for software development related to this use of EVS technology were put on hold due to improvements in commercially-available software. Software development will now take place from October 2006 – March 2007 and a developer has been hired to undertake this task. As well as a software developer, the team at Glasgow University will include two project coordinators/evaluators (recruitment currently in hand) to support Dr Draper in delivering assessment re-design in the departments of Psychology and Computing Science and to facilitate evaluation and dissemination activities across the institution. It is anticipated that these staff members may also be deployed at the University of Strathclyde to assist in support of new departments engaged in assessment re-engineering activities. Dr Draper has provided training sessions in EVS technology to staff members at Glasgow Caledonian University and participated in the project’s workshop at the Learner Driver event at Heriot Watt University in May 2006. He is currently working with Dr David Nicol on a paper addressing transformational change issues which will be presented at the ALT-C conference in September.

Section Four: Progress towards Strategic Transformation

REAP is increasingly embedded as a key driver of strategic transformation at the University of Strathclyde and is closely aligned to a number of university initiatives including the First Year Experience, the newly-formed Learning Enhancement Network (a key dissemination outlet for the project), and the HEA National e-Learning Benchmarking exercise. The project is a permanent item on the agenda of the Educational Strategy Group, chaired by the Vice Principal for Teaching and Learning and is a standing item on the agenda of the Teaching and Learning Through Technology Group. REAP principles are embedded in the university’s strategic plan Teaching and Learning through Technology which is about to go to Senate for approval.

The REAP team are currently compiling a range of documents, reports and papers on transformational change at institutional level as well as planning some future activities. Current work includes:

• Embedding of technologies to support assessment: a range of papers examining the practical, logistic and strategic implications of introducing new technologies in higher education institutions are in development. Evidence from a growing number of current and proposed pilots at the University of Strathclyde, Glasgow Caledonian University and the University will inform analysis of the introduction of electronic voting systems, e-portfolio software, electronic feedback software and multiple choice question tests. Issues include the impact of these technologies on support services, teaching teams, timetabling and room allocations and on registry and other central services.

• Comparing models of strategic change: Dr David Nicol and Dr Steve Draper (University of Glasgow) are producing a paper (available from the REAP website at the end of August 2006) comparing traditional approaches to funding change in UK education with the model implemented by Carol Twigg and her team at the National Center for Transformational Change in the USA. The paper, informed by the REAP team’s workshop at the recent Learner Driver event, will identify some of the drivers and barriers to transformation at institutional level and will offer some recommendations. This paper should help to inform policies for the future roll out of e-learning transformation across the Scottish sector.


Section Five: Dissemination

(a) External Dissemination

Conference presentations

• The REAP Project Director presented a paper on REAP at the CAA Conference in Loughborough on 5

th July 2006

• The REAP team hosted a workshop session :Two-way traffic: transforming assessment practices in tertiary education at The Learner Driver, an RSC-sponsored event at Heriot Watt University on 31

st

May 2006.

• Invited presentation on Assessment and the first year experience by the REAP Project Director at a meeting of the joint working groups responsible for the Integrative Assessment and First Year Experience Themes of Scottish Quality Assurance Agency. Edinburgh University on 30

th May 2006.

• The REAP team hosted and participated in the HEA/CRA e-Portfolios conference at the University of Strathclyde on 15

th May 2006.

• The REAP Project Director presented Assessments: adding value rather than effort at conference for academic staff at University of Newcastle, March 10

th 2006.

• The REAP Project Director presented Educative assessment: research and practice at the University of Durham on 19

th April 2006.

• REAP team members presented the project at the CETIS Assessment SIG in March 2006.

• The REAP Project Director presented E-Portfolios and Assessment: models, issues and practices at the JISC CETIS Assessment and Portfolios Special Interest Group Meeting, University of Wolverhampton on 24

th January 2006.

• A paper about the REAP project has been accepted for the Northumbria Assessment Conference in August 2006

• A paper on transformational change has been accepted for presentation at ALT-C in September 2006.

Joint events with SFC e-Learning Programme Projects

• REAP team members hosted a joint workshop session with the ECT project at the SFC e-Learning Transformation Programme workshop in Dunfermline, May 2006

• REAP team members have participated in SFC e-Learning Transformation Programme joint managers meetings.

• Discussions have been held with TESEP project about collaborative activities.

Journal articles

• A paper detailing the REAP project’s approach to evaluation was accepted in June 2006 for publication to the British Journal of Educational Technology (BJET).

• A paper on ‘prospects for transformational change’ has been drafted and will be circulated to the SFC and the National Center for Academic Transformation for comment. (Publication in September 2006)


Section Five: Dissemination (cont…/)

(b) Internal dissemination at Strathclyde

• The REAP project has been presented and discussed at a number of departmental learning and teaching events and faculty committees in all five faculties at the University of Strathclyde.

• In April 2006, the REAP project hosted a half-day showcase sponsored by the Learning Enhancement Network which gave project teams from all three participating universities the opportunity to share outcomes from their initial pilots.

• An article on REAP appeared in the March edition of PRISM, the Strathclyde University publication.

• The REAP team gave one of the keynote presentations at the university-wide Learning Enhancement Network event in March 2006 and jointly presented a session with the Department of Childhood and Primary Studies.

• Monthly ‘Brown Bag’ lunches held at both the University of Strathclyde and Glasgow Caledonian University have given projects the opportunity to share outcomes and ideas across the REAP network and with other staff members.

• Workshops on assessment re-engineering have been piloted through the Effective Learning Using Technology series at CAPLE

Reports to internal management, teaching and learning committees and key initiatives:

• Reports on project progress have been made at a number of university committees including the Educational Strategy Group (a senior management group led by the vice principal responsible for teaching and learning). The REAP Project Director has input into strategy development through this group.

• The project is a standing item on the agenda of the Teaching and Learning Through Technology Group (previously the Virtual Learning Environment Implementation Group), comprising representatives from all faculties and support services. This group is chaired by Professor Jim Boyle (also a member of the REAP Local Management Group).

• REAP has been invited to report to a number of other university committees including: � Teaching Infrastructure Strategy Committee � Steering Committee for the Centre for Academic Practice and Learning

Enhancement � Senate Committee for Lifelong Learning � Education Faculty Committee on Sustainable Assessment � Business School Information Technology Committee

(c) Internal dissemination at Glasgow Caledonian University

• Four pilot projects were presented at the GCU Learning & Teaching Showcase on May 25th 2006

• Caledonian Business Schools’ school board has committed to using the experiences and outcomes from the REAP project to inform and influence the undergraduate programme review in 2006-07

• Staff from both Glasgow Caledonian Business School and the University of Strathclyde attended a REAP strategy planning event hosted by Professor Gilly Salmon in July 2006.


Section Six: Evaluation

• Class/module Evaluations: Baseline data has been collected for participating departments/divisions in REAP (see January report). Each departmental team at Strathclyde have conducted their own evaluation with support from the REAP team of their pilot implementations of re-engineered assessment practices. In particular, these evaluations have examined changes in relation to the drivers for assessment re-engineering identified by the department prior to the changes. For example, some departments were concerned about the lack of formative feedback to students or about the lack of opportunities for skills practice. Others were concerned about workload issues and about the need to rationalise, streamline and make assessment practices more consistent. All were concerned about the need to refine the purposes of assessment to support the development of learner self-regulation. Methods used to carry out evaluations include focus groups, questionnaires, scrutiny of exam performance, input measures of staff time etc. At Glasgow Caledonian University all pilot teams have undertaken an initial evaluation of their projects. Data has been collected by various means (for example, Linda Creanor conducted student focus group discussions in Marketing and Finance modules). Instruments developed by the FAST (Formative Assessment in Science Teaching Project) have been used and modified by module leaders where required in addition to locally-developed tools.

• Guidance and Development of Evaluation Instruments: The REAP team members have produced a comprehensive discussion document on evaluation to guide departments in their approach. This identifies evaluation approaches defined by input, process and output measures. In addition, existing instruments to evaluate assessment practices of relevance to REAP project are being mined in order to develop a new questionnaire that could be used by departments to identify the extent to which their assessment and feedback practices support the development of learner self-regulation.

• Consolidated Evaluation of Pedagogical Effectiveness of Case Studies: As the departmental case studies are collated the REAP team are analysing the outputs against a common framework derived from the assessment research. The intention is to use the framework to carry out an evaluation of all projects against a standardised set of assessment principles (that support first year learning and that help develop learner self-regulation). So far three case studies have been reinterpreted in this way and presentations have been made of these case studies at conferences and events.

• Institutional Evaluation of Transformation: The REAP team are evaluating the transformative impact of the project at institutional level using a modification of the MIT change management framework (Scott-Morton, 1991). This is helping the team capture issues raised from a variety of perspectives including: policy and strategy, organisational structures, management processes, individuals and roles, technology and external environment.

• Cost analysis case studies: Work is underway on costing analysis in relation to course redesigns. The intention is to identify both the cost to change teaching, learning and assessment practices during redesign and also to compare stable costs before and after change. There is also an intention to provide some of this as case studies of full economic costing where hidden costs are revealed. This might prove important for SFC and institutions wishing to align research developments with regard to full economic costing with teaching development costs.

• Case Studies of Technologies. A number of case studies of the implications of the increasing use of technology to support assessment practices are being developed. Currently in production are analyses of electronic voting systems, e-portfolios (PebblePad, SPIDER), online MCQ tests and databanks of feedback comments.

• Implications of new Assessment Practices for Support Services: It is planned to carry out cross- institutional focus groups with different stakeholders (administrators, support staff, academics) in order to identify the strategic implications of increased use of technology-supported assessment practices. This work will take place over the next semester.

• Benchmarking of E-Learning: The REAP Project Director has been leading work on the benchmarking of e-learning at the University of Strathclyde as part of Strathclyde’s participation in the national HE Academy e-Learning Benchmarking programme. As part of this initiative the principles underpinning the REAP project have been embedded in the university’s strategic plan Teaching and Learning through Technology which is about to go to Senate for approval.


Section Six: Evaluation (cont…/)

• As stated above, participating departments have undertaken evaluation activities using a number of approaches and tools that reflect both local departmental and subject needs and the requirements of the REAP project. The REAP project has supported departmental evaluation by producing a comprehensive evaluation framework document with guidelines for implementation (see above). This document includes the enhancement of methods initially developed for the FAST project (Formative Assessment in Science Teaching) and for the Scottish SENLEF (Student Enhanced Learning through Effective Feedback) project. The REAP approach examines the productive use of study time (FAST) alongside the development of learner self-regulation using the seven principles of good feedback practice. This research synthesis has produced a framework that can be used both to design an assessment implementation and as a tool to support useful post hoc evaluation of that implementation.

• A paper detailing this evaluative approach has been accepted for publication to the British Journal of Educational Technology (BJET). The methodology has also been presented to a joint meeting of the QAA Enhancement Themes working groups on Integrative Assessment and the First Year Experience.


Section Seven: Issues and Challenges

Key project challenges between January and July 2006:

• Case study preparation

Case studies examining re-engineering activities in participating departments/classes in each partner institution are a key dissemination outcome for the project. Establishing responsibility for creating the case studies and agreeing a format that can inform a number of outputs for different audiences is particularly important. The project team have provided a significant level of support to departments/class leaders to develop detailed case studies, a process which has been more time-consuming than anticipated. A challenge for the project is to collate enough information to support evaluation of each pilot from multiple perspectives: pedagogical, practical, technological, financial and strategic. Responsibility for contributing to case studies is increasingly recognised as a team effort which includes a number of stakeholders both directly involved with each pilot and representing university-wide interests.

• Pedagogy

A major barrier to assessment re-design is the need to ensure each class/module re-designs is based on sound pedagogical principles. The project team may have underestimated departmental requirements for support in creating re-design plans. The initial formulation of the project plan and subsequent departmental agreements has made it difficult to impose ‘sound redesign’ as a criterion after funding had been agreed. The involvement of large teaching teams, particularly at Glasgow Caledonian University, have meant that course re-design is often undertaken as a process of negotiation over time and can result in more conservative changes than initially planned. As confidence within teaching teams grows, plans have the potential to develop and expand in scope with the support of the project team.

During the second phase of implementation (October 2006 – July 2007) re-design will take place in a number of new departments at the University of Strathclyde in addition to those already involved in the project. We propose to contract new participants to produce a full re-design plan before any funding from the project is allocated. This was the model adopted by NCAT for the Pew Foundation and it has some merits. Two-stage funding will be contingent on each department creating a robust re-design plan and a subsequent case study detailing implementation. The project team will support departments to create both outcomes.

• Evaluation development

Much of the focus of the project to date has been on the provision of high quality feedback opportunities via re-engineered formative assessment practices. Re-designing formative assessment activities is unlikely to have a significant impact on university support services and administration (e.g. Registry or IT Services) which are more likely to be affected by changes to summative assessment. Changes to summative assessment have been a focus of activities at Glasgow Caledonian University

It has proved more difficult than expected to produce data on the effects of assessment practices on support services and administration. This is due to the fact that many changes have been on formative assessment practices which do not influence support services and registry processes to then extent that we anticipated. The implications would be greater for summative assessment (marking) processes. In phase two we will address this by trying to find out what the issues are in relation to changes in summative assessment. It is proposed to try to heighten the debate across the institution by preparing a discussion paper and then holding cross-functional groups of staff from different areas to discuss the implications. This will be written up as a report for senior management.

• Ownership and the ‘ripple effect’

A positive development during this period has been a number of unexpected gains attributable to increasing levels of ownership of project activities at department/faculty level. For example, e-portfolio trials in the Department of Childhood and Primary Studies have led to a number of additional modules adopting this approach and subsequent financial and technical support for e-portfolio software from the Faculty of Education and the university’s IT Services which will ensure embedding of this new assessment practice well beyond the REAP funding period.


Section Eight: Support

• REAP team members hosted a joint workshop session with the ECT project at the SFC e-Learning Transformation Programme workshop in Dunfermline, May 2006

• REAP team members have participated in SFC e-Learning Transformation Programme joint managers meetings.

• The team hosted a visit from the programme evaluators Glenaffric in April 2006 and participated in a subsequent telephone interview with Jane Plenderleith in July 2006.

Interim Reporting Template - REAP · Interim Reporting Template ... Pharmacy, Psychology ... AUT...

Documents

Transcript of Interim Reporting Template - REAP · Interim Reporting Template ... Pharmacy, Psychology ... AUT...