© 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic...

© 2003, School of Medical Education Development, University of Newcastle upon Tyne

Electronic Portfolios Joint SEDA/SRHE Conference 2003

Can electronic portfolios Can electronic portfolios facilitate personal facilitate personal

development planning and development planning and meet the requirements for HE meet the requirements for HE

Progress Files and CPD ?Progress Files and CPD ?

On behalf of the consortium On behalf of the consortium ““Managed Environments for Portfolio-based Reflective Learning” Managed Environments for Portfolio-based Reflective Learning”

(http://www.eportfolios.ac.uk).(http://www.eportfolios.ac.uk).

* University of Newcastle upon Tyne* University of Newcastle upon Tyne^ University of Leeds^ University of Leeds

& University of Sheffield& University of Sheffield+ University of Dundee+ University of Dundee

Simon CotterillSimon Cotterill*, *, Tony McDonaldTony McDonald*, Trudie Roberts*, Trudie Roberts^̂, , Chris RobertsChris Roberts&&, Margery Davis, Margery Davis++, Geoff Hammond*, Geoff Hammond*

NewcastleUniversity



Overview

1.1. BackgroundBackground

2.2. Electronic portfoliosElectronic portfolios

3.3. FDTL4 ProjectFDTL4 Project

4.4. Electronic Portfolio Approaches for HE StaffElectronic Portfolio Approaches for HE Staff

5.5. Questions / Discussion !Questions / Discussion !



Background: PDP

• Increasing demands on HEIs to support personal Increasing demands on HEIs to support personal development planning (PDP)development planning (PDP)

• Recommendation 20 of the National Committee of Recommendation 20 of the National Committee of Inquiry into Higher Education 1997 (Dearing Report)Inquiry into Higher Education 1997 (Dearing Report)

“…“…institutions of higher education, over the medium term, institutions of higher education, over the medium term, develop a Progress File. The File should consist of two develop a Progress File. The File should consist of two elements: elements:

– a a transcripttranscript recording student achievement which should recording student achievement which should follow a common format devised by institutions collectively follow a common format devised by institutions collectively through their representative bodies; through their representative bodies;

– a means by which students can monitor, build and reflect a means by which students can monitor, build and reflect upon their personal development.”upon their personal development.”



• QAA recommendations on:QAA recommendations on:- transcripts - transcripts - progress files- progress files

• PDP is primarily a PDP is primarily a formative processformative process by which by which students can monitor, build and reflect upon students can monitor, build and reflect upon their personal and academic development. their personal and academic development.

• Built on the model of the adult / independent Built on the model of the adult / independent learnerlearner

Background: PDP



Background: Internet Approaches to PDP

Internet Personal and Academic Records Internet Personal and Academic Records (PARs), Newcastle/Nottingham(PARs), Newcastle/Nottingham- Two sequential projects funded by the DFES- Two sequential projects funded by the DFES

LUSID (Liverpool), LUSID (Liverpool),

RAPID (Loughborough), RAPID (Loughborough),

PESCA (Exeter) PESCA (Exeter)

ProFile (Bangor). ProFile (Bangor).

Oxford Brookes / Thames Valley Joint Progress Oxford Brookes / Thames Valley Joint Progress FileFile

SHELL (Plymouth+)SHELL (Plymouth+)



11stst Internet-PARs project: Internet-PARs project:

DfEE Recording Achievement Theme:DfEE Recording Achievement Theme:Internet-PARs Project (1998 - 2000)Internet-PARs Project (1998 - 2000)

Partnership between Universities of Newcastle and Nottingham to:-

… … design, evaluate and disseminate a process, design, evaluate and disseminate a process, facilitated by C&IT, which would enable HEIs to facilitated by C&IT, which would enable HEIs to construct PARs accessible on the Internet … to form construct PARs accessible on the Internet … to form the basis of a continuing record of lifelong reflective the basis of a continuing record of lifelong reflective learning and personal development …learning and personal development …

demo via: http://www.internet-pars.ac.uk



• Academic Academic SupportSupport

• ProgressProgress ReviewReview

• DevelopmentDevelopmentPlanningPlanning

Tutor’s view:Tutor’s view:



Newcastle-Nottingham Internet-PARs Project Newcastle-Nottingham Internet-PARs Project Phase 2: Making New Links (2000 - 2002)Phase 2: Making New Links (2000 - 2002)

• DfEE / DfES funded, HEFCE administeredDfEE / DfES funded, HEFCE administered• … … web-based systems to support continuing web-based systems to support continuing

professional development (CPD) …professional development (CPD) …– in in MedicineMedicine (pre-registration house officers) (pre-registration house officers)– in in EducationEducation (newly-qualified teachers) (newly-qualified teachers)

• … … dissemination of experience of implementing dissemination of experience of implementing both paper-based and web-based PARs at the both paper-based and web-based PARs at the institutional level …institutional level …– a range of types of consultancy available to HEIs to a range of types of consultancy available to HEIs to

facilitate planning and PDP implementationfacilitate planning and PDP implementation

The 2The 2ndnd Internet-PARs project Internet-PARs project



MyPIMDWeb portal

Admin data(HR Database)

Personal data communicationscommunicationsrole based communicationrole based communication

Overview of the web portal in Medicine

employmentemploymentrecords of past, present & future records of past, present & future

postsposts

educationeducationrecord courses, leave, & record courses, leave, &

budgetbudget

developmentdevelopmentplan, evidence, & reflect on plan, evidence, & reflect on

experiencesexperiences



MyPIMD http://mypimd.ncl.ac.uk MyPIMD http://mypimd.ncl.ac.uk

• Access controlled by login Access controlled by login (establishes role and level of (establishes role and level of access)access). .

• Contents customised Contents customised (current post, grade, specialty and (current post, grade, specialty and location)location)..



educationeducation



developmentdevelopment

EncouragingEncouragingrecording &recording &reflecting reflecting practicepractice

FacilitatinFacilitating mentoral g mentoral processesprocesses

Assessing Assessing learning needs learning needs and action and action planningplanning



Using admin (HR) and personal data to provide a Using admin (HR) and personal data to provide a chain of evidence for appraisals, assessments, chain of evidence for appraisals, assessments, revalidation…revalidation…

encouraging and facilitating encouraging and facilitating on-goingon-going documentation and documentation and reflection on professional reflection on professional developmentdevelopment



developmentdevelopment

Use of exemplification to encourage reflectionUse of exemplification to encourage reflection



PORTFOLIOS

• What are portfolios ?What are portfolios ?

• FDTL4 project developing electronic FDTL4 project developing electronic portfoliosportfolios



What are Portfolios ?

UnstructuredUnstructured

FormativeFormative

SampleSample

Best workBest work

Factual / Factual / QuantitativeQuantitative

Student OwnedStudent Owned

StructuredStructured

SummativeSummative

All workAll work

RepresentativeRepresentative

Reflective / Reflective / InterprativeInterprative

HEI OwnedHEI Owned



What is an Electronic Portfolio ?

1.1. A computer application which stores the A computer application which stores the user’s information on a local drive.user’s information on a local drive.

2.2. A static Web page eg. CV on the InternetA static Web page eg. CV on the Internet

3.3. A dynamic database-driven systems that is A dynamic database-driven systems that is accessed over a network.accessed over a network.



Benefits of Electronic Portfolios

• Multi-purposeMulti-purpose• OrganisedOrganised• Multiple structures / viewsMultiple structures / views• SearchableSearchable• TransportableTransportable• SharableSharable• Not left on the bus !Not left on the bus !



Potential Weaknesses of Electronic PortfoliosThere may be barriers to effective use of electronic portfolios: There may be barriers to effective use of electronic portfolios:

It may also be important to use portfolios as It may also be important to use portfolios as part of a wider strategypart of a wider strategy to develop the to develop the reflective analytical skills necessary for PDP.reflective analytical skills necessary for PDP.

• Poor access to computing / Internet Poor access to computing / Internet facilitiesfacilities

• Technophobia ! Technophobia ! • Reliability of IT systemsReliability of IT systems• Possible duplication / overlap with paper Possible duplication / overlap with paper

systemssystems



Potential Weaknesses of Reflective PortfoliosIt is unclear if reflective portfolios favour particular groups: It is unclear if reflective portfolios favour particular groups:

• Learning Style - “natural reflectors”?

• Gender and Age

• Cultural BackgroundJohn Mole. The geography of thinking.John Mole. The geography of thinking.Clin Med 2002; 2:343-5 Clin Med 2002; 2:343-5



Formative and Summative Assessment

Portfolios come in many forms and are often used to Portfolios come in many forms and are often used to evidence the achievement of learning outcomes for evidence the achievement of learning outcomes for summativesummative assessment. assessment.

There may be potential problems if the portfolio There may be potential problems if the portfolio serves both fserves both formativeormative and and summativesummative processes. processes.

For example reflection and PDP is less likely to be For example reflection and PDP is less likely to be open and honest if the student knows that the work open and honest if the student knows that the work will be assessed. More fundamentally the will be assessed. More fundamentally the production of a portfolio can itself be a learning production of a portfolio can itself be a learning process ie. it is as much a journey as an end-point process ie. it is as much a journey as an end-point for assessment.for assessment.



The use of electronic portfolios may The use of electronic portfolios may potentially reduce the tension between potentially reduce the tension between formative and summative processes by formative and summative processes by supporting both assessed and non-assessed supporting both assessed and non-assessed areas.areas.

Students may also have the choice to make Students may also have the choice to make certain material available to assessors, certain material available to assessors, appraisers, tutors and others involved in appraisers, tutors and others involved in their education. their education.

Formative and Summative Assessment



Students control which parts of their portfolio can be Students control which parts of their portfolio can be viewed for specific purposes or by specific peopleviewed for specific purposes or by specific people

Data Ownership ModelData Ownership Model

Corporate(HR) data

PersonalData

Tutor’sData



Managed Environments for Portfolio-based Reflective Learning

Integrated Support for Evidencing Outcomes. The project consortium: The project consortium:

Newcastle University (lead site) Newcastle University (lead site) Leeds University Leeds University Sheffield University Sheffield University Dundee University (for consultancy) Dundee University (for consultancy)

3 years funding through The HEFCE Fund for the 3 years funding through The HEFCE Fund for the Development of Teaching and Learning (FDTL).Development of Teaching and Learning (FDTL).

Developing Web based portfolios to support reflective Developing Web based portfolios to support reflective approaches for evidencing the attainment of approaches for evidencing the attainment of programme programme outcomesoutcomes in undergraduate Medicine. in undergraduate Medicine.

Closely integrated with on-line curricula and study Closely integrated with on-line curricula and study guides to become an integral component of guides to become an integral component of managed learning environmentsmanaged learning environments for Medicine. for Medicine.



What is a managed learning What is a managed learning environment ?environment ?

TLTP3-86: Facilitated Network Learning in TLTP3-86: Facilitated Network Learning in Medicine & Health Sciences (Networked Medicine & Health Sciences (Networked Learning Environments) Learning Environments)

Newcastle University – MedicineNewcastle University – Medicine

Nottingham University – MedicineNottingham University – Medicine

Durham University – Biomedical SciencesDurham University – Biomedical Sciences

Northumbria University – subjects allied to medicineNorthumbria University – subjects allied to medicine

Sheffield University – MedicineSheffield University – Medicine

Study GuidesStudy Guides Learning Resources Learning Resources

Module Outline FormsModule Outline Forms Self-assessmentSelf-assessment



Aims of the FDTL-4 projectAims of the FDTL-4 project• To develop an on-line portfolio system to support a To develop an on-line portfolio system to support a

reflective approach to evidencing the attainment of reflective approach to evidencing the attainment of programme outcomes.programme outcomes.

• To promote the development of the To promote the development of the reflective reflective capabilitiescapabilities of medical students, giving greater of medical students, giving greater responsibility for managing their own learning and responsibility for managing their own learning and preparing for aspects of work-based and lifelong preparing for aspects of work-based and lifelong learning.learning.

• To develop strategies to facilitate To develop strategies to facilitate assessment assessment of of curriculum outcomes that are not amenable to curriculum outcomes that are not amenable to traditional instruments of assessment.traditional instruments of assessment.



Newcastle NLEcurriculum map

- outcomes

quality process

delivery - campusand at distance

student record system

registers

off-line learning

learning resources

other collegesother agenciesbusiness systems

self-assessmenttutor/mentor

support

communication tracking

Managed Learning EnvironmentManaged Learning Environment (after JISC)

Undergraduate portfolioand transcript for reflective

learning- triggered by learning episodes

related to outcomes



communicate

study

experientialexperientiallearninglearning

cyclecycleBenchmarks Benchmarks Outcomes OutcomesDP Specs DP Specs Study Guides Study Guides

Learning ResourcesLearning Resources

learninglearningenvironmentenvironment

experience /experience /practicepractice

review /review /reflectreflect

experiment /experiment /applyapply

think /think /conceptualiseconceptualise



communicate

study

evidence




personal &personal &academicacademic

recordrecord

learninglearningloglog

evidenceevidencelearninglearning



communicate

study

assess

evidence

studentstudenttranscripttranscript


cyclecycle


Benchmarks Benchmarks Outcomes OutcomesDP Specs DP Specs Study Guides Study Guides



recordrecord

assessassessassessassess



communicate

study

reflect

assess

evidence

studentstudenttranscripttranscript



recordrecord




assessassessassessassess



The doctoras a professional

The Right PersonDoing It

How the doctorapproaches his/her

practice

Doing the Thing Right

What the doctoris able to do

Doing the RightThing

LearningLearningOutcomeOutcomess

Who?

How?

What?

When?

Where?



Recording against terminal learning outcomes:Recording against terminal learning outcomes:



Defining intended learning outcomes for options:Defining intended learning outcomes for options:



Life Long Learning / CPD• Electronic portfolios can support life-long Electronic portfolios can support life-long

learning by drawing on personal and learning by drawing on personal and academic information prior to HE, such as academic information prior to HE, such as the UCAS profiles. the UCAS profiles.

• The HE portfolios could later form the The HE portfolios could later form the foundation for vocational or post-graduate foundation for vocational or post-graduate portfolios, especially where portfolios are portfolios, especially where portfolios are structured around learning outcomes which structured around learning outcomes which are geared towards key-skills and/or are geared towards key-skills and/or professional requirements. professional requirements.



Life-long Learning Approach in Medicine:Life-long Learning Approach in Medicine:

portfoliosportfoliosreflection / PDPreflection / PDP-integrated into -integrated into

an outcomes based an outcomes based curriculumcurriculum

continuing need to continuing need to demonstrate CPD:demonstrate CPD:• annual RITA assessmentsannual RITA assessments• annual consultants appraisalannual consultants appraisal• 5 yearly revalidation5 yearly revalidation

Undergrad.Undergrad. Postgrad.Postgrad. 5yrs5yrs 10yrs10yrsPre-Pre-HEHE

The use of electronic portfolios aims to foster the The use of electronic portfolios aims to foster the skills of independent learning, reflection and PDPskills of independent learning, reflection and PDP



Portfolios for HE Staff

Baume D, Yorke M. The reliability of Baume D, Yorke M. The reliability of assessment by portfolio on a course to assessment by portfolio on a course to develop and acredit teachers in higher develop and acredit teachers in higher education. Studies in higher education. education. Studies in higher education. 2002; 27: 7-25 2002; 27: 7-25

HE Staff may also benefit from electronic HE Staff may also benefit from electronic portfolio approaches including personal portfolio approaches including personal profiles and research plans which tie in to profiles and research plans which tie in to management information systems and management information systems and networked learning environments.networked learning environments.



The Personal Plan at Newcastle• Teaching and Learning (UG and PG teaching and Teaching and Learning (UG and PG teaching and

teaching development)teaching development)

• Research (Research programmes, awards, staff and Research (Research programmes, awards, staff and students)students)

• Third Strand (spinouts, outreach work and Third Strand (spinouts, outreach work and industrial collaborations)industrial collaborations)

• Roles (administrative duties, training and support Roles (administrative duties, training and support roles)roles)

Integrates centrally held data sets: Integrates centrally held data sets: • RIS awards data under "Research Grants held" RIS awards data under "Research Grants held" • Graduate School records of PG student supervision Graduate School records of PG student supervision

under "Research Students". under "Research Students".



Evaluation and Research Issues

•Portfolio assessment – validity and reliability issuesPortfolio assessment – validity and reliability issues

Enhancing reliability in portfolio assessment: discussions between assessors. Enhancing reliability in portfolio assessment: discussions between assessors. Pitts J, Coles C, Thomas P, Smith F. Pitts J, Coles C, Thomas P, Smith F. Med Teach 2002; 24(2):197-201 Med Teach 2002; 24(2):197-201

The use of portfolios for assessment of the competence and performance of doctors in The use of portfolios for assessment of the competence and performance of doctors in practice. practice.

Wilkinson TJ, Challis M, Hobma SO, Newble DI, Parboosingh JT, Sibbald RG, Wakeford R. Wilkinson TJ, Challis M, Hobma SO, Newble DI, Parboosingh JT, Sibbald RG, Wakeford R. Med Educ 2002: 36(10):918-24 Med Educ 2002: 36(10):918-24

Portfolio-based assessments in medical education: are they valid and reliable for summative Portfolio-based assessments in medical education: are they valid and reliable for summative purposes? purposes?

Roberts C, Newble DI, O'Rourke AJ. Roberts C, Newble DI, O'Rourke AJ. Med Educ. 2002: 36(10):899-900 [Commentary] Med Educ. 2002: 36(10):899-900 [Commentary]

AMEE Medical Education Guide No. 24: Portfolios as a method of student assessment AMEE Medical Education Guide No. 24: Portfolios as a method of student assessment Friedman M, Davis MH, Harden RM, Howie PW, Ker J, Pippard MJ Friedman M, Davis MH, Harden RM, Howie PW, Ker J, Pippard MJ Medical Teacher. 2001; 23:535-51 Medical Teacher. 2001; 23:535-51

Portfolio assessment in medical students' final examinations. Portfolio assessment in medical students' final examinations. Davis MH, Friedman M, Harden RM, Howie P, Ker J, McGhee C, Pippard MJ, Snadden D. Davis MH, Friedman M, Harden RM, Howie P, Ker J, McGhee C, Pippard MJ, Snadden D. Med Teach 2001; 23(4):357-366 Med Teach 2001; 23(4):357-366

An evaluation of a standards based portfolio. [Nursing / Outcomes Based] An evaluation of a standards based portfolio. [Nursing / Outcomes Based] Gallagher P Gallagher P Nurse Educ Today 2001;21(5):409-16 Nurse Educ Today 2001;21(5):409-16



• Reflective ability / practiceReflective ability / practice

- agreeing a context sensitive definition of “reflection”agreeing a context sensitive definition of “reflection”

- qualitative analysis of anonymised portfolio dataqualitative analysis of anonymised portfolio data based on this definition of reflectionbased on this definition of reflection

- quantitative analysis of anonymised portfolio dataquantitative analysis of anonymised portfolio data

- what factors influence reflective ability / practice ?- what factors influence reflective ability / practice ?- gender, age-group, ethnic groups etc.- gender, age-group, ethnic groups etc.

- any relationship with outcomes (grades) ?- any relationship with outcomes (grades) ?




•Learning styles / learning approachesLearning styles / learning approaches

- Honey & Momford used in study skills course- Honey & Momford used in study skills course

- ASSIST questionnaire- ASSIST questionnaire

•User evaluationUser evaluation

- questionnaires + focus groups- questionnaires + focus groups

•Student attitudesStudent attitudes

- questionnaires + focus groups (longitudinal)questionnaires + focus groups (longitudinal)

- effect of implementation techniques ?effect of implementation techniques ?




•Impact of portfolios on the Education Impact of portfolios on the Education EnvironmentEnvironment

Roff S. Development and validation of the Roff S. Development and validation of the Dundee Ready Education Environment Dundee Ready Education Environment Measure (DREEM). Medical Teacher, 1997; Measure (DREEM). Medical Teacher, 1997; 19: 295-299 19: 295-299

-Does the introduction of electronic -Does the introduction of electronic portfolios change the educational portfolios change the educational environment ?environment ?

•Other issuesOther issues

-Can computer mediated environments Can computer mediated environments promote open reflection, sharing and sence promote open reflection, sharing and sence of ownership ?of ownership ?

-Does the use of exemplification benefit or -Does the use of exemplification benefit or hinder reflection ?hinder reflection ?




Conclusions

• Electronic portfolios have the potential to Electronic portfolios have the potential to facilitate PDP and meet the requirements facilitate PDP and meet the requirements for HE Progress Files. for HE Progress Files. – Offer flexibility for both formative and Offer flexibility for both formative and

summative purposes.summative purposes.

• Electronic portfolios in HE may provide the Electronic portfolios in HE may provide the foundation for portfolios in CPD.foundation for portfolios in CPD.– Transfer of data Transfer of data 17/18 17/18 HE HE

CPDCPD– Common outcome themesCommon outcome themes– Particularly valid where the curriculum is driven Particularly valid where the curriculum is driven

by professional requirementsby professional requirements



Conclusions (2)

• There remain many unanswered questions There remain many unanswered questions about the educational impact of electronic about the educational impact of electronic portfolios.portfolios.– Importance of evaluation and researchImportance of evaluation and research

• Electronic portfolio approaches may also be Electronic portfolio approaches may also be applied to staff in HE for a range of applied to staff in HE for a range of purposes.purposes.



further informationfurther information

http://www.eportfolios.ac.uk

http://www.internet-pars.ac.uk

http://nle.ncl.ac.uk

http://www.ltsn.ac.uk/genericcentre

http://www.recordingachievement.org

© 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic...

Documents

Transcript of © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic...