Post on 08-Jun-2015
Thinking the unthinkable:
USING PROJECT RISK MANAGEMENT WHEN INTRODUCING COMPUTER-
ASSISTED ASSESSMENTS
Ian Harwood & Bill Warburton
University of Southampton, UK.
Thinking the unthinkable…
• Introduction
• Overview of PRM and risk efficiency
• Methodology / Case background
• Identified risks
• Conclusions
Introduction
• Implementing CAA is a risky activity
• Limited coverage in CAA literature
• Aims of this paper:– Build links between CAA and PRM literature– Show how formal PRM can be beneficial
Viewing CAA introduction as a project• Project has been defined as:
'a temporary endeavour undertaken to create a unique product or service' PMI (2000)
• A project is:‘a step into the unknown, fraught with risk
and uncertainty’ Lock (2000)
Increasing interest in Risk
• Now in the 'risk society' (Beck, 1992).
• Broad base of risk literature covering:– environmental issues (GM foods, nuclear waste)
– data processing
– security management (terrorism, insurance)
– project management
– health and safety (design criteria, disaster planning)
– finance and economics (derivatives, risk/return)
What is ‘Risk’?
• ‘a choice rather than a fate’ Bernstein (1996)
• risk or uncertainty?– risk is present when future events occur
with measurable probability– uncertainty is present when the likelihood
of future events is indefinite or incalculable Knight (1921) ‘Risk, Uncertainty and Profit’
Thinking the unthinkable…
• Introduction
• Overview of PRM and risk efficiency
• Methodology / Case background
• Identified risks
• Conclusions
PRM interest groups
• Association for Project Management: Risk Management Specific Interest Group (http://www.eurolog.co.uk/apmrisksig/)
• Centre for Risk Research, School of Management, University of Southampton, UK (http://www.management.soton.ac.uk/risk/default.asp)
• The Project Management Institute: Risk Management Specific Interest Group (http://www.risksig.com/).
The Project Risk Analysis and Management process (Simon et al 1997)
Identify risks
Assess risks
Plan riskresponses
Write reports
Focus PRAMDefine project
Risk Register&
Risk Model
DocumentedRisk Management
Procedure
… part of broaderproject plans
RiskAssessment
Reports
Reduce risks bychange to project
Achievementof projectobjectives
Risk efficiency
• From theory of portfolio management in Economics (Markowitz, 1959)
• ‘Risk’ and ‘Return’ inextricably linked
• Now used in the project environment– CCTA ‘Project Risk Management’
document
Theoretical risk efficiency framework
Risk
Ret
urn
A
Brisk efficient frontier
Key:
B
A C1…6: :
:
Minimum risk
Maximum return
Cases 1 to 6
feasibleCAA
applications
x : Desired risk efficiency
x RI2
RI2 : Case 2 risk inefficiency(as an example)
C1
C4C3
C2
C5
C6
Risk efficiency migration due to PRM
Risk
Ret
urn
A
Brisk efficient frontier
Key:
B
A
C1…6
:
::
Minimum risk
Maximum return Cases 1 to 6
feasibleCAA
applications
x : Desired risk efficiency
x
C1
C4
C3
C2
C5
C6
Thinking the unthinkable…
• Introduction
• Overview of PRM and risk efficiency
• Methodology / Case background
• Identified risks
• Conclusions
Overview of cases
• Background to Soton MLE strategy
• Roll out of PerceptionCase Date
(m/y) Discipline Formative /
summative usage
Number / level of students
Contribution to overall
assessment 1 05/03 Science Summative 80 2nd year UG,
16 3rd year UG 20%
2 10/03 Humanities Diagnostic 60 1st year UG N/A 3 11/03 Healthcare Form. & Sum. c.300 PGR
100 PGR Completion is a
course requirement 4 11/03 Soc. Sci. Form. & Sum. 280 1st year UG 30% 5 12/03 Science Summative 30 2nd year UG 1% 6 01/04 Science Summative 60 1st year UG 20%
Methodology
• Case study approach (Yin, 2003)
• Semi-structured interviews
• Cross-case analysis (Miles and Huberman, 1994)
• (retrospective) risk register generated
Thinking the unthinkable…
• Introduction
• Overview of PRM and risk efficiency
• Methodology / Case background
• Identified risks
• Conclusions
Identified risks- before & during delivery Table 2•Wrong assessment published (C3,5)
•Authors ‘reinvent the wheel’ (C5)
•Publication date missed (C1)
•Communication gaps (C1,3)
•Environmental issues (C4)
•Cheating (C1,2,3,5,6)
•Failure at point of delivery- scalability issues (C3,4)
•Tests submitted prematurely (C2)
Identified risks- during & after delivery Table 2•Informal appeals- protest about fairness of CAA (C1,5)
•Formal appeals:-workstation issues (C1,2,6) -unfamiliar test environment (C1)
-test construction (C1)-negative marking (C6)-QA issues (C5)
CAA abandoned:-changes in course architecture (C3) -staff turnover (C6)-weak implementation (C5)-perceived unfriendliness (C5)
‘Proactive’ responses: mitigating actions 1 Table 2•Pre-emptive Load-test at realistic levels
•Allow sufficient time to fix the problem
•Run a formative version of the test
•Keep channels of communication open
•Help staff to comply with internal CAA procedures
•Ease of use- capture & address author feedback
•Keep channels open between authors
•Comply with internal CAA procedures
•Use prefabricated templates
‘Proactive’ responses: mitigating actions 2 Table 2•Make training a prerequisite for access to the service.
•Help staff to comply with internal CAA procedures
•Use random selections from large question pools
•Allocate sufficient time to test workstations before exam
•Use a class register to book sufficient workstations
•Practice with formative versions
•Maintain tutorials & practice tests on public workstations
•Help tutors choose appropriate assessment methods
‘Reactive’ responses: Contingency Plans Table 2•Have a paper copy
•Substitute marks from formative assessments
•Capture and address author feedback
•Discount compromised items
•Re-issue test
•Discount compromised items
•Use trained invigilators
•Have a spare workstation area available
•Monitor and address student feedback
Thinking the unthinkable…
• Introduction
• Overview of PRM and risk efficiency
• Methodology / Case background
• Identified risks
• Conclusions
Benefits of formal PRM
• Enables better informed (and more believable) plans, schedules and budgets
• Increases the likelihood of a project going to plan
• Allows more meaningful assessment of contingencies
• Contributes to the build-up of statistical information to assist in better management of future projects
• Enables a more objective comparison of alternatives
• Identifies, and allocates responsibility to, the best risk owner
Benefits of formal PRM cont…• Improves general communication
• Leads to a common understanding and improved team spirit
• Assists in the distinction between good luck / good management and bad luck / bad management
• Helps develop the ability of staff to assess risks
• Focuses attention on the real and most important issues
• Facilitates greater risk-taking, thus increasing the benefits gained
• Demonstrates a responsible approach to customers
Risk Management Maturity (Hillson, 1997)
• Four levels:1) naïve2) novice3) normalised4) natural
• Assists in identifying areas for improvement in risk management
Decay of risk exposure through iterative PRM
Preparation risks
Aut
hori
ng r
isks
Delive
ry ri
sks
C1C4
C3
C2
C5
C6
C1
C4
C3
C2
C5C6
Realised risk exposurein this case study
Potential risk exposureusing project risk mgt.
Preparation risksA
utho
ring
ris
ks
Delive
ry ri
sks
Closing thoughts…
Six 'first time' applications of Perception replace paper-based exams- riskier than future repeat applications– iterative approaches to formalised project risk
management can build institutional corpi of CAA risk knowledge
– trend of total risk exposure progresses from risk inefficient to more risk efficient
– a full-scale system could be released that already catered for the greatest risks
– institution becomes more risk mature (Hillson, 1997)
Any questions…?