FINAL REPORT

Healthcare Improvement and Rapid PDSA Cycles of Cha nge: A

Realist Synthesis of the Literature

Esther Curnock, MPH MRCGP Julie Ferguson, PhD John McKay, MD FRCGP Paul Bowie, PhD FRCP (Edin) NHS Education for Scotland Department of Postgraduate General Practice Education 2 Central Quay GLASGOW G3 8BW

Further information: paul.bowie@nes.scot.nhs.uk The study was commissioned and funded by the Patient Safety Multi-Professional Steering Group of NHS Education for Scotland.

Abstract Background

Plan-Do-Study-Act (PDSA) is a quality improvement (QI) tool used in healthcare. It

pursues effective changes in healthcare processes to enhance health outcomes, using

rapid small-step change cycles. There is limited research into the effectiveness of QI

programme components. This systematic ‘realist synthesis’ aimed to identify and

summarise the implementation and contextual factors that may influence the

effectiveness of PDSA.

Methods

We established outline theories relating to whether, how and why PDSA works by

reviewing recent papers. We systematically searched for articles in Medline, Embase,

British Nursing Index, PsycINFO, HMIC, MIDIRS and EBM (2005-2009). Articles

referencing or assumed to use PDSA (e.g. having employed the Institute of Healthcare

Improvement (IHI) collaborative methodology) were included. Any outcome measure

reported as leading to QI was included, whether or not it was clinical.

A data extraction form, informed by PRISMA guidelines was employed. Using agreed

inclusion and exclusion criteria we selected papers which would add to, or refine, the

theories. Articles were added through snowballing from citation lists. The

data/findings were extracted and synthesised according to the contextual and

implementation issues which facilitated or limited successful PDSA outcomes.

Results

94 articles were reviewed as relevant to our research question with a total of 44

contributing to our evaluative framework and full theory development. PDSA has led

to changes in practice and health outcomes in a variety of settings, geographies and

patient groups. Several factors were identified as influencing effectiveness. These

included: organisational size, context and stability; setting clear and measurable

improvement goals underpinned by evidence; staff engagement and ownership; speed

and size of cycles; actions taken to spread and sustain changes; and resources.

Conclusions

PDSA may be more successful in smaller, defined settings with stable staff and patient

groups. Culture seems more important than size. Success is more likely where:

evidence linking process goals to intended outcomes exists; is understood and visible to

staff; the workforce is actively engaged; and, staff have prior QI experience. Although

difficult to achieve, quicker and shorter PDSA cycles may lead to faster/more sustained

practice change. Improvements are spread and sustained more when progress and

associated data are shared and visible. Existing support structures and additional

resources lead to greater success.

Registration:

Not applicable

Keywords:

PDSA, Quality Improvement, Patient Safety, Realist Synthesis, Evidence Review,

Healthcare

Background

PDSA (Plan-Do-Study-Act) is a tool that is utilised in a diverse range of quality

improvement (QI) programmes and projects in healthcare systems worldwide. The aim

of PDSA is to pursue, sustain and rollout effective changes in care processes that

favourably affect outcomes, using rapid small-step change cycles. The model involves

four cyclical stages: hypothesis formation (Plan); implement the new process with data

collection (Do); interpreting the results (Study); and a decision as to what to do next

(Act) [1]

PDSA (also known as Plan-Do-Check-Act, PDCA) was developed as a method for

achieving efficiency in Japanese car manufacturing [2] and was influenced by earlier

work on industrial statistical quality control [3]. It was one of the key tools developed

as part of the ‘Total Quality Management’ movement [4]. Langley et al [5] used PDSA

as the central component of their ‘Model for Improvement’, and were arguably the first

to propose the use of PDSA cycles in health care QI.

The Model for Improvement is popularised by the Institute of Healthcare Improvement

(IHI) through their ‘Breakthrough Series’ collaborative approach [6]. IHI methods are

strongly influenced by ‘industrial’ continuous QI methods that originated in

manufacturing industries and organisational change theories that are perceived to be

complementary [7]. In the United Kingdom (UK) in recent years several large-scale QI

programmes such as the Scottish Patient Safety Programme [8] and the Health

Foundation’s Pilot ‘Safer Patients Initiative’ [9] have been implemented in partnership

with the IHI. However, empirical evidence of the effectiveness of QI programmes, as

well as evaluation about their impact on healthcare quality is limited [10]. Few

controlled studies of QI collaboratives have been undertaken, and the results are

equivocal. For example, Landon et al [11] evaluated the effectiveness of a QI

collaborative to improve HIV patient care using a prospective controlled study design,

involving 9986 patients in 69 clinics. No significant differences in quality of care

measures were found between control and intervention clinics).

A systematic review of QI collaboratives from 1995 to 2006 had limited scope. In it a

small number of controlled trials showed limited positive effect and two showed no

significant benefit [12]. Their objective was specifically to assess for empirical

evidence of effectiveness, but they were unable to draw any conclusions from the vast

majority of reports due to uncontrolled study design. A systematic review of the

effectiveness of teaching QI methods to clinicians also had mixed results [13].

A common criticism of QI initiatives has been their tendency to favour action over

evidence, with the rapid dissemination of innovative but unproven initiatives, with

potential unintended consequences of harm and opportunity cost [14-16].

Although evidence of the effectiveness of many QI approaches is lacking [17] several

factors that appear to be critical to the success or failure of QI collaborative projects

have been identified. A review [13] found that using incremental tests of change (i.e.

the PDSA approach), and having access to performance data appeared to be important

factors, but were themselves insufficient for ensuring beneficial outcomes. The review

was unable to analyse the determining factors any further or make specific

recommendations regarding curricula.

Using randomised control trial (RCT) designs to evaluate interventions where the

context, content, and implementation have a high degree of heterogeneity can be

problematic. Such methods when applied to less stable and complex interventions, and

when unaccompanied by a detailed process evaluation, may fail to identify that

interventions work in some instances rather than others, or to explain why they work

(i.e. the potential casual mechanisms). This reduces the generalisability of both

research and subsequent review conclusions.

In addition, strict eligibility criteria mean that much useful contextual information has

been lost in the existing traditional systematic review literature, including information

required by healthcare managers and policy makers to help them decide whether to

implement a specific QI approach in their local setting [18].

The need for an alternative approach to evaluation and the systematic review of QI

programmes to elicit this contextual and generalisable learning is recognised by many

authors [1, 16, 18-22]. It has been argued that because programmes are embedded in a

range of attitudinal, individual, institutional, and societal processes they are not things

that can be claimed to ‘work’ or ‘not work’: instead they contain certain ideas that work

for certain subjects in certain situations [23]. Theory driven approaches to both

evaluation and evidence review that try to understand the links between context,

content, implementation and outcomes are being developed and tested to see if they can

reduce such problems [15, 18].

At the point of embarking on this review in 2010 there was to our knowledge no

systematic theory informed review of the evidence available which identifies the key

mechanisms within many QI programme initiatives - PDSA - that are likely to lead to

practice and health outcome change and improvement.

Review purpose

Given the above limitations in current knowledge and by using elements of a realist

synthesis approach, as espoused by Pawson et al [24], this review attempts to address

the following question:

“What are the implementation and contextual factors that may enhance or

reduce the effectiveness of PDSA as a QI tool in healthcare settings?”

The review was commissioned and funded by NHS Education for Scotland, a special

health board with responsibility for the education, training and life-long learning of the

healthcare workforce in Scotland. Training in QI concepts and methods - including

PDSA – is a learning need that can be identified across the health care workforce. If

such interventions are to be rolled out it is vital to understand how best to implement

them and in what circumstances and contexts they could work. We present the

discussion and learning in relation to the key theories uncovered via the review

processes. This learning then informs our recommendations regarding the training of

health care professionals in PDSA methods, as well as our guidance on the effective

implementation of the technique as part of ongoing improvement initiative.

Methods

Key theories to be explored

An initial background search was conducted examining 37 papers on PDSA published

over the previous year (2009) from study commencement, and browsing references

identified from these papers. This allowed mapping of the diverse perspectives and

theories about PDSA. Inclusion criteria for the review were refined in light of emerging

learning from this process. Informal networking with known experts in the field also

informed this process. The initial resultant theories derived from this initial process are

shown in Table 2 and more refined versions in the findings section. These informed our

data search and extraction processes.

Determining the search strategy

We subsequently conducted a systematic search of papers using the terms in Figure 1

See Figure 1: Formal Search Criteria (additional file 1)

Only English language papers were included from over a five year period from 2005 to

the time of the search in September 2009. These time limits were selected due to

resource limitations and pragmatism. Either the setting or the participants had to be

directly involved in healthcare. Any outcome measure in the papers that led to QI was

included, regardless of whether or not it was clinical.

Study Quality Assessment and data extraction

An initial process was carried out to screen abstracts in order to exclude papers of low

relevance or low worth. Two questions were applied:

1. Do the authors refer to either a PDSA or PDCA process, or IHI collaborative

methodology for use in QI in healthcare?

2. Does the paper go beyond superficial description or commentary, i.e. is it a broadly

competent attempt at research, enquiry, investigation or study?

Papers were rejected at this stage if the answer to either of these questions was ‘no’ and

the rationale for this was documented. Papers for which the abstract did not offer

enough information to determine eligibility were retrieved for full text review. The

same initial questions were asked of all full texts obtained for review, and papers were

subsequently rejected if either answer was ‘no’. Papers that met the criteria of

relevance and worth underwent full data extraction.

A data extraction form shown in Figure 2 was developed.

See Figure 2: Data Extract Form (additional file 2)

This data extraction form was informed by PRISMA (Preferred Reporting Items for

Systematic reviews and Meta-Analyses) guidelines [25], and the data extraction

framework used in a systematic review with a similar approach [26]. The SQUIRE

(Standards for QI Reporting Excellence) guidelines [27] were used to establish a guide

to quality criteria however no formal quality scoring system was employed, in keeping

with similar systematic realist reviews [28]. Excluding a large number of relevant

studies on grounds of rigour would reduce the validity and generalisability of the

learning from realist synthesis review findings [29]. The form went through an iterative

cycle of improvement in a piloting phase involving the three reviewers until a final

format was agreed upon.

The data extraction variables are shown in Table 1

Table 1: Data Extraction variables

Data Extract Variables • Year of Publication • Paradigm (theoretical lens used) • Type of Paper • Perspective (unit of analysis) • Aim & Objectives • Design

o Study design o QI methods used o Method of evaluating outcomes

• Funding • Key actors

o who supporting – how, why o who delivering – roles, training o who receiving o who comparing to

• Nature of the Improvement • Context factors

o Country / region o Urban / rural o Primary / secondary healthcare o Other local / structural factors of influence

• Implementation process o Costs of implementation o Plans for ongoing monitoring

• Data Collection o Timeline

• Data Analysis o Handling of disconfirming observations

• Results o Summary of main results o Facilitators / strengths / successes of implementation o Barriers / weaknesses / difficulties of implementation o Strength of relationship between intervention & changes

• Dissemination Process • Conclusions • Reflexivity (consideration of potential biases & conflicts of interest)

• Ethical implications

The principal summary measures were factors influencing and critical to the

implementation of the PDSA process (see Table 2) and the associated theories in the

findings section below.

A final summary page asked the reviewer the following questions:

1. Does the paper have an important message for our review question?

2. Does the paper fulfil the established quality criteria in its domain?

3. What factors does the paper identify as being critical to the use of PDSA or to the

delivery of education on PDSA?

Papers were randomly allocated to three reviewers (EC, JM, and JF). Twenty percent of

papers were extracted independently by two reviewers (pairing between reviewers

varied to ensure consistency across the group). Reviewers met in pairs after each had

completed extraction of a cycle of 10 papers. Any disagreements on the data in the

papers extracted in duplicate were then resolved by consensus. The meeting also

provided the opportunity for discussion of uncertainties arising with any of the other

papers completed by each reviewer. The approach taken in a realist review makes such

dialogue between researchers essential for the iterative process of classification.

For each article, efforts were made to uncover any sources of bias from either the

authors or the main support team of the initiative. Each article was examined for

evidence of who or what had triggered the decision to initiate the QI project; why it had

occurred at that time; funding sources; and whether the authors had commented on

potential conflicts of interest. A judgement was made on the transparency of data

presentation and the degree to which the reader could analyse results objectively.

Finally we used a snowballing technique to scan the references of the most relevant

articles identified in the systematic search. This technique was used to identify key

papers prior to 2005, and any further papers not identified by the formal search process

that could contribute to populate the synthesis framework.

Following this a final search for additional studies from the grey literature was

conducted to strengthen specific aspects of the evaluative framework (Table 2) and

theories. Additional studies considered included sources such as Government Reports

and national and international PDSA/Collaborative group papers (e.g. National Primary

Care Collaborative [English], Australian Primary Care Collaborative [part of

Improvement Foundation], and Health Foundation Reports).

Evidence/data synthesis

To facilitate the synthesis process an evaluative framework that informed the

development of our initial six theories was developed and populated using the

evidence/learning. This framework identified crucial dimensions of the theories that

related to mechanisms (core elements of the PDSA cycle, or how it was applied) and

contexts (staff motivation, stability, etc). We were seeking data and learning on these

areas to allow us to test and further refine our initially identified theories. The

framework is illustrated in Table 2.

Data and learning were therefore synthesised in order to determine what implementation

and contextual factors (circumstances and constraints) that may enhance or reduce the

effectiveness of PDSA as a QI tool in healthcare settings. ‘Contradictory’ evidence was

used to generate insights about the influence of the PDSA mechanisms or contexts.

To ensure transparency within the subsequent discussion and findings sections we have

referenced the authors which support our assertions. In addition we have used codes

which can be used to identify the actual key finding/learning from within each paper

that has been used to refine our theories and inform our conclusions. The T codes

identify which of our contexts/mechanisms the text relates to (e.g. T1 – T9 shown in

Table 2 and the C code relates to the precise contextual issues (or constraining factors)

identified by the author that has influenced the success of the PDSA intervention

studied/reviewed. The author, paper and constraint or contextual factor relating to each

code can be seen by referring to Figure 5.

Conclusions were drawn by synthesising the data in relation to the evaluative

framework (Table 2) and are presented as a narrative account. Draft recommendations

and conclusions were tested with key stakeholders, and the review team worked

alongside practitioners and policy-makers to apply recommendations in particular

contexts.

Results

Review statistics/ initial theories and synthesis framework.

Recent relevant papers (n=37) from the year preceding the review start date were

identified and reviewed and from this process several draft implementation and

contextual mechanisms were identified which might influence the effectiveness of

PDSA. These are shown in Table 2:

Table 2: Data synthesis framework based on initial draft identified mechanisms and contexts

MECHANISM enhancing outcomes –practice changed/sustained

T1. T2. T3 T4

Measurable process goals that are linked to outcomes via evidence

Changes are more likely to be sustained by ensuring the impact of change (data/progress) is visible

Changes broken down into small-step cycles completed in rapid succession lead to faster sustained change

Local spread to surrounding settings/context can be achieved via key actions

CONTEXT (Constraining and facilitating Factors)

T5 T6 T7 T8 T9

Frontline staff may have relevant knowledge of changes & implementation skills that can influence success

Engagement of frontline staff can influence participation and impact

Existing resources can sustain PDSA/ changes

The tool is can achieve practice change/outcomes in multiple disciplines & organisational contexts

The context/stability/ size of the organisation can impact on PDSA cycles/impact

These mechanism and contexts informed the data extraction process. They were

subsequently integrated and reordered into six key theories about what influences the

effectiveness of PDSA. We have kept the initial codes above however to allow

reference back to the data extraction process. The more collapsed /integrated theories

are:

• Theory One: PDSA is can achieve practice change/outcomes across different

health care disciplines, organisational contexts and settings (see T8/9 in Table

2).

• Theory Two: PDSA interventions are more likely to lead to practice change

when the intended changes are measurable in the short term and when

practitioners can see the changes are evidence based (see T1 & T2 in Table 2)

• Theory Three: PDSA leads to more substantial and sustained practice change

when frontline staff are fully engaged with the process (see T5 & T6 in Table 2)

• Theory Four: Rapid PDSA cycles lead to faster improvement, longer-term

change and sustained momentum for change (see T3 above Table 2 in Table).

• Theory Five: Achieved changes can spread into other settings and contexts (see

T4 above in Table 2).

• Theory Six: Improved practice in PDSA can be achieved and sustained through

existing resources (see T7 in Table 2)

Figure 3 shows the results from the data base search (from 2005 -2009) and the reasons

for exclusion of papers.

See Figure 3 Data base search results and exclusions (additional file 3)

The search identified a total of 762 papers of which 94 were eligible after the

application of our review criteria. These 94 papers were identified as relevant to our

research question and one or more of our subsidiary theories and contained some

worthwhile information. Of these, a significant proportion described ‘before and after’

studies, using baseline data as the comparison group. The quality of these papers varied

significantly, but nevertheless many were able to contribute useful information to refute

or verify our postulated theories and add new knowledge to existing evidence [7]. Of

these 94 papers, 32 were identified as holding relevant information to populate the

evaluation synthesis framework that informed our initial theories (Table 2). The

remaining 62 papers provided information on the general effectiveness of PDSA (e.g.

Theory one only) and detailed information on the settings it was applied within, but did

not provide sufficient detail on any individual PDSA components (e.g. cycle speed) or

contexts (e.g. previous QI culture) to add substantially to our synthesis framework.

The snowballing technique used to identify key papers prior to 2005 involved scanning

the references of the most relevant 32 articles identified in the systematic search that

populated the synthesis framework. This technique identified a further 12 papers.

Therefore, a total of 44 (32 plus 12) papers were drawn on to populate the evaluation

synthesis framework and refine our theories. The discussion and findings in this paper

are based mainly on these 44 papers.

Review Limitations

This review has focussed on how PDSA can be used to best effect in healthcare settings.

However PDSA is often used in conjunction with other components of QI. In particular

it has been used most widely within the ‘collaborative’ approach, which can involve

multiple sites sharing knowledge, learning days, and external support from ‘experts’. It

is therefore difficult to disentangle the dimensions which influence the effectiveness of

PDSA from the dimensions that effect the wider approach utilised.

We have been unable to produce definite statements about ‘what works’; instead, we

have added further detail to our initial theories and produced advice on the likely

implications of different contexts on the success of PDSA. This outcome is a natural

consequence of the type of question being asked, rather than a failing of the scope of the

review or its methodology. The contextual details within individual studies were

limited in many cases. It is unclear whether we have omitted other important contextual

factors that either aren’t identified by study authors, or are considered not influential

enough to report. However by collecting the data available on a large number of papers

we hope we have been able to identify most of the significant elements that influence

the effectiveness of PDSA.

Although we were systematic in our data extraction processes and the application of our

criteria, it is feasible that we missed some relevant articles. This will be particularly

true for articles published prior to 1995, as prior to this date we used a snowballing

process for papers cited in our most relevant articles identified in the systematic search.

Publication bias is a significant concern and one that is difficult to quantify. Successful

organisations have a vested interest in promoting their successes [14]. Once a QI

initiative has been undertaken there is little incentive to devote resources to

investigating its efforts, particularly where private resources have been invested.

Financial interests may be an influence in the popularity of specific QI methods [18].

Mittman (2004) suggests objective evaluations of QI methods are lacking due to

‘demand-led bias’ and points out that much of the published information is located in

management and practitioner-orientated journals such as the ‘The Joint Commission

Journal on Quality in Healthcare’, which tend to emphasise only successful accounts

[16]. Our findings should be considered in light of the likelihood of significant

publication bias. However due to the nature of our question, we would argue that it is

less of a concern than if we had attempted to answer the question ‘does PDSA work?’

Some common criticisms of realist synthesis are that: the iterative approach can

introduce bias; there can be a tendency to treat all forms of evidence as equally

authoritative; and this can lack transparency in the choice of evidence used. Several

steps were taken to minimize these weaknesses. A significant period of time was spent

refining the research question at the scoping stage, giving stability to the question being

asked at the point data was extracted, and therefore reducing risk of bias introduced.

SQUIRE guidelines were used to determine the authoritativeness of different sources of

evidence. In the first phase of data extraction the reviewers examined all eligible

studies within a specified time frame, rather than take a purposive sample, in order to

ensure the transparency of study selection. In taking these steps the realist approach

was modified to suit our specific needs and resources. This is in part a reflection of the

complexity of synthesising the evidence on organisational interventions. If complex

interventions are by nature ‘leaky’ and susceptible to local adaptation, it is perhaps

inevitable that the review methodology applied to them will also need local adaptation.

Discussion

Theory One: PDSA can achieve practice change/outcomes across different

disciplines and organisational contexts/settings.

Figure 4; Illustrates the numbers and proportion of studies used for data extraction that

detailed specific factors of interest to the review question, theories and our discussion.

Figure 4 –Number and proportions of the 94 studies used for data extraction

studies detailing specific factors of interest to the review question (additional file 4)

Application of PDSA across settings and organisations and patient groups

PDSA is a mechanism for QI that has been used to change practice in a wide variety of

disciplines and organisational contexts. The examination of 94 papers included found

that in most cases the context was sufficiently well described that the findings could be

related to other settings (Figure 4 Table 26). 70% of papers were based in either the

USA or Canada, and a further 12% were UK based (see Figure 4 Table 27). Both urban

and rural contexts were described (see Figure 4 Table 28). The projects were based in

secondary care in 63% of cases, and primary care in 18% of cases (see Figure 4 Table

29). Of the 29 hospitals that gave additional descriptive information 12 were described

as a tertiary centre, and a further 12 were described as academic or teaching hospitals.

The patient group receiving the improvement were sometimes defined by clinical

diagnosis (e.g. diabetes, asthma, depression, stroke), some by a procedure or risk factor

(e.g. requiring ventilation, at risk of falling), and some by location of setting (e.g. ITU,

radiology, A&E, haemodialysis, oncology ward). Overall there was a very wide

spectrum of clinical problems and scenarios in which PDSA was used as a tool for

healthcare improvement in some capacity (see notes at the end of Figure 4).

Figure 5 details the references and the associated codes for the information contained in

the subsequent discussion. These codes can be used to identify the key findings from

the review used in our evaluation synthesis (Table 2) to support our discussion and

findings. All of the codes in the subsequent discussion therefore relate back to Figure 5

See Figure 5: – Codes for evidence used in the evaluation synthesis (additional file

5)

Organisational culture, characteristics and stability

The evaluation of the UK patient safety project found no specific organisational

characteristics associated with greater perceived value of PDSA [30, T8/C17], and no

association was found between site characteristics and the achievement of sustaining

and spreading change in a US mental health project [31].

Whilst there is no clear characterisation of the organisation that predicts effective use of

PDSA there are some organisational contexts and cultures in which PDSA appears to be

more or less effective.

A small community hospital A&E department found using PDSA cycles to be a vey

useful tool for achieving improvements [32, T8/C5]. Projects that involve changes in a

well defined geographical area such as an ICU or a single clinic [33, T2/C3] appear

ideal. This may relate at least in part to the visibility of practice change as staff

feedback can be centralised and focused.

Although larger settings may be more likely to contain internal QI expertise, it can be

harder to sustain and spread changes in these contexts possibly due to difficulty

obtaining a ‘critical mass’ that makes changes routine. The impact of any small changes

in larger settings can become diluted, and the changes less visible [31, T8/C3].

A lack of on-site QI expertise may lead to teams struggling [34, T8/C4]. Contexts

which contain QI expertise, most likely larger settings, can be supportive of change

cycles, especially if PDSA is integrated into existing QI programmes. Leadership and

management may also be more supportive where QI is routine. However as identified

above larger settings can have drawbacks.

Organisations with a ‘developmental’ rather than hierarchical culture were associated

with greater staff motivation to conduct PDSA cycles [35, T8/C6]. In contexts where

staff are less familiar with discussing care processes the tool may be less applicable.

A strong culture of clinician autonomy can also be a barrier [30, T8/C13]. A Dutch

mental health project using PDSA found the model did not fit in to the existing culture

of primary care professionals, who found it difficult to apply [36, T8/C7]. A lack of

staff with a prior interest in QI was found to be an important factor in an initiative with

clinical trainees where the PDSA model did not translate into rapid cycles of change

[37, T8/C8].

Organisations able to combine the use of PDSA with other features of QI projects where

information and reporting is shared between other organisations may find it more

effective [38, T8/C16].

A primary care diabetes QI project also found that some changes were implemented

simply as a result of discussion and agreement. There was no requirement in this small

setting to test the changes and roll them out gradually [39, T3/C18]. In these situations

PDSA as a QI tool is made redundant.

Staff, disciplines and management support

When several disciplines were involved poor communication between groups was

identified as a barrier [40, T8/C14]. Offering a ‘virtual’ environment to facilitate

communication does not appear to have been helpful if the staff were used to such

technology [36, T8/C15].

Leadership distraction with other priorities was a significant barrier [41, T8/C2]. Poor

leadership support had a significant impact on the effectiveness of a US mental health

QI project [31, T8/C11]. Senior support is often assessed as crucial [42, T8/C1], with

the possible exception of units which are self-contained or operate fairly independently

such as a neonatal ICU [43, T8/C12].

Organisational and staff stability

As PDSA cycles start small and gradually expand they may be more resilient to

instability and more suitable for changing situations than other QI tools. However

large-scale organisational changes will adversely affect staff motivation to implement

and sustain cycles [34, T9/C1; 42, T9/C2].

Staff turnover was a key barrier to implementation of cycles [44, T9/C3; 45, T9/C4],

particularly if there was reliance on a small number of ‘champions’ [34, T9/C5; 46,

T9/C6]. Staff turnover and retention were often a barrier in contexts such as teaching

hospital [47, T9/C7] and a US mental health collaborative [31, T9/C9] and settings with

a high proportion of part-time staff [34, T9/C8].

Goal definition

PDSA is more effective if goals have clear operational definitions [48, T1/C13]. This

distinction can be described as the difference between control and learning QI projects

[4, T1/C11]. Goals that are applicable in all situations (rather than discretionary)

appeared to be more successful in terms of achieving and sustaining changes when

using PDSA [4, T1/C11]. Such goals are more common and appropriate in settings with

high control, for instance ICU [49, T1/C12].

Stability of patient population

The impact of changes were harder to measure (and so visibility of success reduced)

when the patient population was very transient [44, T2/C9], or when an acute clinical

problem left little time for staff to document an implemented change [46, T2/C10; 40,

T2/C11]. Measuring and documenting the impact of changes may be easier in contexts

where patient populations are more stable, less transient and where patient turnover is

generally lower.

Staff skills, capacity and training

Implementation of the PDSA cycle is difficult without some degree of process-thinking

skill [34, T5/C2]. Some form of training was given to staff in 68% of papers used for

data extraction (see Figure 4, Table 21), and in one third the training covered both the

intervention itself and some type of input on the use of PDSA. In a further third the

training focussed only on the intervention (see Figure 4, Table 22). The concept of

identifying appropriate measures for PDSA can be difficult to grasp [50, T5/C13; 41,

T5/C14]. Skills were also needed to collect and present performance data and these

could also be difficult to learn [50, T5C15]. Training on basic concepts of the process

may need to be consolidated once put into practice [51, T5/C16; 41, T5/C17].

A UK Accident & Emergency-based project found the medical consultants grasped the

concept more easily than senior managers [52, T6/C13]. Ideas regarding

implementation strategies and appropriate measures did not always reside within staff

knowledge bases [53, T5/C5; 50, [T5/C6], although frontline staff took ownership of

sourcing implementation ideas within an ICU based project [49, T5/C4].

External input regarding knowledge of scientific literature and clinical organisational

features from a support team were sometimes required [54, T5/C7]. External support

may also be required for goal setting, for making recommendations about which tools

might be used, and for supplying template tools to implement [31, T5/C10; 43, T5/C11;

50, T5/C9; 55, T5/C12]. Top-down instruction was favoured in a primary care mental

health collaborative [48, T5/C1]), but reduced frontline ownership. Use of external

expertise was cited as key in helping motivate staff in a US secondary care project over

58 sites [42, T6/C24], and staff learning sessions were rated as essential to staff

motivation in a UK patient safety collaborative [30, T6/C25].

Financial incentives for staff could help overcome initial resistance, and were critical to

frontline staff engagement in a Canadian primary care collaborative project [56,

T6/C26].

A lack of dedicated time to conduct a QI project using PDSA can hinder progress [36,

T7/C2]. If staff time is already saturated building in additional processes to implement

change can lead to staff having to work in their own time [34, T7/C3]. A project based

on an in-patient cardiology ward found that the equivalent of a 1.5 full-time nurse’s post

was required for the time spent on daily monitoring, reviewing results, and giving staff

feedback [47, T7/C4].

The additional staff time required was found to be a limiting factor in several projects

[51, T7/C7; 57, T7/C5; 58T7/C6;]. A project based in nursing homes found that other

demands on staff time led to difficulty in persuading staff of the importance of the QI

project [59, T7/C8]. However as the extra work associated with a change became part

of ‘normal routine’ one project observed that workload was reduced as staff worked

more efficiently as a consequence of the improvements [51, T7/C7]. Staff turnover is a

key barrier, particularly if there has been an over–reliance on local ‘champions’. As

such, the engagement of a range of committed staff may help sustain and spread change

to other suitable settings.

Theory Two: PDSA interventions are more likely to lead to practice change when

the changes are measurable, visible and evidence based/plausible.

Theory two postulates that PDSA interventions are more likely to lead to practice

change amongst staff when the changes anticipated (practice changes or changes in

patient outcomes) are measurable in the short term (e.g. within each/early PDSA cycles)

and when practitioners can visibly see that the anticipated changes are taking place, are

based on evidence and plausibly linked to longer term changes in patient/health care

outcomes.

Measuring short term practice outcomes or longer-term health care outcomes

QI projects using PDSA cycles often have an overall aim with a patient or clinical care

focus. However initiatives are often analysed in terms of processes-of-care measures

rather than clinical outcomes. One advantage of process-based measures is that it is

much more likely that an improvement can be demonstrated within a short-time frame:

goals can be defined tightly and specific targets set. A major disadvantage of focussing

on processes-of-care is that the link with clinical benefit can often be unclear, which has

a significant impact on staff motivation. Additionally, although the evidence base for

the overall clinical aim may be robust, the evidence relating the achievement of the

practice change measure to this can be lacking.

The review learning suggests that the most successful projects are those in which the

relationship between process goals and clinical benefit was evident, and in which

clinical benefit for patients could be observed by staff. Translation of process

improvements into clinical improvements may be difficult in the short timeframe of a

specific project however [36, T1/C15; 50, T1/C14], and a cause and effect relationship

between the change made and clinical benefit can be difficult to establish [30, T1/C17;

48, T1/C17].

Both a paediatric asthma QI programme and a project to reduce hospital drug errors

found that outcomes relating to processes of care were more successful than patient

level outcomes [50, T1/C4; 60, T1/C3). Identifying and implementing appropriate

measures could be a difficult task however [30, T1/C5].

It can also be unclear as to what constitutes ‘enough’ evidence when clinical aims are

being set. In a critical care QI project evidence emerged suggesting they might actually

be causing harm instead of benefit [46, T1/C7]. In reality clinical goals were chosen on

the basis of both evidence base and stakeholder priorities [50, T2/C8]. When the risk of

harm was low, for instance improving patient knowledge or reducing patient waiting

times, an evidence base was less important as these non-clinical aims were deemed

beneficial in themselves [38, T1/C9; 61, T1/C10.

Providing feedback on goals and keeping progress visible

Ensuring visibility of data relating to the goals set was also important [48, T2/C4]. A

smaller setting/ sub unit also enables the display of data in a central location that is

visible to all relevant staff, for example a data wall displaying progress towards the

goals set [49, T4/C5].

Visually displaying baseline measures can demonstrate the need for change to staff and

build motivation [48, T6/C1], as can clear presentation of the evidence-base for the

improvement goals [40, T6/C2].

Initial staff resistance can be overcome by ensuring positive changes are highly visible

[62, T6/C30]. Celebrating successes even when small and creating a sense of

competition between departments were found to be effective [46, T6/C34]. However

high visibility acted as a discouragement to staff where little progress was made towards

the goals despite staff effort [32, T6/C32]. A neonatal pain management project

highlighted the need for perseverance even when early results may have been perceived

as a failure, as follow-up data demonstrated success [40, T6/C33]. The difficultly with

interpreting early results has been viewed as ‘demoralising and confusing’ in other

projects [53, T6/C31].

One project using PDSA found the focus on individual patients was very beneficial for

gaining staff enthusiasm as the benefits were directly seen, however without process-

thinking skills the motivation gained did not translate into action with a wider group of

patients [34, T6/C27]. Focussing on small scale changes can lower expectation of the

eventual clinical benefit, and staff may be less motivated to be involved [61]. Other

projects found small changes fuelled enthusiasm that led on to motivation to see larger

changes [50, T6/C28].

Sustaining the changes

Regular feedback is associated with greater ability to sustain change [31, T2/C7], and in

some studies there were periods where the frequency needed to be particularly high [57,

T2/C8]. Even when changes had become part of accepted routine care, ongoing

monitoring and display of measures were necessary to sustain the change [57, T2/C12;

48, T2/C13].

Direct comparison between patients that were included in a PDSA cycle and those that

had not was helpful for sustaining momentum, especially when the link with clinical

benefit was clear, such as a care bundle for ventilated patients [48, T2/C1]).

Improvements are of most value when they are sustainable, and when secondary spread

to other relevant areas is achieved. Sustainability appears to be aided by ensuring

feedback to staff is given regularly, and progress towards goals is kept visible. This is

much more likely to be achieved in situations when the improvements are being carried

out within a defined team in a defined setting.

Theory Three: PDSA leads to greater and more sustained practice change when

frontline staff take ownership of and are engaged with the process

Frontline clinical staff were responsible for delivering the improvement process in 78%

of papers used for data extraction (see Figure 4, Table 23). Active engagement of

frontline clinical staff seemed to be important for the success of projects [63, T6/C6],

and participation itself may have been perceived to create a sense of ownership amongst

staff [50, T6/C9].

Achieving engagement and buy in

Although staff ‘buy-in’ to participating in QI in their area of work is crucial to its

success it is not always easy to achieve. It is unlikely that this difficulty is particular to

PDSA cycles. The small-step cycles concept suggests that it achieves a greater level of

staff empowerment than more radical ‘top-down’ changes. However the reality appears

to be that this is often not the case.

Staff engagement and motivation can be essential in ‘championing’ the spread of

changes to other sites. Capturing the interests of different staff groups may require a

variety of techniques. For example, busy leadership personnel will need to be updated

using a time-effective method, such that their attention is not lost. However, clinicians

may be more engaged if outcomes of clinical interest and relevance are emphasised.

Front line staff have been found to be more engaged when they have a sense of

ownership over the change outcome decisions.

Factors associated with staff resistance

Staff can also demonstrate resistance due to concerns regarding loss of clinical

judgement and control, particularly when changes were protocol and target driven [64,

T6/C4]. Resistance from clinical staff in particular was an issue [50, T6/C13; 31,

T6/C14]. An evaluation of a UK based orthopaedic collaborative using PDSA cycles

found ‘clinicians did not appear to wish to lead’ [53, T6/C10]. The evaluation of the

orthopaedic project found that of 17 components of the collaborative initiative, PDSA

was rated the lowest, with many unconvinced of its value for driving change [53].

However the PDSA process was rated as highly important by both clinical staff and

board-level staff in a UK based patient safety project [30]. A different picture emerged

from a UK Accident & Emergency project in which staff motivation and empowerment

amongst frontline staff was high [52, T6/C5].

Motivation can be difficult to achieve, particularly if more than one hospital department

or team is involved [61, T6/C17; 37, [T6/C18]. Process measures were often judged on

documentation and if there was a perception these tasks were already being done (but

not documented) motivation was poor as there were low expectations of clinical

improvement [57, T6/C4].

Allowing the staff team to decide on change ideas was effective in building ownership,

but was also inefficient as lessons learned elsewhere are re-learned [41, T6/C8]. A

further consequence of a strong sense of ownership was slowing down of the

improvement process as ‘everyone has their say’ [58, T6/C7]. It seems that there needs

to be a balance between frontline ownership and stalling progress as everyone has their

say [58, T3/C22]. Although staff motivation or ‘buy-in’ was critical to achieving

improvement goals, it varies widely between contexts and was not easily predicted [54,

T6/C15].

Participation on a voluntary basis was found to increase motivation [65, T6/C16].

Ensuring a broad range of staff were on the improvement team was important if the

initiative involved multiple sites and disciplines, for example using PDSA cycles to

improve patient transportation [66, T6/C19]. Focussing on ‘early adopters’ or those

staff most favourable disposed to PDSA at the start promoted the involvement of other

staff [48, T6/C20].

Supervisory and management support

Where supervisor support is perceived by staff it facilitates empowerment [35, T6/C21],

but as empowerment is associated with increased scrutiny and responsibility if things go

wrong, staff can be ambivalent about taking greater ownership, and managers may be

reluctant to release staff if they feel it could threaten the security of their role [4,

T6/C22]. The UK A&E project met resistance from the senior management who

appeared threatened by the empowerment of frontline staff.

Theory Four: Rapid PDSA cycles lead to faster improvements and sustained

momentum for change

There are two opposing philosophies in QI, that of ‘gradualism’ or ‘incrementalism’ (to

which PDSA belongs), and that of radical transformational change. Bate et al (2002)

suggest ‘much hangs on which is correct’[53]. Plsek (1999) suggests that ‘pace is

crucial’ in the utilisation of PDSA cycles [67], however the reality captured in our

review seems to be that momentum can be difficult to both achieve and sustain.

The likely speed of PDSA cycles and of change

There was reasonable evidence of the relationship between changes observed and use of

PDSA cycles in 46% of papers used for data extraction (see Figure 4, Table 19). There

were examples of projects having been able to conduct cycles rapidly, for instance a US

drug error QI project conducted 739 cycles over 15 months, of which 63% were

described as ‘real tests of change’, the remainder being ‘educative’ [50, T3/C2].

There were, however, numerous examples in the literature of single PDSA cycles taking

place over a number of weeks or months [37, T3/C6; 53, T3/C7; 68, T3/C8; 69, T3/C8;

45, T3/C9; 57, T3/C10; 70, T3/C11; 71, T3/C13]. Along with other enthusiasts of the

PDSA method Kilo (1998) states ‘dramatic, rapid changes in outcomes can (and do)

occur within months’ [72]. Our review suggests that this expectation is not often met,

and it is often more realistic to prepare for slower and less radical change. This

somewhat contradicts the ‘rapid-cycle’ terminology used to describe PDSA.

Reasons for the relatively slow change

There seems to be a variety of reasons for slower cycle speeds such as difficulties

translating the goals of a project into small steps that can be tested within cycles

conducted in rapid succession. Concerns exist that as cycles involve increasing

numbers of patients, inadequate time for learning may be given if cycles are conducted

too rapidly, especially when new skills amongst staff need to be developed [33,

T3/C14]. Process changes that involve a small number of patients (for instance a

clinical problem which presents less than once a week) means that changes are harder to

break down into small steps and cycles are slower [61, T3/C19]. Spending too long

collecting baseline data was thought to be a common cause of loss of momentum in a

US drug error initiative [50, T3/C20]. Projects that were required to generate ideas

internally were found to conduct cycles more slowly [43, T3/C1]. In addition

inclusivity and engagement of a wide range of staff all slow cycles down.

Some changes necessitated large-scale disruption to supporting services, even when

applied to small number of patients. In these cases the disruption to the system was

greater if tested in small steps than if changes were made in one large cycle 61,

T3/C15]. A UK orthopaedic QI initiative found that specifying changes must be

achieved in small steps limited the changes that were achievable [53, T5/C16]. There

were also some changes which were implemented immediately on a wide scale with

little difficulty, for instance making a decision to remove bags of potassium chloride

from ward supplies [50, T3/C17]. A primary care diabetes QI project also found that

for some changes it was sufficient to discuss and agree on the changes that were needed

in order to see them implemented.

There seems also to be a relationship between this and Theory Two concerning the

importance of the visibility of changes and cycle speed. Visibility of the changes was

co-dependent on the speed & size of the cycles – a large change would go through the

cycle at a lower speed, reducing the visibility of the change due to too little timely

feedback [61, T2/C6]. Given this slower cycle speeds may impact on both effectiveness

and sustainability of resultant changes.

Critics of the incremental approach suggest that energy and enthusiasm is worn away

over time as everyday problems continue to present themselves, without the time,

support or capability to overcome them. Our learning on cycle speed and resultant

change may add weight to this view.

Theory Five: Changes can be spread into other settings and contexts

Descriptions of spread of the improvements within the local context (or plans to do so in

the future) were given in only 41% of studies used for data extraction (see Figure 4,

Table 20). There is little evidence of diffusion of improvements locally without

intentional action or the setting intentional targets [36, T4/C8; 31, T4/C9; 34, T4/C7].

Barriers to sharing good practice

Implementation of ideas to achieve goals were often context specific, so spread could be

difficult where systems differed in the new setting, even if the overall aim was

unchanged [61, T7/C10]. Differing IT resources for example acted as a barrier to

exchange between settings [31, T4/C11].

Even within settings some studies report the need to start again with small cycles in a

new setting to encourage the spread of good practice even if only small changes were

made to implementation [48, T4/C3]. For example a chronic pain QI project based in

nursing homes found that there was no uptake of changes in other homes [59, T4/C4].

There was also difficulty when staff involved in the first site moved on before spread

was achieved [41, T4/C6].

What encourages the spread of good practice?

Senior management and administrative support is likely to be required if spread is to

occur in terms of both facilitation and finances, but the evidence of benefit provided by

PDSA cycles may not be deemed robust enough to justify this [46, T4/C1; 41, T4/C2].

It seems valuable to use staff from the initial context to help start momentum in a new

setting [48, T4/C5]. Presentation of results to wider audience, for example at hospital

teaching meetings, stimulated interest [37, T4/C12] although this was less of an issue on

smaller sites [31, T4/C13].

A psychiatric in-patient QI project recognised that widespread change was unlikely

without achieving systematic change regardless of the use of PDSA cycles [65,

T4/C14]. An evaluation of a UK orthopaedic collaborative also found that more radical

system design was required (alongside small step changes), to achieve broader

achievement of QI [53, T4/C15]. With regard to this theory the literature seems to

suggest that spread can often be slow and needs active input.

Theory Six: Improved practice in PDSA can be achieved and sustained as aresult

of existing (rather than new) resources

Few of the papers used for data extraction gave information on their costs or resource

requirements. Costs were stated clearly in only 11% of papers, and only further 11%

gave any information at all (see Figure 4, Table 23). The oversight or support for

projects was described in 80% of papers, and often this came in part from an external

team, although the actual nature of the relationship was often unclear (see Figure 4,

Table 24).

Are existing resource sufficient for successful implementation of PDSA?

The concept that PDSA cycles make use of existing resources and can therefore be

conducted without additional investment does not often appear to hold true. Financial

investment was often needed for staff training, staff time, implementation costs, and

monitoring costs. This was sometimes considerable [73, T7/C14; 31, T7/C15; 50,

T7/C16].

A local co-ordinator may be employed to support teams, for which financial resources

may need to come from external sources [36, T7/C17]. Training costs can be reduced

by telephone or ‘virtual’ provision, but may not be favoured by staff compared to face-

to-face training [41, T7/C18]. Changes in a large highly systematised organisation may

be more costly to achieve than in smaller independent practices [54, T7/C19]. In

centrally coordinated projects the resource demands can be too prohibitive to enable

participation by all [50, T7/C20].

Cost related to monitoring process and outcome data

If monitoring involved data collection not already gathered routinely on existing IT

systems it consumed a significant amount of staff time [61, T7/C9]). If a specific time

limited team coordinated this, the monitoring may have been subsequently been

integrated into normal staff roles [54, T2/C14]. Plans for ongoing monitoring and

maintenance of improvements were stated clearly in only 13% of the papers used for

data extraction; a further 28% contained some information, and 55% had no information

(see Figure 4, Table 18).

Information technology infrastructure was critical to successful integration of changes

into routine care [74, T7/C10], but was sometimes newly created and supported [44,

T7/C11; 30, T7/C12] and could be expensive [54, T7/C13]. In view of the financial,

staff, IT, and training resources that may be required it is important that an assessment

of existing capacity in each of these areas is conducted before the initiation of a project

[30, T7/C1].

Findings and recommendations

This review has identified a number of factors that seem to enable PDSA to be used

more or less effectively. We summarise this learning below.

PDSA has been used successfully across a variety of: geographies; settings;

organisations; clinical populations; and, patient groups. There seem to be both

advantages and disadvantages relating to the size of the organisation. The culture of an

organisation is perhaps more important than its size.

The effectiveness of PDSA in a particular setting cannot be fully predicted as it will be

impacted on by a variety of implementation and contextual factors. It appears however

that it may stand a greater chance of success in smaller and well defined settings. A

‘developmental’ rather than ‘hierarchical’ organisational/team culture is associated with

a greater motivation to conduct PDSA cycles.

Success is also likely to be greater in settings where staff and /or patient groups are

stable and less transient where more time is available and it is easier to document and

measure change. Figure 7 shows a template developed by us in response to the learning

above that may aid organisations decide whether they have setting/context in which

PDSA would work well.

See Figure 6 – Template: Assessing the applicability of using PDSA to achieve

quality improvement in your context & objectives (additional file 6)

PDSA projects are more likely to improve practice when process goals are linked via

evidence/logic to the intended outcomes (changes in practice and patient health) and

where these links are evident to, and observable by, staff.

Similarly effectiveness will be greater where the practice and clinical outcomes sought

are clearly identified and measurable within reasonably short time frames.

Focussing on small scale change may lower expectation of eventual clinical benefits but

significant evidence suggests that small changes can fuel enthusiasm and in turn, a

motivation to see greater changes.

Framing PDSA as ‘a scientific method’, ‘real-time science’ or the ‘science of change’

[66, 72, 75]. rather than as an improvement tool may credit it with too much ‘scientific

rigour’, establish unrealistic expectations, and lead to disappointment and diminishing

commitment from staff. A better (heuristic) approach might be to describe PDSA as a

model for ‘daily work application’ [76] or to build it into existing knowledge and skill

in audit methodology, rather than as a new approach. PDSA cycles should be integrated

into current QI programmes of work where they exist.

Approaching QI by small-scale changes does not appear to be possible, appropriate or

necessary in all situations and in some cases a different improvement tool may be

needed. PDSA projects are more likely to be effective if staff are actively engaged in

the process (identification and measurement of the changes) and if they have prior

exposure to QI processes or support from others who have QI skills. A broad range of

staff should be on the improvement team if the intervention is seeking change in

multiple sites or disciplines. Early adopters can be used to motivate engagement

amongst other staff. Training in process thinking skills is likely to enhance

understanding and participation.

Whilst improved practice is likely to happen faster and be sustained longer when PDSA

cycles are quicker and shorter such speed is difficult to achieve especially in contexts

with high or increasing patient numbers or addressing rare conditions. These contexts

provide more limited time for implementation and measurement. Ensuring staff

engagement and participation can also slow the change process.

It seems to be harder to sustain and spread good practice within larger settings however

such settings may have a stronger QI culture, greater QI expertise and leadership and

management support available which are likely to enhance outcomes.

Enhanced practice is sustained and likely to be spread more widely when evidence of

progress/feedback is highly visible to staff, frequent and provided in central locations.

This is particularly important within longer PDSA cycles

In order to implement PDSA cycles effectively staff required knowledge of the

evidence base, knowledge of appropriate measures to achieve a particular aim, and

knowledge of their organisational dynamics. Technical skills relating to the monitoring

progress are also required. External input to train and support frontline staff in these

aspects is often required. Resourcing capability and support internally is also important

for reducing potential organisational structural barriers to change and spreading change.

Before embarking on a PDSA project it might be useful to conduct a needs assessment

of capacity and capability and (where necessary) follow this up with a realistic business

plan. Depending on the scale and aims of the project, organisations participating in

PDSA QI initiatives need to provide sufficient additional finances, staff time, and IT

infrastructure. Careful thought as to how current IT capability in healthcare

organisations may change the dynamics of the PDSA

Achieving spread of improvements to other areas once they have been tested and

implemented in one area involves intentional and planned input.

Conclusions

This review and synthesis of the literature reporting evidence of PDSA implementation

in healthcare QI initiatives has made a small contribution to advancing our knowledge

and understanding of how and why application of the tool by clinicians and others may

or may not lead to successful improvements in patient care, particularly with regard to

the influential role of the contextual and socio-cultural factors at play.

List of abbreviations

A & E Accident and Emergency

IHI Institute of Healthcare Improvement

NHS National Health Service

PDCA Plan-Do-Check-Act

PDSA Plan-Do-Study-Act

PRISMA Preferred Reporting Items for Systematic reviews and Meta-Analyses

QI Quality Improvement

RCT Randomised Controlled Study

SQUIRE Standards for QI Reporting Excellence

Competing interests The authors state that there are no competing interests..

Authors' contributions

EC co-led the study design, conducted the literature searches, coordinated data

collection, analysis, synthesis and interpretation, and wrote the first draft. AB assisted

with study design, theory refinement, data synthesis and interpretation and contributed

to the writing of the manuscript. JF, HH and JM assisted with literature searches, data

analysis, synthesis and interpretation and contributed to the writing of the manuscript.

PB conceived the study idea, acquired funding, co-led the study design, assisted with

theory refinement, data synthesis and interpretation and contributed to the writing of the

manuscript.

Acknowledgements

We thank Dr Alison Powell, University of St Andrews, Dr Michelle Beattie, University

of Stirling, and Dr Avril Blamey, Planning & Evaluation Consultant (Avril Blamey and

Associates), for critically reviewing and commenting on earlier drafts of this report. We

are also grateful to Drs Claire Tochel and Helen Allbutt, NHS Education for Scotland,

for providing advice on systematic literature review methods.

The study was funded by the Patient Safety Multi-Professional Steering Group of NHS

Education for Scotland, a special health authority responsible for the education &

training and life-long development of the healthcare workforce in Scotland.

References

1. Speroff T, O’Connor GT. Study designs for PDSA quality improvement research.

Quality Management in Healthcare, 2004;13:17-32.

2. Deming, W. Edwards. Out of the Crisis. MIT Press (1986).

3. Shewhart, Walter A. Statistical Method from the Viewpoint of Quality Control.

(1939) Dover Publications December 1, 1986. ISBN 0-486-65232-7. OCLC 13822053.

4. Bloor G. Organisational culture, organisational learning and total quality

management: A literature review and synthesis. Australian Health Review,

1999;22:162-179.

5. Langley GJ, Moen RD, Nolan KM, Nolan TW, Norman CL, Provost LP. The

Improvement Guide 2nd Ed. Josey-Bass (2009).

6. Institute for Healthcare Improvement. The Breakthrough Series. Boston (2003).

(www.IHI.org Accessed 10th July 2012).

7. Boaden R, Harvey G, Moxham C, and Proudlove N. Quality improvement: theory

and practice in healthcare. NHS Institute for Innovation and Improvement,

University of Warwick, Coventry UK (2008).

8. Scottish Patient Safety Programme. Welcome to the Scottish Patient Safety

Programme. http://www.scottishpatientsafetyprogramme.scot.nhs.uk/programme

[Accessed 10th July 2012]

9. Safer Patients Initiative. Learning Report: Safer Patients Initiatives. The Health

Foundation, February 20011. http://www.health.org.uk/publications/safer-patients-

initiative/ Accessed 10th July 2012

10. Grol R, Berwick D, Wensing M. On the trail of quality and safety in health

care. BMJ, 2008;336:74-76.

11. Landon BE, Wilson IB, McInnes K, Landrum MB, Hirschhorn L, Marsden PV,

Gustafson D, Cleary PD. Effects of quality improvement collaborative on the

outcomes of care of patients with HIV infection: the EQHIV study. Annals of

Internal Medicine, 2004;140:887-896.

12. Schouten LMT, Hulscher MELJ, van Everdingen JJE, Huijsman R, Grol RPTM.

Evidence for the impact of quality improvement collaboratives: a systematic

review. BMJ, 2008;336:1491-4.

13. Boonyasai RT, Windish DM, Chakroborti C, Feldman LS, Rubin HR, Bass EB.

Effectiveness of teaching Quality improvement to clinicians – A systematic review.

Journal of the American Medical Association, 2007;298: 1023-1036.

14. Auerbach AD, Landefeld CS and Shojania KG. The tension between needing to

improve care and knowing how to do it. New England Journal of Medicine.

2007;357: 608-613.

15. Walshe K. Understanding what works - and why – in quality improvement: the

need for theory driven evaluation. International Journal for Quality in Health Care,

2007;19:57-59.

16. Mittman BS. Creating the evidence base for quality improvement

collaboratives. Annals of Internal Medicine, 2004;140:897-900.

17. Ovretveit J, Bate P, Cleary P, Cretin S, Gustafson D, McInnes K, McLeod H,

Molfenter T, Plsek P, Robert G, Shortell S, Wilson T. Quality collaboratives: Lessons

from research. Quality and Safety in Healthcare, 2002;11:345-351.

18. Walshe K. Pseudoinnovation: the development and spread of healthcare

quality improvement methodologies. International Journal for Quality in Health

Care, 2009; 21:153-159.

19. Lindenauer PK. Effects of quality improvement collaboratives. BMJ,

2008;336:1448-1449.

20. Soper B, Buxton M, Hanney S, Oortwijn W, Scoggins A, Steel N, Ling T.

Developing the protocol for the evaluation of the health foundation’s ‘engaging

with quality initiative’ - an emergent approach. Implementation Science, 2008;3:46.

21. Lavis JN, Davies HTO, Gruen RL, Walse K, Farquhar C. Working within and

beyond the Cochrane collaboration to make systematic reviews more useful to

healthcare managers and policy makers. Healthcare policy, 2006;1:21-33.

22. Pawson R.(2002). Evidence-based policy: the promise of ‘realist synthesis’.

Evaluation, 8:340-358.

23. Pawson R, Tilley N. (1997). Realistic Evaluation. Sage Publications.

24. Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist synthesis: an

introduction. ESRC research methods programme working paper series. ESRC:

August 2004.

25. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche P, et al. and the PRISMA

Group. The PRISMA statement for reporting systematic reviews and meta-

analyses of studies that evaluate health care interventions: Explanation and

elaboration. PLoS Med 6: e1000100. 2009; doi:10.1371/journal.pmed.1000100

26. Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O, Peacock R.

Storylines of research in diffusion of innovation: a meta-narrative approach to

systematic review, Social Science & Medicine, 2005;61:417-430.

27. Davidoff F, Bataldan P, Stevens D, Ogrinc G and Mooney SE. Publication

guidelines for quality improvement studies in health care: evolution of the

SQUIRE project. BMJ, 2009;338:a3152.

28. Greenhalgh T, Potts HWW, Wong G, Bark P, Swinglehurst D. Tensions and

paradoxes in electronic patient record research: a systematic literature review

using the meta-narrative method. The Millbank Quarterly, 2009;87:4.

29. Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist review – a new method of

systematic review designed for complex policy interventions. Journal of Health

Services Research 2005’10:S1 21-34.

30. Benn J, Burnett S, Parand A, Pinto A, Iskander S and Vincent C. Perceptions of the

impact of a large-scale collaborative improvement programme: experience in the

UK safer patients initiative. Journal of Evaluation in Clinical Practice, 2009;5:524-

540.

31. Meredith LS, Mendel P, Pearson M, Wu S, Joyce G, Straus JB, Ryan G, Keeler E,

Unutzer J. Implementation and maintenance of quality improvement for treating

depression in primary care. Psychiatric Services, 2006;57:48-55.

32. Warburton RN. Preliminary outcomes and cost-benefit analysis of a community

hospital emergency department screening and referral program for patients aged

75 or more. International Journal of Health Care, 2005;18:474-484.

33. Varkey P, Sathananthan A, Scheifer A, Bhagra S, Fujiyoshi A, Tom A, Murad MH.

Using quality-improvement techniques to enhance patient education and

counselling of diagnosis and management. Quality in Primary Care, 2009;17:205-13.

34. Macintosh-Murray A. Challenges of collaborative improvement in continuing

care. Healthcare Quarterly, 2007;10:49-57.

35. Lin MK, Marsteller JA, Shortell SM, Mendel P, Pearson M, Rosen M, Wu S.

Motivation to change chronic illness care: results from a national evaluation of

quality improvement collaboratives. Health Care Management Review, 2005;30:139-

156.

36. Franx G, Meeuwissen JA, Sinnema H, Spijker J, Huyser J, Wensing M, de Lange J.

Quality improvement in depression care in the Netherlands: the depression

breakthrough collaborative. International Journal of Integrated Care, 2009;9:1-11.

37. Tomolo AM, Lawrence RH, Aron DC. A case study of translating ACGME

practice-based learning and improvement requirements into reality: systems

quality improvement projects as the key component to a comprehensive

curriculum. Quality & Safety in Health Care, 2009;18:217-224.

38. Nembhard IM. Learning and improving in quality improvement collaboratives:

which collaborative features do participants value most? Health Services Research,

2009;44:359-378.

39. Geboers H, Grol R, van den Bosch W, van den Hoogen H, Mokkink H, van

Montfort P, Oltheten H. A model for continuous quality improvement in small scale

practices. Quality in Health Care, 1999;8:43-48.

40. Dubar III AA, Sharek PJ, Mickas NA, Coker, KL, Duncan J, McLendon D,

Pagain C, Puthoff TD, et al. Implementation and case study results of potentially

better pratcies to improve pain management of neonates. Pediatrics. 2006;118:

Supplement 2. 587-595

41. Gould DA, Lynn J, Halper D, Myers SK, Simon L. The New York city palliative

care quality improvement collaborative. The Joint Commission Journal on Quality an

Patient Safety, 2007;33: 307-316.

42. Leape LL, Rogers G, Hanna D, Griswold P, Federico F, Fenn CA, Bates DW, Kirle

L, Clarridge BR. Developing and implementing new safe practices: voluntary

adoption through statewide collaboratives. Quality & Safety in Health Care,

2006;15:289-295.

43. Wilson T, Berwick DM, Cleary PD. What do collaborative improvement projects

do? Experience from seven countries. Joint Commission Journal on Quality and

Safety, 2003;29:85-93.

44. Wagner EH, Glasgow RE, Davis C, Bonomi AE, Provost L, McCulloch D, Carver

P, Sixta C. Quality improvement in chronic illness care: a collaborative approach.

Journal on Quality Improvement, 2001;27:63-80.

45. Carboneau CE. Achieving faster quality improvement through the 24 hour team.

Journal for Quality in Healthcare, 1999;21: 4-10.

46. Lipshutz AK, Fee C, Schell H, Campbell L, Taylor J, Sharpe BA, Nguyen J,

Gropper MA. Strategies for success: a PDSA analysis of three QI initiatives in

critical care. The Joint Commission Journal on Quality and Patient Safety,

2008;34:435-444.

47. Nolan E, VanRiper S, Talsma A, Mageno L, Richter A, Kearly G, Kendrick C,

Leggett S, Crissey J, et al. Rapid-cycle improvement in quality of care for patients

hospitalized with acute myocardial infarction or heart failure: moving from a

culture of missed opportunity to a system of accountability. The Journal of

Cardiovascular Management, 2005;Jan:14-19.

48. Pulcini C, Crofts S, Campbell D, Davey P. Design, measurement, and evaluation

of an education strategy in the hospital setting to contact antimicrobial resistance:

theoretical considerations and a practical example. Disease Management & Health

Outcomes, 2007;15:151-163.

49. Krimsky WS, Mroz IB, McIlwaine JK, Surgenor SD, Christian D, Corwin HL,

Houston D, Robison C, Malayaman N. A model for increasing patient safety in the

intensive care unit: increasing the implementation rates of proven safety measures.

Quality & Safety in Health Care, 2009;18:74-80.

50. Leape LL Kabcenell AI, Gandhi TK, Carver P, Nolan TW, Berwick DM. Reducing

adverse drug effects: lessons from a breakthrough series collaborative. Journal on

Quality Improvement, 2000;26:321-331.

51. Fremont AM, Joyce G, Anaya HD, Bowman CC, Halloran JP, Chang SW, Bozzette

SA, Asch SM. An HIV collaborative in the VHA: do advanced HIT and one-day

sessions change the collaborative experience? Joint Commission Journal on Quality

and Patient Safety, 2006;32:324-336.

52. Walley P, Gowland P. Completing the circle: from PD to PDSA. International

Journal of Heatlh Care Quality Assurance, 2004;17:349-358.

53. Bate P, Robert G and McLeod H. Report on the ‘breakthrough’ collaborative

approach to quality and service improvement within four regions of the NHS.

Health Services Management Centre Research report No. 42. February 2002.

54. Liu CF, Rubenstein LV, Kirchner JE, Fortney JC, Perkins MW, Ober SK, Pyne JM,

Chaney EF. Organizational cost of quality improvement for depression care. Health

Services Research, 2009;44:225-244.

55. Moeschler JB, Amato, RS, Brewster, T, Burke, L, Dinulous, MB, Smith, R, Smith

W, and Miller, P. Improving genetic healthcare: A Northern New England pilot

project addressing the genetic evaluation of the child with developmental delays or

intellectual disability. American Journal of Meidcal Genetics. 2009;151c: 3; 241-254

56. Green CJ, Fortin P, Maclure M, Macgregor A, Robinson S. Information system

support as critical success factor for chronic disease management: necessary but

not sufficient. International Journal of Medical Informatics, 2006;75:818-828.

57. Gordon DB, Rees SM, McCausland MP, Pellino TA, Sanford-Ring S, Smith-

Helmenstine J, Danis DM. Improving reassessment and documentation of pain

management. The Joint Commission Journal on Quality and Patient Safety,

2008;34:509-517.

58. Bolch D, Johnston JB, Giles LC, Whitehead C, Phillips P and Crotty M. Hospital to

home: an integrated approach to discharge planning in a rural south Australian

town. Australian Journal of Rural Health, 2005;13:91-96.

59. Buhr GT, White HK. Quality improvement initiative for chronic pain

assessment and management in the nursing home: a pilot study. Journal of the

American Medical Directors Association, 2006;7:246-253.

60. Mangione-Smith R, Schonlau M, Chan KS, Keesey J, Rosen M, Louis TA, Keeler

E. (2005). Measuring the effectiveness of a collaborative for quality improvement

in pediatric asthma care: does implementing the chronic care model improve

processes and outcomes of care? Ambulatory Pediatrics, 5:75-82.

61. Vos L, Duckers MLA, Wagner C, van Merode GG. Applying the quality

improvement collaborative method to process design: a multiple case study.

Implementation Science, 2010;5:19.

62. Hampton DC, Griffith D, Howard A. Evidence-based clinical improvement for

mechanically ventilated patients. Rehabilitation Nursing, 2005;30:160-165.

63. Gitomer RS. Improving access, quality of care, and patient satisfaction in a

general internal medicine practice. Journal of Clinical Outcomes Management 2005

May;12(5):245-249.

64. Nicholls, S, Cullen R, and Halligan A. Clinical Governance ...after the review-

What next? Agreement and implementation. British Journal of Clinical Governance.

2001;6:2:129-135

65. Barry S, Hughes G, LawtonSmith S. Small is beautiful. Mental Health Today.

2006 Jun.pp.30-3 2006.

66. Esmail R, Banack D, Cummings C, Duffet-Martin J, Rimmer J, Shultz J, Thurber

T, Hulme T. Is your patient ready for transport? Developing an ICU patient

transport decision scorecard. Healthcare Quarterly, 2006;9:80-86.

67. Plsek PE. Quality improvement methods in clinical medicine. Paediatrics,

1999;103: 203-214.

68. Henry D, Muriel FR, Hirway P. Sustaining improvement in surgical infection

prevention measures for hysterectomy. Journal for Healthcare Quality, 2007;29:50-

56.

69. Bittle MJ, Charache P and Wassilchalk DM. Registration-associated patient

misidentification in an academic medical center: causes and corrections. Joint

Commission Journal on Quality and Patient Safety, 2007;33: 25-33.

70. Marang-van de Mheen PJ, Stadlander M, Kievit J. Adverse outcomes in surgical

patients: implementation of a nationwide reporting system. Quality & Safety in

Health Care, 2006;15:320-324.

71. Gray J, Eden G, and Williams M. Developing the public health role of a front

line clinical service: integrating stop smoking advice into routine podiatry services.

Journal of Public Health. 2007;29:2:118-122

72. Kilo CM. A framework for collaborative improvement: lessons from the

institute for healthcare improvement’s breakthrough series. Quality management in

health care, 1998;8:1-13.

73. Mercier CE, Barry SE, Paul K, Delaney TV, Horbar JD, Wasserman RC, Berry P,

Shaw JS. Improving newborn preventative services at the birth hospitalization: a

collaborative, hospital-based quality-improvement project. Paediatrics,

2007;120:481-488.

74. Green CJ, Fortin P, Maclure M, Macgregor A, Robinson S. Information system

support as critical success factor for chronic disease management: necessary but

not sufficient. International Journal of Medical Informatics, 2006;75:818-828.

75. Berwick, DM. Harvesting knowledge from improvement. The Journal of the

American Medical Association 1996;275 877-878.

76. Batalden PB and Stoltz PK. A framework for the continual improvement of

health care: building and applying professional and improvement knowledge to

test changes in daily work. Joint Commission Journal on Quality Improvement.

1993;19: 424-452.

Figure 1: Formal Search Criteria

a) Search Terms:

1. PDSA

2. “plan do study act”

3. PDCA

4. “plan do check act”

5. SDSA

6. “standardi?e do study act”

7. “review agree* implement* demonst*”

8. “rapid cycle change”

9. “rapid cycle improvement$”

10. “rapid improvement cycle$”

11. “Deming cycle”

12. (Deming and cycle)

13. “Deming wheel”

14. (Deming and wheel)

15. “Shewhart cycle”

16. (Shewhart and cycle)

17. “FOCUS-PDSA”

18. “small step change method*”

19. “breakthrough method*”

20. “breakthrough series”

21. “breakthrough collaborative$”

22. “model for improvement”

23. “Rapid Improvement Event$”

b) Databases:

Medline 1996+

Embase 1996+

British Nursing Index 1994+

Pychinfo 2002+

HMIC: Health Management Improvement Consortium

MIDIRS: Maternity & Infant Care

All EBM Review

Figure 2: – Data extraction form (Developed by EC, drawing on TG

(2005) data extraction form, PRISMA & SQUIRE guidelines, and discussions with team

members)

DATA EXTRACTION FORM FOR A PAPER BEING CONSIDERED FOR NES REVIEW OF RAPID

CYCLE IMPROVEMENT PROCESSES IN HEALTH CARE

AUTHOR/TITLE OF PAPER

YEAR OF PUBLICATION

NAME OF REVIEWER

A. [FIRST SIFT] Is the paper relevant to our resear ch question and worthy of further consideration?

Relevance. Do the authors refer to either a PDSA or PDCA process, or IHI

collaborative methodology for use in quality improvement in healthcare?

Worth. Does the paper go beyond superficial description or commentary – i.e. is

it a broadly competent attempt at research, enquiry, investigation or study? [If a

confident ‘no’ to either of these, reject now]

B. How does the paper fit into larger framework?

1. Paradigm. What is

the predominant

theoretical ‘lens’ used?

[if more than one, put

double circle round the

dominant one]

1. Business

management theory

2. Manufacturing

theory

3. Education / learning

theory

4. Leadership theory

5. Translational

medicine theory

(implementing EBM)

6. Complexity /

general systems theory

7. Health economics

theory

8. Organisational

theory

9. Other (specify)

NOTES

2. Type of paper.

How does the paper

fit into our taxonomy

[classify as the MAIN

pitch of the paper]

1. Theory or

conceptual framework

2. Editorial review,

commentary or

opinion

3. Systematic review 4. RCT

5. Non-RCT

experimental or quasi-

experimental study

6. Questionnaire

survey

7. Qualitative interview

study (incl focus

group)

8. Ethnographic study

(‘anthropological’ case

study)

9. Mixed methodology

case study

10. Action research 11. Tool/ checklist /

component of model

12. Guideline/pathway

implementation

13. Comparative case

study

14. Network analysis 15. Other (specify)

NOTES

3. Perspective. What is

the paper’s main unit

of analysis?

Individual Group or

team

Department Organisation Inter-

organisational

Regional

/

national

Multi-level

NOTES

Figure 3 Algorithm of data base search results and exclusions

762 papers identified on electronic

database search

231 duplicates identified by OVID

531 papers abstracts screened for exclusion criteria:

1. Relevance. Does the paper refer to any type of rapid cycle process used in QI

in healthcare?

Total 385 papers excluded

172 rejected as topic not relevant to

question

33 rejected as no data of worth

134 papers excluded as review limited to

last 5 yrs

146 full text articles assessed for

eligibility 52 further papers excluded

25 rejected as topic not relevant to

question

Data extraction from 94 papers (PART B of

review); 44 in total populated our evaluation

Figure 4 –Number and proportions of studies detaili ng specific factors of interest to the review question

Table No.

Description Frequency Percentage

1 B2. Type of paper

1. Conceptual framework 1 1%

2. Editorial review 0 0%

3. Systematic review 0 0%

4. RCT 1 1%

5. Quasi-experimental 47 50%

6. Questionnaire 4 4%

7. Qualitative interview study 5 5%

8. Ethnographic study 0 0%

9. Mixed methods case study 8 9%

10. Action research 0 0%

11. tool / checklist / component of model 3 3%

12. Guideline/pathway implementation 21 22%

13. Comparative case study 1 1%

14. Network analysis 0 0%

15. Other 3 3%

descriptive cost analysis (1) 94 100%

review, not systematic (2)

2 B3. Perspective

1. Individual 23 24%

2. Group / team 23 24%

3. Department 24 26%

4. Organisation 11 12%

5. Inter-organisation 9 10%

6. Regional/national 2 2%

7. Multi-level 2 2%

94 100%

3 C2. Objective

1. PDSA itself 31 33%

2. Outcomes only 63 67%

94 100%

4 3D. Evaluation of outcomes

1. Records 30 32%

2. Other 53 56%

3. Unclear 11 12%

94 100%

5 12A. Data collection

1. Clearly explained 61 65%

2. Not clear 23 24%

3. Partial information 10 11%

94 100%

6 13A Sufficient data presented

1. Yes 47 50%

2. No 35 37%

3. Partial 12 13%

94 100%

7 3C. Comparison group

1. None 34 36%

2. Baseline only 49 52%

3. Yes 6 6%

4. N/a 5 5%

94 100%

8 8. Control

1. stated 6 6%

2. No info 0 0%

3. N/a 88 94%

94 100%

9 13B. Disconfirming observations

1. limited info 69 73%

2. partial info 19 20%

3. clear account 6 6%

94 100%

10 16. Conclusions appropriate

1. Yes 57 61%

2. No 22 23%

3. Unclear 15 16%

94 100%

11 4. Funding acknowledged

1. No 53 56%

2. Yes 41 44%

94 100%

12 17. Reflexivity - potential conflicts of interest

1. No statement 68 72%

2. State none 18 19%

3. Some info 8 9%

94 100%

13 C1. Clear aim

1. Yes 91 97%

2. No 3 3%

94 100%

14 5C / 5D. Trigger

1. Stated 57 61%

2. Unclear 34 36%

3. N/a 3 3%

94 100%

15 7. Receiving the improvement

1. Patient 79 84%

2. Other 12 13%

3. Mixed 3 3%

94 100%

16 9. Improvement aim

1. Safety 16 17%

2. Clinical effectiveness 41 44%

3. Patient centredness 19 20%

4. Timeliness 0 0%

5. Efficiency 8 9%

6. Equity 0 0%

7. Cost Effectiveness 0 0%

8. Other 10 11%

94 100%

17 B1. Paradigm (theoretical lens used)

1. Business management theory 3 3%

2. Manufacturing theory 1 1%

3. Education / learning theory 11 12%

4. Leadership theory 1 1%

5. Translational medicine theory 50 53%

6. Complexity /general systems theory 4 4%

7. Health economics theory 1 1%

8. Organisational theory 22 23%

9. Other - Behavioural theory 1 1%

94 100%

18 11B. Ongoing monitoring plans

1. None stated 52 55%

2. Some info 26 28%

3. Clear plans 12 13%

4. N/a 4 4%

94 100%

19 14D. PDSA evidence

1. Good 17 18%

2. Some 26 28%

3. Poor 38 40%

4. None 13 14%

94 100%

20 15. Local dissemination process

1. Stated 39 41%

2. Not stated 51 54%

3. N/a 4 4%

94 100%

21 6C. Training provided

1. Yes 64 68%

2. No 4 4%

3. Unclear 21 22%

4. N/a 5 5%

94 100%

22 6D. Focus of training

1. PDSA 7 7%

2. Intervention 28 30%

3. Both 27 29%

4. Unclear 24 26%

5. N/a 8 9%

94 100%

23 6A. Delivering changes

1. Clinical staff 73 78%

2. Other 7 7%

3. N/a 5 5%

4. Unclear 9 10%

94 100%

24 11A. Costs

1. Stated 10 11%

2. Not stated 71 76%

3. Limited info 10 11%

4. N/a 3 3%

94 100%

25 5A. Support

1. Stated 75 80%

2. Not stated 3 3%

3. Unclear 14 15%

4. N/a 2 2%

94 100%

26 10E. Sufficient contextual information

1. Yes 69 73%

2. No 24 26%

3. N/a 1 1%

94 100%

27 10A. Country

1. USA 57 61%

2. UK 11 12%

3. Other Europe 5 5%

4. Australia 5 5%

5. Other world 5 5%

6. Mixed 3 3%

7. Canada 8 9%

94 100%

28 10B. Geography

1. Rural 9 10%

2. Urban 19 20%

3. Mixed 17 18%

4. Unclear 49 52%

94 100%

29 10C. Setting

1. Primary 17 18%

2. Secondary 59 63%

3. Mixed 11 12%

4. Unclear 3 3%

5. Other 4 4%

94 100%

Other methods of evaluating outcomes (total greater than 53 as some using >1 method):

Questionnaire / survey 24 Interviews 12 Focus group 7 Specific measure tool 8 Direct observation 6 Incident reports 4 Text analysis 2 Appointment books 2 Others: cost analysis, participation rates, central register, theoretical framework Patient receiving improvement In community/primary care setting: Diabetics 3 Asthma 4 Patients with depression 5 Nursing/residential care 3 Osteoarthritis 1 Screening – women 1

Awaiting appointment 1 Podiatry patients 1 At risk of falls 1 At risk of DVT 1 Palliative 1 Non-specific primary care pt 1 In hospital/secondary care setting: Group defined by location:

ITU/HDU 10 Radiology 2 ‘Hospitals’, not-specified 4 Psychiatry in-patient 2 Others (1 in each): A&E, haemodialysis unit, postnatal ward, ‘bone clinic’, general medical wards, rehabilitation, oncology

Group defined by problem / procedure: Stroke 3 Having transplant 3

Psychiatry 4 Requiring ventilator 2 Having hysterectomy 2 At risk falls 2 Orthopaedic – joint 4 Others (1 in each):

MI, chronic heart failure, COPD, asthma, diabetes, requiring cardiothoracic surgery, incontinence, requiring analgesia, hyperglycaemia, sepsis, learning difficulties, depression, at risk of DVT, labour, colonic surgery, taking atypical anti-psychotics, requiring hospital discharge, palliative, counselling, requiring nasogastric tube, requiring genetic appointment, at risk of pressure ulcers

Further info given on hospitals involved:

Tertiary centre 12 Academic / teaching hospital 12 General hospital 2 Rural / community hospital 3 In-house QI expertise 7 No previous QI experience 1

- 62 -

Figure 5: – Codes for evidence used in the evaluati on synthesis

MECHANISM

Co

nst

rain

t

T1

Measurable process goals that are linked to

outcomes via evidence

T2

Changes are more likely to be sustained

by ensuring the impact of change

(data/progress) is visible

T3

Changes broken down into small-step cycles

completed in rapid succession lead to faster

sustained change

T4

Local spread to surrounding settings/context can

be achieved via key actions

1 C. Start with an easy task & easy to measure

goals (Pulcini et al, 2007)

C. Momentum sustained when cycles run

in setting patients observed in real time

in same location - ventilated pt in ICU

(Pulcini et al, 2007)

C. Collaborative that generated improvement

change ideas internally perceived as less rapid

than those with external advice (telephone

interview 15 collabarative leaders) (Wilson et

al 2003)

C. Buy-in also needed from administrators who

control resources allocation, providing cost-

effectiveness data crucial for ongoing support of

critical care project (Lipshutz et al, 2008)

2

C. Changes relating to healthcare processes

& management easier to implement than

ones directly relating to pt care (Meredith et

al, 2006)

C. Neonatal pain management project

found visibility key to maintaining

momentum (Dunbar et al, 2006)

C. US 40 hospital drug error collabarative

conducted total of 739 tests of change over

15 months over 209 'ramps', real tests of

change in 63% - rest were consensus-building

or educative (Leape 2000)

C. Lack of rigid measures for success & no. of pt

small senior leaders not sufficient evaluation

evidence of benefits for organisational leaders to

facilitate spread of improvements (Gould et al,

2007)

- 63 -

3

C. Processes of care outcomes showed more

consistent impact of paediatrics asthma

collaborative than patient-level outcomes

(Mangione-Smith et al 2005)

C. Method deemed acceptable, feasible &

efficient in creating optimal pt education

tool over 3 week using 5 PDSA cycles in

single endocrinology out-pt clinic

(Varkey et al. 2009)

C. Continuous retesting of score cared for

ICU pt transport was successful in

developing effective tool ( Esmail et al, 2006)

C. Start small again when next ward involved,

even if not changing what you're doing (Pulcini

et al, 2007)

4

C. Changes that were most successful

involved processes not people in US hosp

drug error collaborative (Leape 2000)

C. Ensure process measure data is visible

to staff, patients & relatives to generate

interest (Pulcini et al, 2007)

C. Canadian primary chronic care project

found model was both an instigating &

sustaining factor in making changes, but

other factors also critical (Green et al 2006)

C. Attempt to spread chronic pain project

improvements to other NH facilities - positive to

resources but none implemented PDSA cycles

(Buhr and White 2006)

5

C. Process measurement highly important

but difficult to implement initially (Benn et

al, 2009).

C. Real time 'data wall' ensured visibility

in ICU QI project (Krimsky et al, 2009)

C. 58 site collaborative in secondary care

patient safety US - didn't meet regularly,

approx 20% didn’t use PDSA at all. But when

did do rapid cycle tests had better outcome

(Leape, 2006)

C. Involve people in first team to spread to the

next ward (Pulcini et al, 2007)

6

C. Purpose of PDSA could be confused with

providing a research base for making

changes if evidence not provided -

inappropriate use (Bate et al, 2002)

C. if cycles are slow or no breakdown

into smaller steps then no feedback is

given and there is no stimulation to

further change (Vos et al, 2010)

C. Application of cycles took place over

several months in trainee QI project - 5

partial & overlapping cycles over 10mth

period (Tomolo et al, 2009).

C. US palliative care project found 'emerging

champions' move ahead before hard-won gains

are consolidated & spread (Gould et al, 2007)

- 64 -

7

C. Difficulty knowing when is 'enough'

evidence to warrant a change - critical care

project 2 recommendations later evidence

against emerged (Lipshutz et al, 2008)

C. regular feedback of changes to staff

associated with sustaining change

(Meredith et al, 2006)

C. On average only 3 cycles achieved over (12

month) period of UK orthopaedic

collaborative (Bate et al, 2002)

C. changes can remain stuck at small scale

implementation unless targets are set for it to

spread to a greater number of patients

(Macintosh- Murray 2007)

8

C. US state-wide patient safety initiatives

agreed on goals by consensus process with

stakeholders (Leape 2000)

C. US secondary care pain assessment

project - monthly audit too slow -

needed intense input of 2 hourly rounds

24/7 for 2 weeks to check

implementation, then to daily audits

(Gordon et al, 2008)

C. 3 cycles over 28 month period conducted

in hysterectomy infection prevention project

in US (Henry, Muriel & Thirway 2007

C. Spread beyond the QI project did not occur in

a large collaborative with Dutch mental health

care teams (Franx et al, 2009)

9

C. Aim of outpatient based clinic was to

improve pt understanding at end of visit.

Presumed to result in errors & worse

outcomes but no clear EB (Varkey et al

2009)

C. Difficulty encountered in org with

transient patient populations (Wagner et

al 2001)

C. Cycle 1. 4 months, cycle 2. 4 months, cycle

3. 2 months over 3 yr period in project to

reduce pt ID errors in US hosp, delay due to

mass retraining effort in cycle 1 & 6 month

data check after (Bittle, Charache and

Wassilchalk 2007).

C. Sites that successful in implementing &

sustaining changes were less likely to spread

changes in US mental health collaborative

(Meredith et al, 2006)

- 65 -

10

C. Process improvements eg. Reduced

waiting times/patient journey times not

considered to need evidence base (Vos et al,

2010)

C. Data collection can be difficult when

dealing with very acute conditions - does

not allow time for completion of data

tool in critical care project, but will be

done if part of standard care (Lipshutz et

al, 2008)

C. US hosp goal was to achieve a PDSA cycle

in 6 month period (previous QI took 1-2 yrs)

(Carboneau 1999)

C. Solutions to process outcomes are specific to

the local context so there will be lower levels of

exchange (Vos et al, 2010)

11

C. There can be a distinction between QI

control projects & QI learning projects.

Control goal appropriate when 'fully

understood' (Bloor 1999).

C. Patient volume, acuity at time of

implementation, & poor communication

between disciplines found to be barriers

in neonatal pain management project

(Dunbar et al, 2006)

C. US secondary care pain assessment QI

project implemented single large-scale PDSA

cycle - improvement only reached with

intense input (Gordon et al, 2008)

C. Difficulty with computerised registries limited

spread (Meredith et al, 2006)

12

C. In ICU QI project aims not only EB but

also classed as 'disciplinary' rather than

'discretionary' - aim for 100% (Krimsky et al,

2009)

C. Support & visibility of improvements

needed for at least 6-9 months to sustain

the improvement. Use shared database

with unit specific data to show trends on

units (Gordon et al, 2008).

C. Dutch project to implement database for

reporting of surgical adverse outcomes - used

one large-scale PDSA, poor uptake of new

database, ?due to IT problems & lack clarity

re public availability of data (Marang Van de

Mheen et al 2006)

C. Presentations on QI projects at hospital

teaching meetings led to approaches from other

departments with new project ideas (Tomolo et

al, 2009)

13 C. Goals should have reliable & valid

operational definitions (Pulcini et al 2007)

C. continuous data monitoring required

even when considered that changes are

embedded into normal culture (Pulcini et

C. Smoking cessation advice in podiatry care -

conducted 1 large PDSA cycle 'tackled

everything at once' over a 6 mth period,

C. Less need to spread changes in smaller sites as

most of site included already in initial

implementation (Meredith et al, 2006)

- 66 -

al, 2007) acknowledged after would be better to break

down - unclear why didn't (Gray, Eden and

Williams 2007)

14

C. Collaborative time frames often too short

for changes to processes of care to translate

into measurable outcome results (Leape

2000)

C. Once changes become routine care

need to continue monitoring & retraining

staff - must transition away from QI

experts delivering this (Liu et al, 2009)

C. Anticipated problems in applying PDSA to

larger numbers and allowing adequate time

for learning in endocrinology outpatient QI

project (Varkey et al, 2009 )

C. Acute psychiatric in-patient project stress that

widespread & enduring changes not be achieved

without broader, systemic level changes (Barry)

15

C. Short project time frame (12 month)

perceived as too short for real change in

Dutch mental health collaborative (Franx et

al, 2009)

C. Not applicable when a small change would

necessitate large-scale disruption to

supporting processes - testing bit by bit

therefore causes more disruption (Vos et al,

2010)

C. Incremental small-step change found to be

limited in scope in UK orthopaedic collaborative

- evaluation suggest combine with radical

redesign approach also (Bate et al, 2004)

16

C. Focus should be on process outcomes (eg

compliance with guideline) not clinical

outcomes as these harder to interpret - eg.

Won't account for secular trends (Pulcini et

al 2007)

C. Scale of changes achieved in UK

orthopaedic collaborative perceived to have

been limited by method used, no

'breakthrough' (Bate et al, 2004)

- 67 -

17

C. Difficulty in establishing cause & effect

relationship in UK pt safety collaborative

(Benn et al, 2009).

C. US drug error collaborative removing KCl

from wards 'unlikely to be reversed', (Leape

2000)

18

C. Small GP practice diabetes QI project -

discussing changes enough to implement -

was no need for specific implement plans /

testing to be made (Geboers et al, 1999)

19

C. Rapid cycles are not possible when focus

on processes involving a small number of

patients - will be spread out over time (Vos et

al, 2010)

20

C. Common cause of failure in US hosp drug

error collaborative was spending too long

collecting baseline data - lost momentum

(Leape 2000)

- 68 -

21

C. Short feedback cycles deemed key to staff

compliance with changes as asked to explain

failures while pt still remembered - within few

days (Nolan et al, 2005)

22

C. Discharge planning project in rural

Australia - high level of ownership slowed

development of interventions - ?implication

that were too many iterations of each stage

(Bolch et al, 2005)

23

C. "Real improvement work is often messier

than the logical sequence of the steps, rigid

conformance to steps in the model delays

testing change" (Batalden & Sholtz 1993)

24

C. Incomplete cycles can inform further

cycles, the path to improvement is rarely

linear, some cycles are less important than

others. Strict adherence to steps in sequence

inhibited natural progression of responses to

challenges /opportunities. More complex

- 69 -

modified model needed to be developed

(Tomolo et al, 2009)

- 70 -

CONTEXT

Constraint/

facilitator

T5

Frontline staff may have relevant

knowledge of changes &

implementation skills that can

influence success

T6

Engagement of frontline staff can

influence participation and impact

T7

Existing resources can

sustain PDSA/

changes

T8

The tool is can achieve practice

change/outcomes in multiple

disciplines & organisational

contexts

T9

The context/stability/

size of the organisation

can impact on PDSA

cycles/impact

1 C. Top down goal setting and

detailed intervention toolkit &

instructions to achieve favoured

by GPs in mental health

collaborative (Franx et al 2009)

C. Provide staff with visual

stimulus of room for improvement

based on baseline data (Pulcini et

al, 2007)

C. Assessment of existing

capability (support,

experience, resources, pre-

existing routine monitoring)

may be appropriate before

initiating QI project (Benn et

al, 2009).

C. US secondary care pt safety

project found engagement of

senior administrator important

for success (Leape et al 2006)

C. Large scale organisational

changes will affect staff

motivation to implement &

sustain cycles (Macintosh-

Murray, 2006)

2 C. Blending improvement work

into daily work is a skill to be

learnt - process thinking skills,

not just education on the

intervention (Macintosh-Murray

C. Neonatal pain management

project found staff buy-in difficult

to achieve - seeking staff feedback,

high visibility, present evidence

base helps (Dunbar et al, 2006)

C. Dutch mental health QI

project - lack of dedicated

professional time hindered

project (Franx et al, 2009)

C. Leaders often distracted by

other 'worthwhile goals' & fail to

lead - US palliative care project

(Gould et al 2007)

C. Downsizing & mergers

made US drug error

collaborative low priority &

slowed progress in some hosp

(Leape 2000)

- 71 -

2007)

3 C. UK A&E projects found

consultants more freq accept

PDSA approach than senior

managers (Walley and Gowland

2004)

C. Process goals measured by

documentation - staff can perceive

already doing but not

documenting, so no change to pt

outcome - low motivation

(Gordon et al, 2008)

C. Building change processes

into normal working time

does not work if this is

already saturated -

precipitation elsewhere ie

work by staff in own time

(Macintosh-Murray 2007)

C. Smaller clinics more likely to

sustain change as project more

visible & easier to manage by a

few committed people. Larger

clinics less likely to sustain

change - harder to reach critical

mass via training, dilution of

efforts, less visible changes

(Meredith et al, 2006)

C. Rapid turnover in leadership

and staff identified as barrier in

23 org diabetes collaborative

(Wagner et al, 2001)

4 C. Changes in implementation

were sourced from frontline staff

in ICU QI project (Krimsky et al

2009)

C. Resistance to implementation

from staff initially - loss of control

independent judgement by

following protocol introduced -

improved as saw clinical benefit

(Nicholls , Cullen and Halligan

2001)

C. Average 1.5 fulltime

equivalent input nursing time

spent on daily monitoring,

reviewing results, giving staff

feedback - input secondary

care cardiology ward (Nolan

et al, 2005)

C. A team from an organisation

without QI structures on site

may not be able to learn all it

needs to in whilst doing the

improvement work (Macintosh–

Murray 2007)

C. Large proportion of new

employees may lead to lack of

understanding of inter-

department dependence on

each other (Carboneau 1999)

- 72 -

5 C. Method found not to address

the 'how' of implementation -

difficulty in this area, overall

poor uptake of concept (Bate et

al, 2002)

C. UK A&E project - small

nunmber. sites kept senior control

- changes proposed large scale.

Most frontline staff took control -

these most successful. (Walley and

Gowland 2004)

C. Audit documentation of

pain reassessment QI project

US secondary care -

insufficient staff (Gordon et

al, 2008)

C. Found to be practical & useful

tool for use in emergency dept of

small community hospital with

limited resources (Warburton

2005)

C. Reliance on a few

champions to sustain changes

means changes are fragile when

instability of context occurs

(Macintosh-Murray 2007)

6 C. Specific strategies & templates

of tools needed to implement

recommendations in US

secondary care project - but staff

still found difficult & was huge

task itself (Leape 2000)

C. Active physician involvement

crucial in general medicine access

& quality project (Gitomer, 2005)

C. Discharge planning

project rural Australia limited

by poor IT support, poor

admin support, staff

shortages (Bolch et al 2005)

C. Dominant hierarchical culture

negatively & development

culture positively associated with

motivation to implement PDSA

(Lin et al, 2005)

C. Departure of champion

from critical care project -

realised should have acted

quickly to replace - this with

staff turnover affected

motivation (Lipshutz et al,

2008)

7 C. Intensive technical support

team in US depression QI

project - in depth knowledge of

scientific literature & clinical

organisation structure deemed to

be required (Liuet al, 2009)

C. High level of ownership &

interest in discharge planning

project rural Australia - but slowed

down development as 'everyone

had their say' (Bolch et al, 2005)

C. Limited staff resources

assessed as factor explaining

clinics difficulty in

implementing PDSA in HIV

community clinic, but Extra

work ass with change

became 'part of normal

C. PDSA cycles hard to apply &

did not fit into existing culture of

primary care profession in Dutch

mental health collaborative - not

used to discussing care processes

(Franx et al, 2009)

C. context of teaching hospital

more difficult as house officers

& med students always rotating

- needed monthly input for US

secondary care project on MI

(Nolan et al, 2005)

- 73 -

routine' & improved

efficiency reduced their

workload (Fremont et

al,2006)

8 C. Kilo suggests BTS method

use external ideas for change

which are then adapted locally

(Kilo 1998)

C. US palliative care project chose

not to 'dictate changes' - effective

in building ownership (downside -

multiplicity of effort) (Gould et al,

2007)

C. Difficulty persuading

some staff of importance of

chronic pain assessment

project due to other

demands on their time (Buhr

and White 2006)

C. Trainee educational initiatives

struggle to take off if the clinical

setting staff don not have an

interest in QI - needed

'champions in the department -

PDSA model did not translate to

the application of rapid cycle

change in the clinical context in a

medical trainee QI project

(Tomolo et al, 2009).

C. Contexts with high

proportion of part-time staff

will be more difficult to

implement (Macintosh- Murray

2006)

9 C. Teams in US hosp drug error

collaborative used established

changes - did not dev own

changes from human factors

concepts provided as starting

points for doing so (Leape 2000)

C. Involvement with UK patient

safety collaborative found to create

sense of ownership (Benn et al,

2009)

C. Measures that are not

gathered on existing hospital

IT systems require significant

staff time to collect data

(Vos et al, 2010)

C. The PDSA model was found

to shift the organisational culture

in UK pt safety collaborative to

that more favourable to QI

(Benn et al, 2009).

C. Difficulty in retaining staff

commonly stated (65% sites) as

barrier to change in US mental

health collaborative (Meredith

et al, 2006)

- 74 -

10 C. Mental health collaborative in

US relied on implementation of

change guidance from external

support team (Meredith et al

2006)

C. UK orthopaedic collaborative

'clinicians did not appear to wish to

lead' - resulted in project manager

led collaborative, local ownership

not achieved at scale anticipated

(Bate et al, 2002)

C. ICT component viewed as

critical to success in

Canadian chronic disease

management primary care

project (Green et al, 2006)

C. May have been some evidence

of implicit culture change as

result of UK orthopaedic

collaborative but not conclusive

(Bate et al, 2009)

11 C. Most collaboratives used

outside expert groups to suggest

improvements rather than

participants themselves

(telephone interview with 15

collaborative leaders worldwide)

(Wilson et al 2003)

C. PDSA was lowest rated

component (of 17) of UK

orthopaedic collaborative - not

seen as the main driver for change.

Views were mixed but most not

convinced of their value (Bate et al,

2002)

C. Administrative & IT

support needed for US

diabetes collaboratrive

(Wagner et al, 2001)

C. Poor leadership support

significantly affected projects in

US mental health collaborative

(Meredith et al, 2006)

12 C. Genetic evaluation project felt

that lack of quality improvement

expertise contributed to slower

implementation (Moeschler et al

2009)

C. PDSA methodology rated as

highly important component of

UK hosp patient safety

collaborative - no sig diff in rating

between frontline staff & board

level staff (Benn et al, 2009)

C. Data collection systems

often newly created for UK

patient safety collaborative

(Benn et al, 2009)

C. Senior leadership support

perceived as crucial, except in

contexts with teams in self-

contained units where already

working more independently eg.

Neonatal ICU (Wilson et al,

2003)

- 75 -

13 C. Concept of 'small tests of

change' difficult for most

healthcare workers to learn, lack

knowledge as to how to collect

performance data, 1/2 day

instruction insufficient (Leape

2000)

C. Physician / nurse resistance

could be 'formidable barrier' in US

hosp drug error collaborative –

possibly due to anxiety / including

workload (Leape 2000)

C. Integrating process

changes into electronic

medical record is high cost,

but support sustainability &

spread once implemented

(Liu et al, 2009)

C. Culture of clinician autonomy

need constrained in UK patient

safety collaborative (Benn et al,

2009)

14 C. improvement targets set by

individual teams in US palliative

care project - inc ownership but

difficulty identifying appropriate

measures (Gould et al, 2007)

C. primary care provider resistance

to change encountered in 76% of

sites involved in US mental health

collaborative, but associated with

more change activity (Meredith et

al, 2006)

C. Estimated cost of

newborn preventative service

project was $29000 per

hospital (excl staff time in

doing QI work) (Mercier et

al, 2007)

C. Patient volume, acuity at time

of implementation, & poor

communication between

disciplines found to be barriers

in neonatal pain management

project (Dunbar et al, 2006)

15 C. Teams in US hosp drug error

collaborative found defining

measures & collecting data

difficult (Leape 2000)

C. Motivation is complex & not

easily predicted by contextual

factors (Lin et al, 2005)

C. cost of mental health

collaborative ranged $30,000-

252,000 in US mental health

collaborative - mainly staff

time costs (Meredith et al, et

al, 2006)

C. Virtual project environment

created by website as main

source of information not

helpful to professions unused to

this format of communication

(Franx et al, 2009)

- 76 -

16 C. Learning sessions shortened

in HIV collaborative - found

that more than half of teams

struggled with basic concepts at

end of session, needed to

practice in further LS (Fremont)

C. Acute psychiatric inpatient

project UK found teams joining

later did so out of choice & were

'best performers', less success with

those 'volunteered' by trust (Barry

et al 2006)

C. Complex organisational

change required dedicated

staff time to test the changes,

measure the impact, refine

the approach, & spread the

changes - sig funding (Leape

2000)

C. Collaboratives work better in

organisations combining intra-

org features (such as PDSA) with

inter-org features (such as

external reporting) (Nembhard

2009)

17 C, US mixed primary &

secondary palliative care project -

found between learning session

coaching needed re PDSA cycles.

Many teams came to report

waves to consolidate (Gould et al

2007)

C. Process outcomes that affect

several departments - resistance

more likely as motivation focussed

on their area of work. Political

solution may be needed (Vo et al,

2010s)

C. Strong local team co-

ordinator needed in Dutch

mental health QI project -

employed by governmental

health department - external

financial support needed

(Franx et al, 2009)

C. All trusts valued PDSA, but

was sig diff variation in how

much in UK patient safety

collaborative - may be variability

in appropriateness for diff org

(no info on trust characteristics)

(Benn et al. 2009)

18 C. If cross-department

collaboration needed the

supporting group might need to

make initial networking contacts

(Tomolo et al, 2009)

C. Face-to-face training

preferred to long distance

coaching on telephone /

internet in US palliative care

project (Gould et al, 2007)

C. Greater number. of changes

observed in communities with

higher incomes, but no

association between site

characteristic & sustaining &

spreading change (Meredith et

- 77 -

al, 2006)

19 C. Team members from all sites &

disciplines needed for Canadian

ICU transport project to achieve

'early buy-in' (Esmail et al, 2006)

C. QI in a large highly

systemetised organisation

may be more costly than in

small independent practice

(Liu et al, 2009)

20 C. Ownership & interest grew after

initiation by an 'early adopter'

respected by staff (Pulcin et al,

2007i)

C. State-wide pt safety

project in US - 8 hospitals

didn't participate - unable to

gather resources & personnel

for the project (Leape 2000)

21 C. Improving chronic illness care

collaborative in US 43

organisations - high staff

motivation re PDSA , lower with

clinical staff especially depression -

perception of supervisor support

needed for staff to perceive it as a

- 78 -

useful tool (Lin et al 2005)

22 C. Empowerment associated with

increased scrutiny. Staff ambivalent

as have to accept greater

responsibility & 'blame'. Managers

lose security (Bloor 1999 - review)

23 C. motivation high in UK A&E

teams where staff empowered to

make changes - can became threat

to senior management & resistance

from this group instead. Need to

be consistent with overall aims of

organisation - clear at start on

boundaries of this if control given

to frontline staff (Walley and

Gowland 2004)

- 79 -

24 C. Use of national experts helped

motivate staff in US secondary care

project on 58 sites (Leape 2006)

25 C. Learning sessions rated as

essential to staff motivation in UK

pt safety collaborative (Benn et al,

2009)

26 C. Frontline doctor financial

incentives critical to their

participation in Canadian primary

care collaborative (Green et al

2006)

27 C. Focus on individual patients can

increase motivation due to strong

identification of frontline staff with

individual patient stories

(Macintosh- Murray 2007)

28 C. Small changes fuelled

excitement & enthusiasm for larger

changes in US secondary care

- 80 -

patient safety initiatives (though

still only partial implementation)

(Leape 2000)

29 C. Smaller changes can lower

expectations of benefits of a

change and reduce motivation for

involvement (Vos et al, 2010)

30 C. staff compliance biggest barrier

to ventilated pt initiative -

overcome by demonstrating

positive pt outcomes (visibility)

(Hampton et al, 2005)

31 C. Reporting could be seen as

burdonsome, demoralising,

confusing, and of little meaning

when results shown 'too early', and

therefore may not be fully

implemented (Bate et al, 2002)

- 81 -

32 C. Staff awareness of continued

lack of improvement in aims

despite efforts led to

discouragement - decreased

motivation (Warburton 2005)

33 C. Perceived early failure of an

initiative should not be allowed to

stop - neonatal pain management

project found initiative viewed as

failure actually successful on

follow-up data (Dunbar et al 2006)

34 C. Creating competition between

departments & celebrating even

small successes can create

enthusiasm in critical care project

(Lipshutz et al ,2008)

- 82 -

Figure 6 – Template: Assessing the applicability o f using PDSA to

achieve quality improvement in your context & objec tives

Key

Dimension Facilitating factors

Tick if

applies Constraining factors

Tick if

applies

1 Overall aim translates easily into

measurable goals

Overall aim does not translate

appropriately into measurable

goals

2

There is a strong and well-

established evidence base for the

overall aims and measured goals

The evidence-base is weak or

frequently being updated

3

The choice of goals is informed

by previous experience of their

elsewhere

Measured goals are not based on

experience of their use elsewhere

to meet the aim

4 Measured goals can have a 100%

target ‘every patient, every time’

Goals are discretionary – there is

a need to allow clinical judgement

5 Measured goals can be tightly

defined

The measured goals are open to

interpretation

6

There is a clear relationship

between cause and effect in the

achievement of measured goals

Confounders & secular trends are

expected to have an impact on

the progress towards measured

goals

1.1

Imp

rove

men

t go

als

& e

vid

ence

bas

e

7

Clinical benefit is expected to be

demonstrated within the time

frame of the project

Impact on clinical benefit unlikely

to be demonstrated in time frame

of project

1 The cycles are being carried out in

a defined geographical area

The cycles are being carried out

over dispersed locations

1.2.

1 Sust

ain

ing

-

visi

bili

ty

2 A data-wall to present progress

towards measured goals can be

Staff are dispersed and so a

central visual display of data is

- 83 -

located somewhere that is visible

to all relevant staff

difficult to achieve

3

Cycles are planned to be

conducted rapidly (within working

day of a staff member)

Cycles are expected to take place

over weeks or months

4

Staff are given feedback on

progress towards measured goals

in realtime or a very short

timeframe

There is a delay before data is

feedback is given to staff

5

Cycles take place in a location

with a relatively stable patient

population

Cycles take place in a setting with

a fast transition of patients

6

The target population have

chronic problems allowing time

for documentation and collection

of progress towards measured

goals

The target patient population are

those with acute problems

needing immediate action

meaning there is little time to

document achievement of goals

7

Ongoing monitoring of goals is

planned to continue and remain

visible beyond the testing phase

There are no plans for ongoing

monitoring of goals when the

changes become part of routine

care

1

External input is available to

advise on change ideas that might

achieve goals, and so cycles of

tests can start quickly

Staff need to generate the change

ideas themselves to achieve the

goals

2 Goals can be broken into small

steps

It is difficult to break the goals

down into small steps

1.2.

2 Sust

ain

ing

- si

ze &

sp

eed

of

cycl

es

3

Opportunities for conducting

cycles are expected to arise

regularly (at least daily)

The aim applies to a patient

group or clinical situation seen

only occasionally

- 84 -

4

The measures remain relatively

stable unless an intentional

change to the system is initiated

There is significant daily or

seasonal variation in the

measured goals and so baseline

trends must be established over a

prolonged time frame

5

The measurement of the goal

(study) can be carried out at the

time the change is made (act)

There is a delay between when

the action is taken and measuring

the impact of that change

6

There is an ability to take decisive

action to continue cycles when

there is a spectrum of opinion

There is an expectation that all

the staff a change may affect will

be consulted at each stage of the

cycle

7

A flexible approach is taken to the

completion and expansion of

cycles

There is an expectation that all

cycles must be completed fully

and expanded sequentially

1

Senior management are agreeable

to promote the spread the

changes and provide the

resources to do so

Senior management are not

convinced of the benefit of the

changes and so are resistant to

spreading the changes

2

Staff are motivated to restart with

small tests of change and adapt

the change ideas to the new

setting

There is an expectation that the

improvements will spread directly

without any need to start small

tests again in the new setting

3

Staff from the first area of

implementation can help start off

cycles and give advice in the new

setting

Work roles are not flexible, so

staff are unable to learn directly

from those involved at the first

place of implementation

1.3

Sp

read

ing

the

chan

ges

4 Targets to achieve spread of

improvements to an incrementally

increasing number of patients can

No intentional targets to spread

changes are set. Spread is

anticipated to occur without

- 85 -

be set additional input once successful

changes have been implemented

in one area

5

Systems used to help implement

changes or monitor goals are

similar in the area you wish to

spread the improvement to

There are different systems (such

as IT infrastructure, working

patterns, communication systems)

in the area you wish to spread the

improvement to

6

Places of interaction between staff

from different disciplines or

locations already take place on a

regular basis e.g. educational

meetings

The are no pre-established

connections with the with area or

group you wish to spread the

improvements to

7

Spread to a wider area can be

achieved without the need for

broader systematic changes to the

organisation

Systematic change in the

organisation (that is not amenable

to PDSA) is needed to spread the

changes to other areas

1

Staff are able to decide on the

overall aim and set measurable

targets based on their knowledge

of the evidence base

Staff are unfamiliar with the

scientific literature surrounding

their overall aim, and are unaware

of goals which may be used to

measure improvements

2

Staff involved are aware of the

processes & systems that support

their clinical role

Staff have little awareness of the

processes & systems that support

their clinical role

2.1.

1 Sta

ff -

Ski

lls &

kn

ow

led

ge

3

Staff are able to translate their

knowledge of the scientific

literature and processes of care

into specific ideas to implement

changes to achieve the goals set

Staff are likely to struggle to

identify change ideas that they

can implement in their context

without specific training

- 86 -

4

Staff have the skills & experience

needed to collect, collate, present,

and interpret both baseline data

and ongoing monitoring data

Staff have little or no experience

of data collection, collation,

presentation & interpretation

1

The evidence base for the goals

set and the deficiencies at baseline

are clearly presented

The evidence base is not

communicated clearly to staff and

there is no perception of any

need to improve

2

Staff perceive a clear link between

the measured goal and clinical

benefit, and are confident in the

evidence-base

Staff perceive that meeting the

goals will only achieve better

documentation rather than better

clinical care, and are concerned

they will lose the ability to

exercise clinical judgement

3

Frontline staff are involved in all

stages of the deciding the aims &

measurable targets set, and all

stages of the PDSA cycle

The overall aim, measured goals,

targets, and the change ideas are

given to staff by an external

group or a management team

4 Staff are able to take part in the

initiative on a voluntary basis

Staff involvement is coerced or

compulsory

5

The team involved in conducting

cycles come from the same site or

discipline

The cycles involve several

departments and disciplines

making and testing changes

2.1.

2 Sta

ff -

en

gage

men

t

6

There is representation from each

area (if the goal involves several

sites or disciplines) on the team

deciding on changes and

monitoring goals

Some of the sites or teams

involved in implementing

changes are not represented in

the process of setting goals and

monitoring goals

- 87 -

7

‘Early adopters’ can be identified

who are respected by other

frontline staff & can help drive

the cycle forward

There are no obvious frontline

staff members to champion the

changes who have the respect of

their colleagues and who are

enthusiastic about using PDSA

8

Staff are confident of senior

support and are given clear

guidance on the responsibility

they have for making changes

Staff are concerned they will be

held accountable by their seniors

if the changes they introduce are

unsuccessful

9

Managers are willing to allow

frontline staff to take ownership

of changing processes of care

provision and agree clear

boundaries with staff

Managers perceive frontline staff

shaping the organisation of care

delivery as a threat to their role,

resist allowing staff more

responsibility in this area and fail

to make boundaries explicit

10

External expertise is available and

is viewed as a positive support by

staff

External expertise is unavailable,

or is perceived as having a

negative impact

11 Financial incentives can be

provided when goals are achieved

There is no ability to provide

financial incentives when goals

are met

12

Staff can see clear benefit for

individual patients which

encourages them to expand the

changes to a larger group

The changes made in each cycle

are perceived by to be too small

to result in significant clinical

benefit and motivation is lost

13

The cycles can be completed

rapidly and awareness of positive

outcomes helps maintain staff

momentum

The cycles are completed slowly

so staff are less aware of the

outcomes of the changes and lose

enthusiasm

- 88 -

14

The progress towards achieving

the measured goals can be clearly

displayed to all frontline staff

implementing the changes

It is difficult to display the

progress towards the goal visually

on a regular basis or in a location

that is seen by all frontline staff

involved

15

Successes can be acknowledged

and celebrated, and a sense of

competition between teams

implementing the changes can be

created

There is no acknowledgement of

success achieving goals and staff

are unaware of the performance

of other teams

1

A full assessment of both the

current capacity and the financial

and resource implications can be

carried out before the start of the

project

The capacity within the system

currently is unknown and there is

no information or ability to carry

out an assessment of the resource

implications

2

There is capacity to reallocate

staff time to allow involvement in

planning and testing changes

Staff time is fully saturated

already, with no ability to drop

any activities to create space

3 Monitoring of goals makes use of

data gathered routinely already

Monitoring of goals requires a

new system of data collection

4

There is capacity to invest in IT

systems that may be needed to

monitor the progress towards the

measured goals

There are no resources available

to update or create IT systems to

assist in data collection and

monitoring

2.2.

1 O

rgan

isat

ion

– r

eso

urc

es

5

Finances for co-ordination, staff

training, additional staff time,

implementation costs, and

monitoring costs are available

There is no provision for

additional finances to cover the

costs of planning, implementing,

sustaining, and spreading the

changes

- 89 -

1

Senior management make the

improvement process a key

priority for the organisation

The organisation has other

priorities that are unrelated to

quality improvement

2

The organisation is large enough

to provide some internal expertise

and experience, but small enough

to allow improvements in one

area to be visible to others and

allow spread

The organisation is either too

small to provide internal support

to staff, or is so large that the

impact of changes are diluted

within the larger system and are

difficult to perceive

3

There is already a strong

developmental culture in the

organisation, that emphasises

continuous improvement and the

involvement of all staff in that

process

There is a strong hierarchical

culture in the organisation with

changes initiated in a top-down

approach and a history of slow

adaptation to change

4

There is a strong perception of

senior management support for

frontline staff

There is a lack of senior

management support, or senior

management support is not

perceived by the staff

5

There is a strong emphasis on

evidence based medicine and the

need to follow best practice

guidelines within the organisation

There is a strong culture of

clinician autonomy with the

organisation, with a greater

emphasis on clinical judgement

than the evidence base

2.2.

2 O

rgan

isat

ion

-

clim

ate

6

There are clear lines of

communication established

already between the sites and

disciplines involved in the project

Communication between the

groups involved in the

improvement project is poor or

non-existent

- 90 -

7

The organisation has a culture of

sharing information and learning

with other organisations, and has

established relationships and

systems to support this

There is little experience of

sharing learning with other

organisations, and there is no

relational or structural basis on

which to do this

1

There has been recent

organisational stability and no

significant changes are anticipated

There have been significant

structural changes to the

organisation, either recently or

ongoing

2.2.

3 O

rgan

isat

ion

- s

tab

ility

2

There is a high proportion of staff

who have worked in the site for a

long time and are unlikely to

move on, the staff have regular

and predictable working patterns

There is high staff turnover either

due to a high proportion of junior

clinical staff, high proportion of

part-time or shift workers, or

difficulty retaining staff

- 91 -

- 92 -