The How-to guide for measurement for improvement · always work first time. The testing process not...

The How-to guide for

measurement forimprovement

2

Contents

Introduction 3

Part 1: What is measurement for improvement?The Model for Improvement 4The 3 reasons for measurement 6Making measures meaningful 7The different types of measures 7Ratios and percentages 8

Part 2: How do I measure for improvement?Top tips 10The 7 steps to take 10

AppendicesAppendix 1: Measures template 20Appendix 2: Review meeting template 21Appendix 3: Expected number of runs 22Acknowledgements 23

3

To demonstrate if changes arereally improvement, you needthe ability to test changes andmeasure the impactsuccessfully. This is essential forany area that wants tocontinuously improve safety. To do this you may only need afew specific measures linked toclear objectives to demonstratethat changes are going in theright direction.

This guide is designed to help you tothis in your improvement projects. Itis in two parts.

Part 1 explains what measurement forimprovement is and how it differsfrom other sorts of measurement thatyou will have come across.

Part 2 talks you through the processof collecting, analysing and reviewingdata. If you are familiar with theModel for Improvement and how touse it, you can skip Part 1 and gostraight to Part 2.

The How-to guide for measurement for improvement

“All improvementwill requirechange, but not allchange will resultin improvement”G. Langley et al., The Improvement Guide, 1996

Introduction

4

The Model for ImprovementThe basis of measurement for improvement falls naturally out of the Model for Improvement. The Model forImprovement was developed by Associates for Process Improvement (USA, available at www.apiweb.org). It provides aframework around which to structure improvement activity to ensure the best chance of achieving your goals andwider adoption of ideas. The model is based on three key questions used in conjunction with small scale testing:

The document focuses on measurement,which is fundamental in answeringthe second question: “How do weknow a change is an improvement?”but all parts of the model areinextricably linked. An overview of allparts of the model can be found inthe accompanying Campaigndocument “The quick guide toimplementing improvement” (available at www.patientsafetyfirst.nhs.uk).

Small tests of changes that you hopewill have an impact on your rate ofharm need to be measured well. Thispart of the model is an iterative wayas improvements/measures do notalways work first time. The testingprocess not only tells you how wellthe changes are working but howgood your measure and its collectionprocess is. You may find after a testthat your method of sampling or datacollection needs refining.

Implementing changes takes time andmoney so it’s important to testchanges and measures on a smallscale first because:

• It involves less time, money and risk

• The process is a powerful tool forlearning which ones work andwhich ones don’t. How many ofyou have ever designed aquestionnaire or an audit form onlyto realise that it didn’t give you theinformation you needed? This mayhave been because the informationyou requested wasn’t quite right,the way people interpreted thequestions or simply that the formitself wasn’t clear enough for theperson to complete withoutguidance


Part 1: What is measurement for Improvement?

What are we trying to achieve? Constructing a clear aim statement

Choosing the right measures and planningfor how you will collect the right information

Coming up with ideas on how toimprove the current state

How will we know that a change is an improvement?

What changes can we make that will result in an improvement?

Testing them using PDSA cyclesAct Plan

Study Do

5

• It is safer and less disruptive forpatients and staff. You get an ideaof the impact on a small scale firstand work to smooth out theproblems before spreading thechanges more widely

• Where people have been involvedin testing and developing the ideas,there is often less resistance.

Measurement for safety improvementdoes not have to be complicated.Tracking a few measures over timeand presenting the information well isfundamental to developing a changethat works well and can be spread.

Measurement can show us a numberof important pieces of information:

• how well our current process isperforming

• whether we have reached an aim

• how much variation is in ourdata/process

• small test of change

• whether the changes have resultedin improvement

• whether a change has beensustained.

The 3 reasons for measurementThere are three main reasons why we measure: research, judgement andimprovement. Understanding what you are measuring and why is vital as itdetermines how you approach the measurement process

Adapted from: “The Three Faces of Performance Management: Improvement, Accountability andResearch.” Solberg, Leif I., Mosser, Gordon and McDonald, Susan Journal on Quality Improvement.March 1997, Vol23, No. 3.

Clinical colleagues are often more familiar and comfortable with measurementfor research on a large scale with a fixed hypothesis to reduce unwantedvariation. Health service managers and those in more strategic roles may bemore familiar with measurement for judgement as a way of understanding alevel of performance. Measuring for improvement is different. The concept ofsequential testing means that there needs to be willingness to frequentlychange the hypothesis (as you learn more with each test) and an acceptanceof ‘just enough’ data, working with data and information that is ‘goodenough’ rather than perfect. Measurement for improvement does not seek toprove or disprove whether clinical interventions work – it seeks to answer thequestion “how do we make it work here?”

Characteristic Judgement Research Improvement

Aim Achievement New knowledge Improvement of target of service

Testing strategy No tests One large, Sequential, blind test observable tests

Sample size Obtain 100% ‘Just in case’ ‘Just enough’ dataof available, data small, sequential relevant data samples

Hypothesis No hypothesis Fixed hypothesis Hypothesis flexible;changes as learning

takes place

Variation Adjust measures to Design to eliminate Accept consistentreduce variation unwanted variation variation

Determining if No change Statistical tests Run chart orchange is an focus (t-test, F-test, statistical process improvement chi-square, p-values) control (SPC) charts


“Seekusefulness, notperfection, in themeasurement”Nelson et al., Building Measurementand Data Collection into Medical Practice;Annals of Internal Medicine; 15 March1998; Volume 128 Issue 6; Pages 460-466.

6

Making measures moremeaningfulSometimes we ask staff to spend timeand energy testing and implementingchanges that they perceive to haveonly a small impact. It isunderstandable that teams prefer tolook for the ‘big win’; the one changethat will get them where they wantto be. Driver diagrams can be helpfulin showing these teams how thework they are doing not only links tothe organisation’s strategic aims buthow all of the smaller changes addup to achieve it. This can helpmotivate teams by demonstrating theimportance of their role in improvingthe safety of their patients.

Each of the ‘How to Guides’ createdfor the Campaign interventionscontains a driver diagram todemonstrate how the elements of theintervention link to achieving the aim.

The different types ofmeasuresIt can be helpful when you haveselected a range of measures tocheck what type of question they areaddressing. Are they telling yousomething about what happened tothe patient? Or are they telling yousomething about the process of care?Knowing that you have selected all ofone type might cause you to thinkagain about your selections. Thethree types we use in improvementwork are called outcome, process andbalancing measures.

Outcome measures reflect theimpact on the patient and show theend result of your improvement work.Examples within the safety arenawould be the rate of MRSA or thenumber of surgical site infectioncases.

Process measures reflect the wayyour systems and processes work todeliver the outcome you want.Examples within the safety arenawould be % compliance with handwashing or the % of patients whoreceived on time prophylacticantibiotics.

Balancing measures reflect whatmay be happening elsewhere in thesystem as a result of the change. This impact may be positive ornegative. For example if you want toknow what is happening to your postoperative readmission rate. If this hasincreased then you might want toquestion whether, on balance, youare right to continue with thechanges or not. Listening to thesceptics can sometimes alert us torelevant balancing measures. Whenpresented with change, people canbe heard to say things like “if youchange this, it will affect that.”Picking up on the ‘thats’ can lead to auseful balancing measure.

Of course our main purpose is to seeoutcomes improving but how can wedo that? Reliable processes are aproven way to better outcomes. So we need to improve our processesfirst to make them extremely reliablethen improved outcomes will follow.Therefore, we should have bothprocess and outcome measures andwhere necessary a balancingmeasure.


Good measuresare linked toyour aim - theyreflect how theaim is achieved.

7

Rations and percentagesHaving decided on a topic for ameasure, for example surgical siteinfections, we now need to decidehow it should be expressed. Do wewant to express it as a percentage ofpatients seen, the rate per 1000patients or simply as a count (thenumber of infections)? What followsare some guidelines to help youdecide which option to use.

Use Counts when the targetpopulation (for example number ofpatients on a ward) does not changemuch. It has the advantage ofsimplicity but it can be difficult tocompare with others or even withyourself over time. So, expressing ourmeasure as the number of infectionsper month is fine as long as thepatient population we are treatingremains reasonably constant over time.

Use Ratios or rates when you wantto relate the infections to some otherfactor such as patients or bed days. If your target population numbers arequite variable a simple count is notsufficient without the context. In thiscase the measure would be infectionsper 100 patients or infections per1000 bed days. Now a ratio is simplyone number divided by another(infections divided by patients) andstatisticians use specific words todescribe the two numbers thatcomprise a ratio. They would call theinfections number the ‘numerator’and the patients number the‘denominator’.

Use Percentages when you want tomake your focus more specific. For example, if you want to learnabout patient falls in yourorganisation is your focus on theoccurrence of falls or the result offalls in terms of patient harm? If yourfocus is falls then you would measurethis as a rate or ratio. If your focus ison what has happened to the patientyou might select a measure as the %of patients who were harmed by theirfall. In our infection example, themeasure would be percentage ofpatients who had a surgical siteinfection that met your pre determinedcriteria for infection. In both examplesyou would probably be gathering thesame information - just expressing it adifferent way. Notice that we havemoved away from counting infectionsnow to counting patients who had aninfection to allow us to frame themeasure as a percentage - if we werecounting the former we could notexpress this as a percentage becausesome patients may already have morethan one and statistically that meansit would be possible to end up with anumber that is greater than 100%!

Use ‘time between’ or ‘casesbetween’ when you are tracking a‘rare’ event, say one that occurs lessthan once a week on average. If surgical infections occur thisinfrequently then measures expressedas rates or percentages become lessuseful. A count of monthy infectionsmight look something like:2,3,3,3,2,3,4,3,3,2,2,4. A change of1 infection is quite a percentage shiftand therefore our run chart wouldvary wildly but based only on 1 moreor less infection. Clearly this is notvery helpful. In this case express themeasure as the number of cases sincethe last infection. We might now getvalues such as 75, 57, 82, 34 casesbetween infections. When chartedthis gives us something more usefulto look at and it is not affected by the‘small number’ problem that canimpair rates and percentages.


8

Top tipsKey things to remember whenstarting to measure:

• Seek usefulness not perfection –measurement should be used tofocus and speed improvement upnot to slow things down

• Measure the minimum. Only collectwhat you need; there may be otherinformation out there but the aim isto keep things as simple as possible

• Remember the goal is improvementand not a new measurementsystem. It’s easy to get sidetrackedinto improving data quality,especially if you are confrontedwith challenges on the credibility ofthe data (more commonly fromcolleagues who may tend to trusmore rigorous research data) – just ensure it’s ‘good enough’

• Aim to make measurement part ofthe daily routine. Where possibleuse forms or charts that are alreadyroutinely used or addrecording/collection process to onethat is already in place. Thisminimises the burden on staff andalso maximises the chances of itbeing done reliably.

The 7 steps to take

Step 1 Decide your aim

Step 2 Choose your measures

Step 3 Confirm how to collect your data

Step 4 Collect your baseline data

Step 5 Analyse and present your data

Step 6 Meet to decide what it is telling you


Part 2: How do I measure for improvement?

6 Reviewmeasures

4 Collect data

5 Analyse &present

7 Repeat steps 4-6

3 Confirm collection

2 Choose measures

1 Decide aim

9

Step 1 - Decide your aimMore information on setting an aim iscontained in the accompanyingCampaign document ‘The quickguide to implementing improvement’(Model for improvement section)available at:www.patientsafetyfirst.nhs.uk.

The key points to remember aboutaim setting are:

• Those involved in making thechanges should be able tounderstand (and translate) theproject work to the strategic goals.

• The aim statement should beunambiguous clear, specific,numerical, measurable – it MUSTstate “How much’ and ‘By when’.

If the aim seems quite a long wayfrom where your current performancelevel (baseline) is, it is advisable tobreak it down into statements thatmake it seem more achievable e.g.achieving 80% compliance within 1 year but improving this to 95%within 18 months.

Step 2 - Choose your measuresEach of the Campaign interventionHow to Guides gives you an overviewof the recommended measures foreach intervention as well assuggestions for optional measures.You can also view a complete list ofall the Campaign measures in‘Campaign Measures Definitions.doc’(available atwww.patientsafetyfirst.nhs.uk). TheCampaign’s Extranet site also allowsyou to create your own custommeasures so that you can choosemeasures that you feel are importantto you locally. Appendix 1 contains atemplate that helps you define yourown measures (also available atwww. patientsafety first.nhs.uk)

Step 3 - Confirm how you willcollect your dataUse the measurement template tohelp you work through this step. Youwill need to identify the date youneed and where it comes from.Sometimes the data will be alreadycollected but often you may need toset about collecting it yourself. Theprocess of working this out helps youto define exactly what it is you aremeasuring and sometimes you willfind that it might be so complex thatyou need to rethink what the bestmeasure is to ensure the data iscollected reliably. It also can help youadd details to your aim statementsuch as what the pilot population is ifyou are using one.

Steps 1 to 3 - Getting yourself ready


6 Reviewmeasures

4 Collect data

5 Analyse &present

7 Repeat steps 4-6


2 Choose measures

1 Decide aim

6 Reviewmeasures

4 Collect data

5 Analyse &present

7 Repeat steps 4-6


2 Choose measures

1 Decide aim

6 Reviewmeasures

4 Collect data

5 Analyse &present

7 Repeat steps 4-6


2 Choose measures

1 Decide aim

10

Operational definitionsMeasures nearly always require somekind of operational definition. Thismeans specifying exactly what someterms means and applying thisdefinition consistently. For example ifyou want to know how manypatients had ventilator associatedpneumonia (VAP) you need to beexplicit about what constitutes a VAPand what does not. Sometimes thesedefinitions can be very difficult to getconsensus on. One hospital spent ayear having discussions about how todefine a VAP! If there is disagreementfind a few examples from otherhospitals and get the team to pickone and start using it. The team canthen spend as long as they chooseover deciding how they would like todefine a VAP but in the meantime thework can progress

The most important thing is that onceyou have established these definitions,they are applied consistently. If youdo change them for any reason, youwill need to annotate your run chartstating what you changed aboutwhat you measure or the way thatyou measure it.

SamplingWhen do we track 100% andwhen do we track a sample?If your numbers are small enoughthat you can track 100% without toomuch trouble then you should do it. Ifthis is not feasible then you shoulduse a sample. For the Campaignmeasures that require you to select asample, 10 is sufficient. This is alsothe sample size use for ‘ProductiveWard’ measures.

For example, when measuringprogress in reducing VAPs then thenumbe of VAPs is not difficult tomonitor so you would count all casesof VAP that occur. When auditingcompliance with the use of theventilator care bundle you would usea weekly sample of 10.

How do we select the sample?You need to choose a sample that isrepresentative of the overallpopulation that you are measuring.This is so that you do notinadvertently introduce a bias intoyour results. For example, if you areauditing to see how many patientshave had all their physiologicalobservations completed then youwould choose any 10 patients on theward / unit at random. However, ifyou are auditing the number ofpatients who were given fumazenilyou would need to select a samplefrom a patient group who are likely tohave received midazolam (such asfrom a day surgery unit).

It is difficult to ensure a truly randomsample if you are making the choicesmanually. Almost inevitably some biascan creep into those choices unlessyou are very careful. One way youcan avoid this outcome is to use arandom number generator such asthe one contained in Excel. Numberall the patients you want to selectfrom and then use the Excel featureto ‘select’ a sample. If you are notfamiliar with how to do this in Excel,contact your Information Departmentfor assistance.


11


Measurement itself is a process. In itssimplest form it consists of threestages. First you collect some data,then you analyse it and present in anappropriate way to convert it intouseful information and finally youreview your information to see whatdecisions you need to make. TheCollect-Analyse-Review or CAR cyclethen starts all over again.

Step 4 - Collect your baseline dataYou will need to know your baselinebefore you can track the progress ofyour goal against it. By starting yourmeasurement and plotting points youwill be able to create your baseline.

To create a baseline or identify atrend using a run chart, about 25data points are ideal. However, 20data points will provide a robustrepresentation. One way to get morepoints is to measure more frequently.The Campaign measures have beenset up on the assumption of monthlyreporting. Obviously to get a robustbaseline means you will needbetween 1 and 2 years of monthlydata. This is fine if historic data isavailable for you to use.

Often the data you need to measurethough is not being collected. If soyou should start collecting your datastraight away., But you do not haveto wait to start testing small changes.They will not affect your overallsituation so you can be doing thosewhile creating your baseline.

Step 5 - Analyse and present your dataUse the ExtranetThe Extranet is a web-based reportingtool set up especially by the Institutefor Healthcare Improvement (IHI) forThe Patient Safety First Campaign.You need to register via theCampaign website to gain access.Your organisation already has a‘home page’ on the site. From thisyou can select from the campaign listof recommended measures or createyour own custom measures. Then allyou have to do is enter your data andrun charts are created for youautomatically. You can also see thecharts for other Trusts although theability to actually input and changedata is restricted to those individualsthat each organisation hasnominated.

When entering your data there is alsoan opportunity to annotate the chart.This is an extremely useful way ofnoting when you have made changesso that you can see whether they arehaving any effect.

Steps 4 to 6 - the CAR measurement cycle

6 Reviewmeasures

4 Collect data

5 Analyse &present

7 Repeat steps 4-6


2 Choose measures

1 Decide aim

6 Reviewmeasures

4 Collect data

5 Analyse &present

7 Repeat steps 4-6


2 Choose measures

1 Decide aim

12

Why run charts?The way you analyse and presentyour collected data is important. Runcharts are a good way to show howmuch variation there is in yourprocess over time. Also, plotting dataover time is a simple and effectiveway to determine whether thechanges you are making are leadingto improvement.

The figure below shows thepercentage of medicines reconciledon a medical admissions unit. It hasalso been annotated with the datesthat specific changes were tested orintroduced to the medicinesreconciliation process on the ward.

In the first few months, thepercentage reconciled varied between30% and 50%. Once a new formwas introduced in October 2007,performance rose slightly and seemedto stabilise at 55%. The letter fromthe Clinical Director does not seem tohave had much effect whereas theintroduction of pharmacy had a moreobvious one. It is too early to tell fromthis data whether the improvement is

permanent, we would need severalmore months showing 90% beforewe could be confident about that.Nevertheless the run chart showsclearly which interventions had animpact and which ones didn’t. This isimportant to know. We don’t want tobe spending time and energypursuing something that is nothelping us.

One more thing would help us inusing this chart. We should add agoal or target line that representswhere we are trying to get to.Keeping the goal line on every graphensures everyone viewing the graphcan see at a glance where the work isat in relation to achieving the aim.

How do I know whether changesare an improvement?As you will have seen from theprevious example charts may go upand down but we need to have someway of knowing whether this is justrandom chance or the result of a realchange. There are 4 tests that youcan apply to run charts to help youidentify what’s happening after

you’ve made change and thereforedetermine whether it is really animprovement. You can apply the 4tests to your measures on theExtranet by selecting the Run Chartoption from the Reports tab.

Two of the tests make use of themean (average) or median values ofyour data and also the concept of a‘run’. The median is simply the middlevalue of all your values if they werearranged in order. If you are creatingyour own charts, you should calculatethe mean or median and plot it onyour chart – this is called the ‘typicalvalue’. A ‘run’ is a consecutive seriesof points that are above the medianor below it. As a general rule use themean. If the data points look very‘spiky’ (ie there is a frequent widevariation in your lower and upperfigures use the median.

The tests are:Test 1: 6 or more consecutive pointsabove or below the man.These runs indicate a shift in theprocess. Values are still varying butthey are doing so around a newmean or average value. If this is shiftin the right direction, it is likely thatthe change you made is having abeneficial effect. This is the mostfrequent type of change in the datathat you will see.

Test 2: 5 or more consecutive pointsall increasing or decreasing. This indicates a trend and suggeststhat the change you made is havingan effect but you don’t know yetwhere performance will becomestable again. You need to keepmeasuring to find out. This situationis more likely to occur if you arerolling out a change over a period of time.


13

Test 3: Too many or too few runs.Count them up by circling the runs asin the example to the right. Not thatany points that fall on themean/median line should be ignored.Use the table in Appendix 3 to workout whether your variation is due torandom causes. If the number of runsis inside the range this is what wemight expect by chance. If thenumber falls outside the range thensome external factor is having aneffect. Too many runs suggest theprocess has become less consistentand it is possible that your changehas had a detrimental effect. Too fewruns suggests a more consistentprocess.

Test 4: An ‘astronomical’ data point.The example of journey times in thesub section “Testing for changes withSPC charts” explains what this mightlook like. You should use your ownjudgement to assess whether theresult in question really is ‘odd’. Oftensuch markedly out of range results arecaused by a data collection or datadefinition problem so check that first.If the data seems ok then try to findout what might have caused such anodd result. It may cause you to thinkabout creating a contingency plan forif such an occasion arose again.

What is the difference betweenrun charts and statistical processcontrol (SPC) charts?Run charts should be sufficient fornearly all your measurement but theremay be occasion for you to need tounderstand the statistical processcontrol or SPC chart. The SPC chart isa further refinement of a run chart. It introduces the idea of expectedvariation, that is, how much variationdoes my process typically exhibit?

SPC charts still have a ’typical value’line (mean or median) but add 2further lines, the upper and lowerlimits. The purpose of these lines is toshow you that data points appearingwithin the limits, despite going upand down are doing so as part of thenormal variation that we see ineveryday life. If a data point spikesabove or below these limits then youknow something different hashappened – a special event, hencethis is called special cause variation.When events like this are seen on achart you need to investigate whathappened. Even though the event

might be unlikely to occur again it isstill worth considering if there isanything you can do to minimise theimpact if it did.

If our process exhibits just randomvariation, we can use the SPC chart to‘predict’ what future performancewould be like. We would expect anyfuture data points to vary around theaverage and lie within the limits.


Testing for changes with SPC chartsTo illustrate how test 4 describedearlier applies also to SPC charts,consider your journey to work. Somedays are quicker than others but youtend to know on average how long ittakes you, on a ‘good’ day and on a‘bad’ day. If your average journeytime is 30 minutes and it neverusually takes you more than 45minutes or less than 15 minutes, thenyou know how much time to allow.But the day that it took you 90minutes because you had a flat tyreon your car and you didn’t have ajack with you to change it, you andyour colleagues would know thatsomething was different (special)about that day. Even though it mightnot happen again you would stillprobably take the jack out of yourgarage and put it in the boot of thecar so you could change your tyremore quickly if it did. If it actuallydidn’t feel as if it took as long as 45minutes to sort out the problem youmight also check your watch and theclock on the wall in the office to seeif it really did take you that long or ifthere was an inconsistency in therecording of the time.

You can also use run chart tests 1 and2 with SPC charts too. If using SPCcharts there are a few other tests thatyou would do but for the purposes ofmost Campaign improvement workyou are most likely to want to use SPCto assess how likely your current processis to deliver what you want it to.

Assessing process capability withSPC chartsYou can use the process limits of theSPC chart to help you assess howcapable you process is of doing whatyou want it to do i.e. whether you arelikely to reach a particular target. Inthe chart shown, our process isperforming at an average of 50 casesper month with an expected range of28 to 72 cases. If we have a target ofno less than 40 cases per month(shown as a green line), are we likelyto hit it? Yes we are but not all the

time. We would need to either shiftthe whole range up so that the lowerlimit now sat at 40 cases up from 28cases which is an increase of 12cases. But to do this we now need tocomplete an average number of 62cases per month! Alternatively wecould try to reduce the monthlyvariation. We will still complete 50cases on average but now the rangeis 40 to 60 cases. Pursuing the firstoption means we have to do morework to hit our target but opting forthe second means we don’t.

14


If you want to know more about variation the following book is an excellent and concise introduction: Wheeler,Donald J. Understanding variation: the key to managing chaos. SPC Press, 2000.

15

It is vital that you set time aside tolook at what your measures aretelling you. This can be incorporatedinto your Campaign steering groupmeeting if you have one or otherregular meetings. If you don’t havean existing meeting that includes theright people, you will need to set oneup. It needn’t be a long meeting, 30 minutes is perfectly adequate toreview where you are and decide thenext actions. Remember that thepurpose of measurement is to leadyou to making the right decisionsabout your improvement project.

The review meeting template inAppendix 2 may help you to set upand conduct your review meetingeffectively.

Who needs to know what thedata is telling you?‘The How to Guide for Leadership forSafety’ (available atwww.patientsafetyfirst.nhs.uk)outlines the roles senior leaders andthe board play in monitoring progressand driving the execution of projects.There is however a concern amongstthis group that there is a potential forthem to become overwhelmed withdetailed data. For this reason thereneeds to be clear hierarchy ofreporting so that each layer of theorganisation only receives theinformation it requires for assuranceand or decision making.

The key aim is to ensure that eachlayer of staff only receive theinformation they need to assure themthat the changes are progressing inthe right direction and where theystand in relation to the hard red goalline. Balancing measures and lowerlevel process measures may only needto be reported to the Board if there isa stall in progress suggesting there isa problem that requires theirattention or a decision from them.


Step 6 - Review your data to decide what it is telling you

6 Reviewmeasures

4 Collect data

5 Analyse &present

7 Repeat steps 4-6


2 Choose measures

1 Decide aim

Figure: The hierarchy of measurement reporting

Adapted from Lloyd & Caldwell, IHI. 2007

Board & CEO

Higher level outcomemeasures

Focus onoutcome

Focus onprocess

Higher level outcomemeasures

Ba l an

c i ng

mea su

re s

Relevant process + outcome measures

Relevant process + outcome measures

Service managers

Unit/department managers,project staff

Front line staff

16


Repeat steps 4, 5 and 6 each monthor more frequently.

If you are measuring compliance witha process (such as compliance withhandwashing or the ventilator carebundle) aim for a minimum of 95%for non-catastrophic process.Obviously, for a catastrophic process(i.e. one where if it fails it will almostcertainly result in serious injury ordeath) aim for 100%. Keep makingchanges until your data tells you thisis so.

For outcomes (such as surgical siteinfection rate or number of centralline infections), you are aiming toconsistently meet or exceed yourgoal. If you are using SPC charts,ensure the goal sits outside theappropriate upper or lower limit.

When do I stop measuring?The simple answer is “you don’t”. If you are consistently meeting yourgoal you should still look to see ifthere are further improvements thatcould be made. If you aimed for 0%or 100% and are meeting itconsistently you should still continueto measure so that any deviations arepicked up and acted upon quickly. Inthese cases you may decide tomeasure slightly less frequently,however be aware that the process ofmeasuring does have a positive effectin keeping awareness high anddemonstrating that the goals you aremeasuring are important to theorganisation.

Step 7 - Keep going!

6 Reviewmeasures

4 Collect data

5 Analyse &present

7 Repeat steps 4-6


2 Choose measures

1 Decide aim

17

Appendix 1: Measures template

Measures checklist

Measure setup

Measurement process

Measure name:

Measure definition What data item comprises the Numerator?

What data item comprises the Denominator?(some measurement do not require one)

What is the calculation?

Which patient groups are covered?

Goal Setting What is the numeric goal you are setting yourselves?

Who is responsible for setting this?

When will it be achieved by?


Appendix

Collect

AnalyseCalculate measure and present results

Review

Is the data available?Currently available/Available with minor changes/Prospective collection needed

Who is responsible for data collection?

What is the process of collection?

What is the process for presenting results?E.g. enter data in extranet, create run chart in Excel

Who is responsible for the analysis?

How often is the analysis completed?

Where will decisions be made based on results?

Who is responsible for taking action?

18


Appendix 2: Review meeting template

Review Meeting Guidelines

Where: When:

Objectives Participants and roles

Follow up on actions from previous meeting Chair

Understand changes in performance since Otherslast meeting

Discuss Issues, identify next steps and assign responsibility

Who do I contact if I won’t be here or can’t update my chart?

Inputs Outputs

Agreed aims Agreed action and responsibilities

Update measures data

Actions from previous week

Agenda

1. Welcome 1 min

2. Update on actions from previous week 5 min

3. Review charts and discuss changes since last week 5 min

4. Agree what actions to take to improve the measure 5 min

5. Decide who will take each action and by when 5 min

6. Confirm attendance for next meeting 4 min

19


Appendix 3: expected number of runs

Tests for Number of Runs Above and Below the Median

Number of Lower Limit Upper Limit Number of Lower Limit Upper Limit Data Points for Number of runs for Number of runs Data Points for Number of runs for Number of runs

10 3 8 34 12 23

11 3 9 35 13 23

12 3 10 36 13 24

13 4 10 37 13 25

14 4 11 38 14 25

15 4 12 39 14 26

16 5 12 40 15 26

17 5 13 41 16 26

18 6 13 42 16 27

19 6 14 43 17 27

20 6 15 44 17 28

21 7 15 45 17 29

22 7 16 46 17 30

23 8 16 47 18 30

24 8 17 48 18 31

25 9 17 49 19 31

26 9 18 50 19 32

27 9 19 60 24 37

28 10 19 70 28 43

29 10 20 80 33 48

30 11 20 90 37 54

31 11 21 100 42 59

32 11 22 110 46 65

33 11 22 120 51 70

20


Acknowledgements

This guide was produced as part of the Patient Safety Campaign.

Thanks to the English Campaign Team Members and others who havecontributed to this guide.

We also wish to thank and acknowledge the Institute for HealthcareImprovement (IHI) for their support and contribution.

Authors:

Clarke, Julia: Field Operations Manager/Content Development Lead;Patient Safety First Campaign. Associate (Safer Care Priority Programme);NHS Institute for Innovation and Improvement.

Davidge, Mike: Head of Measurement, NHS Institute.

James, Lou: Associate (Safer Care Priority Programme); NHS Institute for Innovation and Improvement.

The How-to guide for measurement for improvement · always work first time. The testing process not...

Documents

Transcript of The How-to guide for measurement for improvement · always work first time. The testing process not...