JBI REPORTS 2004 2 45 ndash 64

Blackwell Science LtdOxford UKJBRJBI Reports1479-697X2004 The Joanna Briggs InstituteMarch 2004224564Primary ResearchBalancing the evidenceA Pearson

Correspondence Professor Alan Pearson School of Nursing and M idw ifery La Trobe University Bundoora Vic 3083 Australia Email apearsonlatrobeeduau

P R I M A R Y R E S E A R C H

Balancing the evidence incorporating the synthesis of qualitative data into systematic reviews

Alan Pearson RN PhD FRC NA FAAG FRC NExecutive D irector The Joanna Briggs Institute Roya l Adela ide Hospita l Adela ide South Austra lia Professor of Nursing School of Nursing and M idwifery La Trobe University Melbourne Victoria and Adjunct Professor Department of Clinica l Nursing The University of Adela ide Adela ide South Austra lia Austra lia

AbstractTechniques for the systematic review of evidence of effectiveness are now well establishedHealth-care professionals argue however for a need to recognise evidence ofappropriateness and feasibility and for the development of methodologies to appraise andsynthesise the results of qualitative research This paper describes a participatory projectdesigned to develop systems to systematically review qualitative evidence The QualitativeAssessment and Review Instrument is described in detail and a suite of programs designedto conduct comprehensive reviews of evidence for health-care practice is outlined Asevidence-based practice increases in sophistication and its influence in health servicedelivery expands the need for broadening the view of what constitutes legitimate evidenceis advanced by clinicians and the approach described attempts to achieve a balance inevidence review that recognises the value of quantitative and qualitative evidence

Key words evidence meta-synthesis qualitative research systematic review

Introduction

Most health professions are increasingly embracing the concept of evidence-based practiceand many use evidence-based guidelines to inform (rather than direct) practice In NorthAmerica considerable resources have been invested in high quality high cost research anddevelopment programs to develop evidence-based clinical guidelines In the UK policyinitiatives have directed health-care provider agencies to develop research and develop-ment strategies to establish research and development units and to promote practicesbased on the best know ledge available At the same time the UK Government hasestablished a number of Centres for Evidence-Based Practice and health research centres

At an international level the Cochrane Collaboration has linked research and develop-ment sites across the world to review and analyse random ised clinical trials from aninternational perspective to generate reports to inform practitioners to influence practiceand to be a resource in the development of consensus guidelines

46 A Pearson

Practical application of rigorously reviewed evidence isnow promoted via the development and dissem ination ofpractice guidelines in most developed health-care systemsC linical practice guidelines consist of statements to assistpractitioner and patient decisions about appropriate healthcare for specific clinical circumstances that are systematicallydeveloped on the basis of consensus w ithin expert groupsAn increasing number of well-constructed practical and evi-dence-based guidelines are being developed

Evidence-based practice is now almost institutionalised inmost industrialised countries especially in Europe the UKNorth America and Australasia Many of these countries haveestablished centres for evidence-based health care evi-dence-based medicine and evidence-based nursing Forexample there are Cochrane Centres in all of these countriesand centres for evidence-based nursing in the UK and NorthAmerica The Joanna Briggs Institute (JBI) based in Australiahas collaborating nursing centres in China Thailand SpainEngland and South Africa in addition to its Australian cen-tres in Queensland New South Wales Victoria SouthAustralia Western Australia and the Northern Territory TheJBI also has multidisciplinary centres for rural health andaged care physiotherapy occupational therapy podiatrymedical radiation and nutrition and dietetics

Essentially evidence-based practice is the combination ofevidence derived from individual clinical or professionalexpertise w ith the best available external evidence to pro-duce practice that is most likely to lead to a positive outcomefor a client or patient The evidence-based approach topractice is of relevance to all professionals who work inhealth care Sackett et a l contend that evidence-basedhealth care is lsquothe conscientious explicit and judicious useof current best evidence in making decisions about the careof individual patientsrsquo1

Evidence-based approaches have provoked some contro-versy however The most controversial issue relates to thecurrent focus on evidence of effectiveness The prevailingorthodoxy in systematically review ing evidence elevates theresults of experimental research to a position of superiorityin terms of quality and applicability to practice disregard-ing the results of non-quantifiable research as legitimateevidence for practice because of its interest in subjectivityand interpretation Critics of the prevailing privileging ofthe random ised controlled trial (RCT) and quantitativeresearch cite the arguments inherent in critiques of tradi-tional science and the emergence of new paradigms forknow ledge For example FitzGerald argues that althoughtraditional scientific method w ith its emphasis on objectiv-ity plays an important in the development of know ledge

and technology lsquo the dom inance of traditional scienceneeds to be challenged [in health if practitioners] w ish tomake a place for different ways of know ing in theirpracticersquo2

Although the RCT is probably the lsquobestrsquo approach to gen-erating evidence of effectiveness nurses medical practitio-ners and allied health professionals are concerned w ith morethan cause and effect questions and this is reflected in thew ide range of research approaches utilised in the health fieldto generate know ledge for practice Although its proponentswould argue that evidence-based practice is not lim ited tothe utilisation of the results of traditional research it is diffi-cult to defend such an argument given the considerableemphasis placed on RCTs and meta-analyses to date Thishas drawn criticism from those professions who regard qual-itative research methods as equally valid forms of researchthat generate legitimate evidence for practice

Know ledge acquired from qualitative approaches toresearch is largely absent in current approaches to system-atic reviews This is partly because the rapid development ofaccepted approaches to the appraisal and synthesis of evi-dence by quantitative researchers has not been accompa-nied by sim ilar efforts by those w ith expertise in qualitativeapproaches to inquiry Questions such as lsquowhat is evidencersquoand lsquowhat are acceptable research results in terms of gener-ating know ledge that amounts to evidence for the purposeof inform ing practicersquo inspire conflicting views These dif-ferences generally align w ith the various positions that char-acterise the long-standing debate between qualitative andquantitative researchers This is clearly not an easily resolvedargument but it is vitally important in terms of ascertainingthe value of research-generated lsquoevidencersquo to health-carepractice

This paper describes an initial attempt to recognise theresults of non-quantitative research as appropriate evidencefor health-care practitioners

What counts as evidence

Evidence in health-care practice

In general terms evidence refers to data or information usedto decide whether or not a claim or view should be trustedIn health care practitioners and patients make numerousdecisions and in doing so weigh up numerous types ofinformation before taking action Although the results ofwell-designed research are an obvious source of evidencethe results of formal research are by no means the only dataused in everyday practice O ther determ inants include thepatient and hisher relevant others the practitionerrsquos own

Balancing the evidence 47

experiences and the nature and norms of the setting andculture in which the health care is being delivered these areall rich sources of evidence to draw upon in making clinicaldecisions

The dom inant orthodoxy of regarding the results of quan-titative research as evidence and all other know ledge assomething other than evidence does not reflect the under-standing of practice held by many clinicians This is notsurprising given the nature and meaning of evidence ineveryday life

Evidence in its most generic sense has been defined asbeing lsquothe available facts circumstances etc supporting orotherw ise a belief proposition etc or indicating whether athing is true or validrsquo3 Evidence in a legal sense has beendefined as being lsquoinformation given personally or drawnfrom a document etc and tending to prove a fact orproposition or statements or proofs adm issible as tes-timony in a law courtrsquo3

For philosophers evidence is understood to be informa-tion bearing on the truth or falsity of a proposition Accord-ing to Audi

A personrsquos evidence is generally taken to be all the informationa person has positive or negative relevant to a proposition Thenotion of evidence used in philosophy thus differs from theordinary notion according to which physical objects such as astrand of hair or a drop of blood counts as evidence Onersquosinformation about such objects could count as evidence in thephilosophical sense4

It is important from a philosophical standpoint to under-stand that the concept of evidence plays a key role in ourunderstanding of know ledge and rationality Traditionallylsquoone has know ledge only when one has a true belief basedon very strong evidencersquo4 Moreover for belief to be rationalit must be based on adequate evidence even when thatevidence is insufficient to ground know ledge

Evidence for health professionals

Some serious consideration has been given to the meaningof evidence in relation to the evidence-based health-caremovement D ixon-Woods et a l for example discuss therelevance of evidence elicited through qualitative researchin systematic reviews and Noblit and Hare describe howethnographic approaches to inquiry generate evidenceappropriate to practice56

According to Humphris the term lsquoevidence-basedrsquo inhealth care lsquoimplies the use and application of researchevidence as a basis on which to make health-care decisionsas opposed to decisions not based on evidencersquo7 Within themainstream health field led by medicine and medical sci-

ence research has been narrow ly confined to the empirico-analytical paradigm focusing on objectivity measurementand statistical significance This process of know ledge gen-eration involves testing a hypothesis or a set of hypothesesby deriving consequences from it and then testing whetherthose consequences hold true by experiment and observa-tion A theory of evidence assists here to the extent that itindicates what relationship should exist between the obser-vation reports and the hypotheses if those reports are toconstitute evidence for the hypotheses

There is some legitimacy in regarding the RCT and otherapproaches that focus on measurement and statistical anal-ysis as the most desirable approach to evidence generationwhen the question relates to cause-and-effect relationshipsHowever health professionals have broader evidence inter-ests that relate to the experience of health illness and healthcare Indeed it is not unusual w ithin the broad field of healthcare to find that the lsquobest availablersquo evidence on a giventopic cannot be reduced to a quantifiable value Expertopinion ndash whether it is expressed by an individual a learnedbody or by a group of experts in the form of a consensusguideline ndash draws on the experience of practitioners Well-designed qualitative research while often not properlyunderstood by researchers grounded in the physical sci-ences constitutes lsquogoodrsquo human science

Pearson argues for a pluralistic approach when consider-ing lsquowhat counts as evidencersquo for health-care practices andEvans and Pearson suggest reviews that include both (oreither) qualitative evidence and quantitative evidence are ofimportance to most practitioners89 However they go on tosuggest that lsquo optimal methods for review ing qualitativeresearch are still evolvingrsquo9

Evidence-based practice

Clinical effectiveness an evidence base for practice

The evidence-based practice movement currently focuses onthe effectiveness of interventions and activities and the termlsquosystematic reviewrsquo is now interpreted as a process thatsummarises and synthesises the results of experimental andother quantitative studies The results of descriptive obser-vational and interpretative inquiry are afforded little if anystatus in most systematic reviews Evans and Pearson aver

It can be argued that the approach of the systematic reviewshould be used for all summaries of the research ensuring thatthe care and rigour that was utilised by the primary researcher ismaintained by the reviewer However this concept has notreceived w ide acceptance and so the focus of most systematicreviews has remained predom inantly fixed on the random isedcontrolled trial9

48 A Pearson

Evans and Pearson go on to argue that the systematicreview process should be expanded to include findings gen-erated from all forms of rigorous research as well as thosefrom RCTs The systematic review process originated in thefield of organisational psychology from which medicalresearch took it and refined it

Writing in 1982 Salipente et a l clearly describe how theprocess was conceptualised to synthesise research groundedin any tradition10 Thus w ithin this context Evans and Pear-sonrsquos plea to lsquoexpandrsquo the rem it of systematic reviews couldbe better expressed as returning to the original nature of thesystematic review

The systematic review

The core of evidence-based practice is the systematic reviewof the literature on a particular condition intervention orissue The systematic review is essentially an analysis of all ofthe available literature (that is evidence) and a judgementof the effectiveness or otherw ise of a particular practiceCurrently the systematic review involves the follow ingsteps1 The development of a rigorous proposal or protocol is

vital for a high quality systematic review The reviewprotocol provides a predeterm ined plan to ensure scien-tific thoroughness and the m inim isation of potential biasIt also allows for periodic updating of the review ifnecessary

2 The protocol should state in detail the questions orhypotheses to be discussed in the review Questionsregarding the patients setting interventions and out-comes to be investigated should be specific

3 The protocol must describe the criteria that w ill be usedto select the literature The inclusion criteria shouldaddress the participants of the primary studies the inter-vention and the outcomes In addition to this it shouldalso specify what research methodologies w ill be consid-ered for inclusion in the review (eg RCTs clinical trialscase studies)

4 The protocol should provide a detailed strategy that w illbe used to identify all relevant literature w ithin a specifiedtime frame This should include databases and bibliogra-phies that w ill be searched and the search terms that w illbe used

5 Critical appraisal of the studies retrieved is important toassess the quality of the research m inim ising the risk ofan inconclusive review w ith excessive variation in studyquality The protocol must therefore describe how thequality of primary studies w ill be assessed and any exclu-sion criteria based on quality considerations

6 It is necessary to extract data from the primary researchregarding the participants the intervention the outcomemeasures and the results

7 Statistical analysis (meta analysis) may or may not be usedand w ill depend on the nature and quality of studiesincluded in the review Although it may not be possibleto state exactly what analysis w ill be undertaken thegeneral approach should be included in the protocolWhen statistical analysis is not possible current practiceis to develop a narrative summary

Pearson in arguing that evidence-based practice includesan interest in research on clinical effectiveness but is notconfined to this interest says

random ised trials are the gold standard for phenomena thatwe are interested in studying from a cause and effect perspectivebut clearly they are not the gold standard if we areinterested in how patients and nurses relate to eachother or if we are interested in how patients live through theexperience of radiotherapy when they have a life threateningillness We have yet to work out how to assess the quality ofalternative approaches to research other than the RCT11

He goes on to suggest that lsquo evidence-based practice isnot exclusively about effectiveness it is about basing prac-tice on the best available evidencersquo11

The diverse origins of problems in health care require abroad interpretation of what counts as valid evidence forpractice and the utilisation of a diverse range of researchmethodologies to generate appropriate evidence Method-ological approaches in this area need to be eclectic enoughto incorporate classical medical and scientific designs andthe emerging qualitative and action-orientated approachesfrom the humanities and social and behavioural sciencesThe development of interdisciplinary research and a greaterunderstanding of the relationship between medical nursingand allied health interventions are also fundamental to thecreation of research methodologies that are relevant andsensitive to the health needs of consumers

There is a small but grow ing amount of literature address-ing the role of qualitative research in evidence-basedpractice which recognises a need to move beyond theeffectiveness of interventions to consider their appropriate-ness and practical feasibility

Lemmer et a l in attempting to conduct a systematicreview in an area of health visiting focusing on the RCT asa lsquogold standardrsquo report on a paucity of trials in this fieldand argue that clinical complexity demands a need to inte-grate qualitative methods into systematic reviews Theyargue that lsquo the comprehensiveness and synthesis of asystematic review are more important to emphasise than

Balancing the evidence 49

whether the literature is outside the clinical rem it of anRCTrsquo12

The need to more fully integrate the results of qualitativeresearch into the systematic process is well stated by Popayand Williams who suggest that lsquo there are many propo-nents of evidence-based decision making w ithin healthcarewho cannot andor w ill not accept that qualitative researchhas an important part to play rsquo13

Popay and Williams assert that the results of qualitativeresearch do more than simply enhance those of quantitativestudies and suggest that qualitative research is capable ofgenerating evidence thatbull explores taken-for-granted practicesbull increases understanding of consumer and clinical

behaviourbull develops interventionsbull illum inates patientrsquos perceptions on qualityappropriate-

nessbull gives guidance to understanding organisational culture

and change management andbull evaluates complex policy initiatives13

Green and Britten stated that

Qualitative research may seem unscientific and anecdotal tomany medical scientists However as the critics of evidence basedmedicine are quick to point out medicine is more than theapplication of scientific rules14

Green and Britten go on to argue that qualitative researchfindings provide rigorous accounts of treatment regimens ineveryday contexts They also contend that there is anincreasing need w ithin the evidence-based practice arena toraise awareness of the fact that different research questionsrequire different kinds of research They are unequivocal intheir assertion that lsquogoodrsquo evidence goes further than theresults of meta-analysis of RCTs

The Cochrane Qualitative Methods Group established in2002 is currently exploring the scope for incorporatingqualitative research into Cochrane reviews There are still nointernationally reviewed guidelines for assessing the qualityof specific qualitative methods and no established proce-dures for ranking or rating qualitative research findingsreported in the literature There have been however a num-ber of attempts to synthesise (as a form of meta-analysis)the results of sim ilar qualitative studies and these are welldescribed by Sandelowski et a l these authors have alsodeveloped an in-depth theoretical approach to the system-atic metasynthesis of qualitative findings that maintain theintegrity of individual studies15 Draw ing on the work ofSandelowski et a l Popay and Williams and Lemmer et a lan approach to qualitative meta-analysis quality assessment

and the development of a quality rating scale for qualitativeresearch results could be used to add appropriateness andfeasibility dimensions to the current effectiveness-orientatedsystematic review process121315

There are signs that the evidence-based practice move-ment is beginning to develop a more comprehensive viewof evidence There are research initiatives attempting toconstruct approaches to assessing and synthesising theresults of interpretative and critical research so that theseforms of evidence can become an integral part of systematicreviews and thus inform practice Several authors havereported on the systematic review of evidence elicitedthrough interpretative and critical approaches to inquiry16ndash

18 An approach to the meta-synthesis of qualitative findingsand the problems associated w ith synthesising the findingsof studies that are essentially context-bound is described byJensen and Allen and Popay and Roen overview a w ide rangeof current initiatives focusing on methods to appraise andsynthesise qualitative research w ithin the framework of thesystematic review process1920

Developing sound and acceptable methodologies toappraise the findings of qualitative research and to synthe-sise the findings of two or more sim ilar studies is bothcomplex and challenging Interesting developments in thisarea have been reported by several authors1721ndash24 Suchevolving approaches to evidence-based practice represent agrow ing body of work supporting the appropriate use ofrelevant evidence in the systematic review process that w illhelp practitioners to perform well in practice and to use theirprofessional judgement in the use of appropriate evidence

Including qualitative data in systematic reviews

Over a period of two years a project utilising participatoryprocesses at three consensus workshops has explored thereview of qualitative evidence and an electronic packagehas been designed to enable reviewers to systematicallyreview qualitative evidence The purpose of this project wasto determ ine how evidence generated through qualitativeresearch could be systematically reviewed and to identifyhow evidence of appropriateness and feasibility could aug-ment evidence of effectiveness in evidence-based healthcare

Project design

The project consisted of four phasesIn Phase 1 acknow ledged experts in qualitative and

action-orientated approaches were invited to attend a con-

50 A Pearson

sensus workshop w ith a view to working w ith the researcherto develop instruments to evaluate and extract data from qualitative research reports

In Phase 2 acknow ledged experts in qualitative andaction-orientated approaches were invited to attend a con-sensus workshop w ith a view to working w ith the researcherto develop a systematic process of extracting and synthesis-ing data from qualitative research reports

In Phase 3 the researcher worked w ith a software devel-oper to develop an electronic system to review qualitativeevidence

In Phase 4 the software developed was piloted w ith theexpert groups and other health professionals attending sys-tematic review training workshops

Outcomes

At the workshops a group of leading Australian qualitativeresearchers were invited to participate in a consensus work-shop to consider how a systematic process of extracting andsynthesising qualitative data can occur to reflect a rigorousprocess equivalent to the existing processes applied to theresults of RCT and other quantitative research while main-taining sensitivity to the contextual nature of qualitativeresearch

More specifically participants were asked tobull design appropriate data extraction toolsbull design appropriate data synthesis tools andbull draft a position statement

N ine of those invited attended the workshops Workshopattendees were Professor Mary FitzGerald University ofNewcastle Associate Professor Jane Stein-Parbury Universityof Technology Professor Colin Holmes James Cook Univer-sity of Northern Queensland Professor M ichael C linton Cur-tin University Professor Desley Hegney University ofSouthern Queensland Dr Ken Walsh The University ofAdelaide Dr Karen Francis Charles Sturt University MrMatt Lew is La Trobe University and Ms Cathy Ward LaTrobe University

Consensus workshops

Expected outcomes were agreed upon at each workshop Itwas also agreed that the workshop outcomes would needto accommodate the nature of qualitative approaches toresearch rather than replicate existing formats The partici-pants emphasised the complexity of interpretative and crit-ical understandings of phenomena but were also aware ofthe need to ensure that outcomes would be practical andusable Participants were also m indful of the complexity ofthe types of research under consideration and wanted to

balance the utility of the outcomes w ith the complexity ofthe material

lsquoMeaningrsquo was identified as a unifying theme of qualita-tive research A list of methodological frameworks was gen-erated and these were then grouped in relation to theirorientation (Appendix I) Judgements were made about thevalue of each methodological position in relation to specificcriteria of meaningfulness appropriateness feasibility andeffectiveness

The participants recognised that they were working w itha w ide range of methodologies each w ith its own strengthsand weaknesses They agreed to develop a lsquomatrixrsquoapproach which was considered to be more appropriatethan a simple hierarchy of evidence With regard to thecentrality of contextual issues in interpretative and criticalresearch the group suggested that this be taken intoaccount by including statements in the summary about thescope of applicability

The group agreed that an overall matrix and the contextstatement should generate a summary statement describingthe strengths and weaknesses of the available evidence andthe associated levels of confidence

Workshop outcomes statement

Consensus was reached on an lsquooutcomes statementrsquo devel-oped collaboratively by the group (Appendix II)

Draw ing on this consensus summary statement proto-type components for a qualitative assessment and reviewinstrument were developed This was named the QualitativeAssessment and Review Instrument (QARI)

Appraising and synthesising qualitative data

Critical appraisal

The central concern in critically appraising experimental orquantifiable data is to lim it bias and thus establish the valid-ity of a study From a quantitative perspective sources ofbias include selection bias performance bias and attritionbias and validity is assessed by establishing the extent towhich a studyrsquos design and conduct address potential bias

This focus on lim iting bias to establish validity is antithet-ical to the philosophical foundations of qualitativeapproaches to inquiry Emden and Sandelowski suggest thatvalidity in quantitative terms measures those things that itpurports to render generalisable In social inquiry howeverthey argue that validity is perceived as criteria of rigour forqualitative research25

There is much dissent in the literature on the appropriate-ness of establishing criteria to assess the validity of qualita-

Balancing the evidence 51

tive research However Pearson takes the view that atransparent approach to appraising qualitative research iscentral to its ongoing credibility transferability and theoret-ical potential26 Denzin and Lincoln concur w ith this conclu-sion and highlight the need for development of a set ofvalidity criteria sensitive to the nature of the qualitativeresearch and its basis in subjectivity27 Popay et a l areunequivocal in their assertion that the development of stan-dards for assessing evidence from qualitative research is bothpossible and desirable28 They provide the follow ing as aguide to common standardsbull evidence of responsiveness to social context and flexibility

of designbull evidence of theoretical or purposeful samplingbull evidence of adequate descriptionbull evidence of data qualitybull evidence of theoretical and conceptual adequacy andbull potential for assessing typicality28

There is a grow ing amount of literature that exam ines theappraisal of qualitative studies and a large number of for-mats are available Draw ing upon this literature and anextensive process of development and piloting Averis andPearson describe a general set of criteria for appraising thevalidity of interpretative and critical research26 These criteriawere incorporated into the critical appraisal scale of the QARIsoftware which was developed by the project

The critical appraisal scale was piloted and refined in threesystematic review training workshops The checklist inAppendix III provides the essential framework for the criticalappraisal of interpretative and critical studies

Data extraction

Data extraction aims to reduce the findings of many studiesinto a single document and summarisebull methodsbull interventions andbull outcomes

Data extraction involves transferring data from the origi-nal paper using an approach agreed upon and standardisedfor the specific review An agreed format is essential tom inim ise error to provide a historical record of decisionsmade about the data in the review and to become the dataset for analysis and synthesis A data extraction instrumentdraw ing on the literature and input from a panel of expertswas developed extensively piloted refined and incorpo-rated into the QARI software (Appendix IV) Based on thestandard approach promoted by the Cochrane Collabora-tion and adopted by the JBI two reviewers are expected toindependently extract data and then confer

Meta-synthesis

The most complex problem in synthesising textual data isagreeing on and communicating techniques to compare thefindings of each study Meta-synthesis relates to the com-bining of separate elements to form a coherent whole Thisinvolves reasoning from the general to the particular usinga process of logical deduction An approach to the meta-synthesis of qualitative data which draws on the literatureand input from a panel of experts was developed for QARIThis involvesbull translating themes metaphors or conceptsbull transferring actual text or summarised text that illustrates

the theme metaphor or concept andbull re-categorising the data obtained to arrive at a

synthesisIn order to pursue this reviewers need to establish the

follow ing before carrying out data synthesisbull their own rules for setting up categoriesbull how to assign findings to categories andbull how to write narrative summaries for each category

The reviewers need to document these decisions and theirrationale in the systematic review report This process isincorporated into the QARI software

The analysis and synthesis of qualitative studies is com-monly termed meta-synthesis and like meta-analysis it isbased on processed data There are major differencesbetween the approach used to synthesise the findings ofRCT and the approach used for qualitative studies Realityfor the qualitative researcher and reviewer is viewed asmultiple and constructed and so undertaking meta-synthesis means that no two reviewers w ill produce exactlythe same results Although meta-synthesis provides only oneinterpretation it aims to capture the essence of the phenom-enon interest

When engaging in the synthesis of the results of qualita-tive studies differing research methods such as phenome-nology ethnography or grounded theory are not m ixed ina single synthesis of all qualitative studies

The aim of meta-synthesis is to portray an accurate inter-pretation of a phenomenon and to compare and contrastthe constructs of individual studies to reach consensus on anew construction of that phenomenon

Meta-synthesis involvesbull identifying findingsbull grouping findings into categories andbull grouping categories into synthesised findings

Findings Findings are conclusions reached by thereviewer(s) after exam ining the results of data analysis (egthemes metaphors) consisting of a statement that relates

52 A Pearson

two or more phenomena variables or circumstances thatmay inform practice

Categories Categories are groups of findings that reflectsim ilar relationships between sim ilar phenomena variablesor circumstances that may inform practice

Synthesised findings Synthesis refers to the combining ofseparate elements to form a coherent whole using logicaldeduction and reasoning from the general to the particularIn QARI a synthesised finding is defined as an overarchingdescription of a group of categorised findings that allow forthe generation of recommendations for practice

Categorising findingsIn order to pursue this reviewers need to establish their ownrules for the follow ing before carrying out data synthesisbull setting up categoriesbull assigning findings to categories andbull writing narrative summaries for each category

Reviewers need to document these decisions and theirrationale in the systematic review report This process isincorporated into the QARI software

The primary reviewer then categorises findings

Synthesised findingsWhen categorisation is complete the reviewers then studythe categories and synthesise these to form a set of synthe-sised findings

Levels of evidence

Current approaches to evaluating evidence utilise a hierar-chy of evidence designed to assess the validity of recom-mendations for clinical guidelines These approaches focuson the effectiveness of treatment and rank only quantitativeevidence according to the rigour of the research designedto lim it bias An approach to categorising the validity ofqualitative evidence which draws on the literature andinput from a panel of experts has been developed forQARI This approach is based on three levels of qualitativeevidence

Unequivoca l The evidence is beyond reasonable doubtand includes findings that are factual directly reportedobserved and not open to challenge

Credible The evidence while interpretative is plausible inlight of the data and theoretical framework Conclusions canbe logically inferred from the data but because the findingsare essentially interpretative these conclusions are open tochallenge

Unsupported Findings are not supported by the data andnone of the other level descriptors apply

These three levels of evidence are incorporated into theQARI software

Levels of applicability

There is little point in accumulating evidence to answer aquestion if it cannot then be used to benefit patients Evi-dence-based practice involves integration of the best avail-able evidence w ith clinical expertise When it comes todeciding whether or not to incorporate a particular activityor intervention into practice some or all of the follow ingconsiderations w ill be relevantbull Is it availablebull Is it affordablebull Is it applicable in the settingbull Would the patientclient be a w illing participant in the

implementation of the interventionbull Were the patients in the studystudies that provided the

evidence sufficiently sim ilar to your own to justify theimplementation of this particular intervention

bull What w ill be the potential benefits for the patientbull What is the potential harm to the patientbull Does this intervention allow for the individual patientrsquos

values and preferencesIn addition to the requirement to define levels of evidence

for practice there is also a need to establish levels of appli-cability The QARI project group developed the lsquoFeasibilityAppropriateness Meaningfulness and Effectivenessrsquo (FAME)scale26 a hierarchy of applicability of evidence that is incor-porated into the QARI software

The QARI levels of applicability of evidence are shown inTable 1

The QARI software

The QARI attempts to establish the results of non-quantita-tive research as appropriate evidence for health-care practi-tioners It was designed to create a system that would enablehealth scientists and health practitioners to review evidencefrom an inclusive position The purpose of the QARI devel-opmental process was to determ ine how evidence gener-ated through qualitative research could be systematicallyreviewed and identify how evidence of appropriatenessmeaningfulness and feasibility could augment evidence ofeffectiveness in evidence-based health care

The QARI software is designed to manage appraise anal-yse and synthesise textual data as part of a systematic reviewof evidence QARI has been designed as a web-based data-base and incorporates a critical appraisal scale data extrac-tion forms a data synthesis function and a reporting function

Balancing the evidence 53

The software has been internationally peer reviewed It hasbeen tested by a group of international systematic reviewersand is currently being piloted by evidence review groups inCanada Australia England and Scotland

The QARI software can be accessed by those who hold alicence by entering the JBI website (http wwwjoan-nabriggseduau) and then clicking on the SUMARI logo

An overview of how QARI is used is presented here todemonstrate the process of appraisal and meta-synthesis

When QARI is opened the main menu is always acrossthe top of the screen (Fig 1)

Reviews

Reviews are the projects to which Studies Extractions andFindings relate The reviews screen lists the reviews availableto the logged on reviewer (Fig 1)

Before a Reviewer can work on a project they must beassigned to the Review either when the Review is first cre-

Table 1 Qualitative Assessment and Review Instrument (QARI) levels of applicability of evidence

Feasibility Appropriateness Meaningfulness Effectiveness

Immediately practicable Acceptable and justifiable w ithin ethical guidelines

Provides a rationale for practice development

Effectiveness established to a degreethat merits application

Practicable w ith lim ited local training or modest additional resources

Acceptable after m inor revision Provides a rationale for local regional or national reform

Effectiveness established to a degree that suggests application

Practicable w ith extensive additional training or resources

Acceptable after major revision Provides a rationale for practice-relevant research

Effectiveness established to a degree that warrants consideration of applying the findings

Practicable w ith significant national reforms

Acceptable after development of new ethical guidelines

Provides a rationale for advocatingchange

Effectiveness established to a lim ited degree

Impracticable Ethically unacceptable Evidence unlikely to make sense to practitioners

Effectiveness not established

Figure 1 Reviews screen listing the reviews available to the logged on reviewer

46 A Pearson

Practical application of rigorously reviewed evidence isnow promoted via the development and dissem ination ofpractice guidelines in most developed health-care systemsC linical practice guidelines consist of statements to assistpractitioner and patient decisions about appropriate healthcare for specific clinical circumstances that are systematicallydeveloped on the basis of consensus w ithin expert groupsAn increasing number of well-constructed practical and evi-dence-based guidelines are being developed

Evidence-based practice is now almost institutionalised inmost industrialised countries especially in Europe the UKNorth America and Australasia Many of these countries haveestablished centres for evidence-based health care evi-dence-based medicine and evidence-based nursing Forexample there are Cochrane Centres in all of these countriesand centres for evidence-based nursing in the UK and NorthAmerica The Joanna Briggs Institute (JBI) based in Australiahas collaborating nursing centres in China Thailand SpainEngland and South Africa in addition to its Australian cen-tres in Queensland New South Wales Victoria SouthAustralia Western Australia and the Northern Territory TheJBI also has multidisciplinary centres for rural health andaged care physiotherapy occupational therapy podiatrymedical radiation and nutrition and dietetics

Essentially evidence-based practice is the combination ofevidence derived from individual clinical or professionalexpertise w ith the best available external evidence to pro-duce practice that is most likely to lead to a positive outcomefor a client or patient The evidence-based approach topractice is of relevance to all professionals who work inhealth care Sackett et a l contend that evidence-basedhealth care is lsquothe conscientious explicit and judicious useof current best evidence in making decisions about the careof individual patientsrsquo1

Evidence-based approaches have provoked some contro-versy however The most controversial issue relates to thecurrent focus on evidence of effectiveness The prevailingorthodoxy in systematically review ing evidence elevates theresults of experimental research to a position of superiorityin terms of quality and applicability to practice disregard-ing the results of non-quantifiable research as legitimateevidence for practice because of its interest in subjectivityand interpretation Critics of the prevailing privileging ofthe random ised controlled trial (RCT) and quantitativeresearch cite the arguments inherent in critiques of tradi-tional science and the emergence of new paradigms forknow ledge For example FitzGerald argues that althoughtraditional scientific method w ith its emphasis on objectiv-ity plays an important in the development of know ledge

and technology lsquo the dom inance of traditional scienceneeds to be challenged [in health if practitioners] w ish tomake a place for different ways of know ing in theirpracticersquo2

Although the RCT is probably the lsquobestrsquo approach to gen-erating evidence of effectiveness nurses medical practitio-ners and allied health professionals are concerned w ith morethan cause and effect questions and this is reflected in thew ide range of research approaches utilised in the health fieldto generate know ledge for practice Although its proponentswould argue that evidence-based practice is not lim ited tothe utilisation of the results of traditional research it is diffi-cult to defend such an argument given the considerableemphasis placed on RCTs and meta-analyses to date Thishas drawn criticism from those professions who regard qual-itative research methods as equally valid forms of researchthat generate legitimate evidence for practice

Know ledge acquired from qualitative approaches toresearch is largely absent in current approaches to system-atic reviews This is partly because the rapid development ofaccepted approaches to the appraisal and synthesis of evi-dence by quantitative researchers has not been accompa-nied by sim ilar efforts by those w ith expertise in qualitativeapproaches to inquiry Questions such as lsquowhat is evidencersquoand lsquowhat are acceptable research results in terms of gener-ating know ledge that amounts to evidence for the purposeof inform ing practicersquo inspire conflicting views These dif-ferences generally align w ith the various positions that char-acterise the long-standing debate between qualitative andquantitative researchers This is clearly not an easily resolvedargument but it is vitally important in terms of ascertainingthe value of research-generated lsquoevidencersquo to health-carepractice

This paper describes an initial attempt to recognise theresults of non-quantitative research as appropriate evidencefor health-care practitioners

What counts as evidence

Evidence in health-care practice

In general terms evidence refers to data or information usedto decide whether or not a claim or view should be trustedIn health care practitioners and patients make numerousdecisions and in doing so weigh up numerous types ofinformation before taking action Although the results ofwell-designed research are an obvious source of evidencethe results of formal research are by no means the only dataused in everyday practice O ther determ inants include thepatient and hisher relevant others the practitionerrsquos own

Balancing the evidence 47

experiences and the nature and norms of the setting andculture in which the health care is being delivered these areall rich sources of evidence to draw upon in making clinicaldecisions

The dom inant orthodoxy of regarding the results of quan-titative research as evidence and all other know ledge assomething other than evidence does not reflect the under-standing of practice held by many clinicians This is notsurprising given the nature and meaning of evidence ineveryday life

Evidence in its most generic sense has been defined asbeing lsquothe available facts circumstances etc supporting orotherw ise a belief proposition etc or indicating whether athing is true or validrsquo3 Evidence in a legal sense has beendefined as being lsquoinformation given personally or drawnfrom a document etc and tending to prove a fact orproposition or statements or proofs adm issible as tes-timony in a law courtrsquo3

For philosophers evidence is understood to be informa-tion bearing on the truth or falsity of a proposition Accord-ing to Audi

A personrsquos evidence is generally taken to be all the informationa person has positive or negative relevant to a proposition Thenotion of evidence used in philosophy thus differs from theordinary notion according to which physical objects such as astrand of hair or a drop of blood counts as evidence Onersquosinformation about such objects could count as evidence in thephilosophical sense4

It is important from a philosophical standpoint to under-stand that the concept of evidence plays a key role in ourunderstanding of know ledge and rationality Traditionallylsquoone has know ledge only when one has a true belief basedon very strong evidencersquo4 Moreover for belief to be rationalit must be based on adequate evidence even when thatevidence is insufficient to ground know ledge

Evidence for health professionals

Some serious consideration has been given to the meaningof evidence in relation to the evidence-based health-caremovement D ixon-Woods et a l for example discuss therelevance of evidence elicited through qualitative researchin systematic reviews and Noblit and Hare describe howethnographic approaches to inquiry generate evidenceappropriate to practice56

According to Humphris the term lsquoevidence-basedrsquo inhealth care lsquoimplies the use and application of researchevidence as a basis on which to make health-care decisionsas opposed to decisions not based on evidencersquo7 Within themainstream health field led by medicine and medical sci-

ence research has been narrow ly confined to the empirico-analytical paradigm focusing on objectivity measurementand statistical significance This process of know ledge gen-eration involves testing a hypothesis or a set of hypothesesby deriving consequences from it and then testing whetherthose consequences hold true by experiment and observa-tion A theory of evidence assists here to the extent that itindicates what relationship should exist between the obser-vation reports and the hypotheses if those reports are toconstitute evidence for the hypotheses

There is some legitimacy in regarding the RCT and otherapproaches that focus on measurement and statistical anal-ysis as the most desirable approach to evidence generationwhen the question relates to cause-and-effect relationshipsHowever health professionals have broader evidence inter-ests that relate to the experience of health illness and healthcare Indeed it is not unusual w ithin the broad field of healthcare to find that the lsquobest availablersquo evidence on a giventopic cannot be reduced to a quantifiable value Expertopinion ndash whether it is expressed by an individual a learnedbody or by a group of experts in the form of a consensusguideline ndash draws on the experience of practitioners Well-designed qualitative research while often not properlyunderstood by researchers grounded in the physical sci-ences constitutes lsquogoodrsquo human science

Pearson argues for a pluralistic approach when consider-ing lsquowhat counts as evidencersquo for health-care practices andEvans and Pearson suggest reviews that include both (oreither) qualitative evidence and quantitative evidence are ofimportance to most practitioners89 However they go on tosuggest that lsquo optimal methods for review ing qualitativeresearch are still evolvingrsquo9

Evidence-based practice

Clinical effectiveness an evidence base for practice

The evidence-based practice movement currently focuses onthe effectiveness of interventions and activities and the termlsquosystematic reviewrsquo is now interpreted as a process thatsummarises and synthesises the results of experimental andother quantitative studies The results of descriptive obser-vational and interpretative inquiry are afforded little if anystatus in most systematic reviews Evans and Pearson aver

It can be argued that the approach of the systematic reviewshould be used for all summaries of the research ensuring thatthe care and rigour that was utilised by the primary researcher ismaintained by the reviewer However this concept has notreceived w ide acceptance and so the focus of most systematicreviews has remained predom inantly fixed on the random isedcontrolled trial9

48 A Pearson

Evans and Pearson go on to argue that the systematicreview process should be expanded to include findings gen-erated from all forms of rigorous research as well as thosefrom RCTs The systematic review process originated in thefield of organisational psychology from which medicalresearch took it and refined it

Writing in 1982 Salipente et a l clearly describe how theprocess was conceptualised to synthesise research groundedin any tradition10 Thus w ithin this context Evans and Pear-sonrsquos plea to lsquoexpandrsquo the rem it of systematic reviews couldbe better expressed as returning to the original nature of thesystematic review

The systematic review

The core of evidence-based practice is the systematic reviewof the literature on a particular condition intervention orissue The systematic review is essentially an analysis of all ofthe available literature (that is evidence) and a judgementof the effectiveness or otherw ise of a particular practiceCurrently the systematic review involves the follow ingsteps1 The development of a rigorous proposal or protocol is

vital for a high quality systematic review The reviewprotocol provides a predeterm ined plan to ensure scien-tific thoroughness and the m inim isation of potential biasIt also allows for periodic updating of the review ifnecessary

2 The protocol should state in detail the questions orhypotheses to be discussed in the review Questionsregarding the patients setting interventions and out-comes to be investigated should be specific

3 The protocol must describe the criteria that w ill be usedto select the literature The inclusion criteria shouldaddress the participants of the primary studies the inter-vention and the outcomes In addition to this it shouldalso specify what research methodologies w ill be consid-ered for inclusion in the review (eg RCTs clinical trialscase studies)

4 The protocol should provide a detailed strategy that w illbe used to identify all relevant literature w ithin a specifiedtime frame This should include databases and bibliogra-phies that w ill be searched and the search terms that w illbe used

5 Critical appraisal of the studies retrieved is important toassess the quality of the research m inim ising the risk ofan inconclusive review w ith excessive variation in studyquality The protocol must therefore describe how thequality of primary studies w ill be assessed and any exclu-sion criteria based on quality considerations

6 It is necessary to extract data from the primary researchregarding the participants the intervention the outcomemeasures and the results

7 Statistical analysis (meta analysis) may or may not be usedand w ill depend on the nature and quality of studiesincluded in the review Although it may not be possibleto state exactly what analysis w ill be undertaken thegeneral approach should be included in the protocolWhen statistical analysis is not possible current practiceis to develop a narrative summary

Pearson in arguing that evidence-based practice includesan interest in research on clinical effectiveness but is notconfined to this interest says

random ised trials are the gold standard for phenomena thatwe are interested in studying from a cause and effect perspectivebut clearly they are not the gold standard if we areinterested in how patients and nurses relate to eachother or if we are interested in how patients live through theexperience of radiotherapy when they have a life threateningillness We have yet to work out how to assess the quality ofalternative approaches to research other than the RCT11

He goes on to suggest that lsquo evidence-based practice isnot exclusively about effectiveness it is about basing prac-tice on the best available evidencersquo11

The diverse origins of problems in health care require abroad interpretation of what counts as valid evidence forpractice and the utilisation of a diverse range of researchmethodologies to generate appropriate evidence Method-ological approaches in this area need to be eclectic enoughto incorporate classical medical and scientific designs andthe emerging qualitative and action-orientated approachesfrom the humanities and social and behavioural sciencesThe development of interdisciplinary research and a greaterunderstanding of the relationship between medical nursingand allied health interventions are also fundamental to thecreation of research methodologies that are relevant andsensitive to the health needs of consumers

There is a small but grow ing amount of literature address-ing the role of qualitative research in evidence-basedpractice which recognises a need to move beyond theeffectiveness of interventions to consider their appropriate-ness and practical feasibility

Lemmer et a l in attempting to conduct a systematicreview in an area of health visiting focusing on the RCT asa lsquogold standardrsquo report on a paucity of trials in this fieldand argue that clinical complexity demands a need to inte-grate qualitative methods into systematic reviews Theyargue that lsquo the comprehensiveness and synthesis of asystematic review are more important to emphasise than

Balancing the evidence 49

whether the literature is outside the clinical rem it of anRCTrsquo12

The need to more fully integrate the results of qualitativeresearch into the systematic process is well stated by Popayand Williams who suggest that lsquo there are many propo-nents of evidence-based decision making w ithin healthcarewho cannot andor w ill not accept that qualitative researchhas an important part to play rsquo13

Popay and Williams assert that the results of qualitativeresearch do more than simply enhance those of quantitativestudies and suggest that qualitative research is capable ofgenerating evidence thatbull explores taken-for-granted practicesbull increases understanding of consumer and clinical

behaviourbull develops interventionsbull illum inates patientrsquos perceptions on qualityappropriate-

nessbull gives guidance to understanding organisational culture

and change management andbull evaluates complex policy initiatives13

Green and Britten stated that

Qualitative research may seem unscientific and anecdotal tomany medical scientists However as the critics of evidence basedmedicine are quick to point out medicine is more than theapplication of scientific rules14

Green and Britten go on to argue that qualitative researchfindings provide rigorous accounts of treatment regimens ineveryday contexts They also contend that there is anincreasing need w ithin the evidence-based practice arena toraise awareness of the fact that different research questionsrequire different kinds of research They are unequivocal intheir assertion that lsquogoodrsquo evidence goes further than theresults of meta-analysis of RCTs

The Cochrane Qualitative Methods Group established in2002 is currently exploring the scope for incorporatingqualitative research into Cochrane reviews There are still nointernationally reviewed guidelines for assessing the qualityof specific qualitative methods and no established proce-dures for ranking or rating qualitative research findingsreported in the literature There have been however a num-ber of attempts to synthesise (as a form of meta-analysis)the results of sim ilar qualitative studies and these are welldescribed by Sandelowski et a l these authors have alsodeveloped an in-depth theoretical approach to the system-atic metasynthesis of qualitative findings that maintain theintegrity of individual studies15 Draw ing on the work ofSandelowski et a l Popay and Williams and Lemmer et a lan approach to qualitative meta-analysis quality assessment

and the development of a quality rating scale for qualitativeresearch results could be used to add appropriateness andfeasibility dimensions to the current effectiveness-orientatedsystematic review process121315

There are signs that the evidence-based practice move-ment is beginning to develop a more comprehensive viewof evidence There are research initiatives attempting toconstruct approaches to assessing and synthesising theresults of interpretative and critical research so that theseforms of evidence can become an integral part of systematicreviews and thus inform practice Several authors havereported on the systematic review of evidence elicitedthrough interpretative and critical approaches to inquiry16ndash

18 An approach to the meta-synthesis of qualitative findingsand the problems associated w ith synthesising the findingsof studies that are essentially context-bound is described byJensen and Allen and Popay and Roen overview a w ide rangeof current initiatives focusing on methods to appraise andsynthesise qualitative research w ithin the framework of thesystematic review process1920

Developing sound and acceptable methodologies toappraise the findings of qualitative research and to synthe-sise the findings of two or more sim ilar studies is bothcomplex and challenging Interesting developments in thisarea have been reported by several authors1721ndash24 Suchevolving approaches to evidence-based practice represent agrow ing body of work supporting the appropriate use ofrelevant evidence in the systematic review process that w illhelp practitioners to perform well in practice and to use theirprofessional judgement in the use of appropriate evidence

Including qualitative data in systematic reviews

Over a period of two years a project utilising participatoryprocesses at three consensus workshops has explored thereview of qualitative evidence and an electronic packagehas been designed to enable reviewers to systematicallyreview qualitative evidence The purpose of this project wasto determ ine how evidence generated through qualitativeresearch could be systematically reviewed and to identifyhow evidence of appropriateness and feasibility could aug-ment evidence of effectiveness in evidence-based healthcare

Project design

The project consisted of four phasesIn Phase 1 acknow ledged experts in qualitative and

action-orientated approaches were invited to attend a con-

50 A Pearson

sensus workshop w ith a view to working w ith the researcherto develop instruments to evaluate and extract data from qualitative research reports

In Phase 2 acknow ledged experts in qualitative andaction-orientated approaches were invited to attend a con-sensus workshop w ith a view to working w ith the researcherto develop a systematic process of extracting and synthesis-ing data from qualitative research reports

In Phase 3 the researcher worked w ith a software devel-oper to develop an electronic system to review qualitativeevidence

In Phase 4 the software developed was piloted w ith theexpert groups and other health professionals attending sys-tematic review training workshops

Outcomes

At the workshops a group of leading Australian qualitativeresearchers were invited to participate in a consensus work-shop to consider how a systematic process of extracting andsynthesising qualitative data can occur to reflect a rigorousprocess equivalent to the existing processes applied to theresults of RCT and other quantitative research while main-taining sensitivity to the contextual nature of qualitativeresearch

More specifically participants were asked tobull design appropriate data extraction toolsbull design appropriate data synthesis tools andbull draft a position statement

N ine of those invited attended the workshops Workshopattendees were Professor Mary FitzGerald University ofNewcastle Associate Professor Jane Stein-Parbury Universityof Technology Professor Colin Holmes James Cook Univer-sity of Northern Queensland Professor M ichael C linton Cur-tin University Professor Desley Hegney University ofSouthern Queensland Dr Ken Walsh The University ofAdelaide Dr Karen Francis Charles Sturt University MrMatt Lew is La Trobe University and Ms Cathy Ward LaTrobe University

Consensus workshops

Expected outcomes were agreed upon at each workshop Itwas also agreed that the workshop outcomes would needto accommodate the nature of qualitative approaches toresearch rather than replicate existing formats The partici-pants emphasised the complexity of interpretative and crit-ical understandings of phenomena but were also aware ofthe need to ensure that outcomes would be practical andusable Participants were also m indful of the complexity ofthe types of research under consideration and wanted to

balance the utility of the outcomes w ith the complexity ofthe material

lsquoMeaningrsquo was identified as a unifying theme of qualita-tive research A list of methodological frameworks was gen-erated and these were then grouped in relation to theirorientation (Appendix I) Judgements were made about thevalue of each methodological position in relation to specificcriteria of meaningfulness appropriateness feasibility andeffectiveness

The participants recognised that they were working w itha w ide range of methodologies each w ith its own strengthsand weaknesses They agreed to develop a lsquomatrixrsquoapproach which was considered to be more appropriatethan a simple hierarchy of evidence With regard to thecentrality of contextual issues in interpretative and criticalresearch the group suggested that this be taken intoaccount by including statements in the summary about thescope of applicability

The group agreed that an overall matrix and the contextstatement should generate a summary statement describingthe strengths and weaknesses of the available evidence andthe associated levels of confidence

Workshop outcomes statement

Consensus was reached on an lsquooutcomes statementrsquo devel-oped collaboratively by the group (Appendix II)

Draw ing on this consensus summary statement proto-type components for a qualitative assessment and reviewinstrument were developed This was named the QualitativeAssessment and Review Instrument (QARI)

Appraising and synthesising qualitative data

Critical appraisal

The central concern in critically appraising experimental orquantifiable data is to lim it bias and thus establish the valid-ity of a study From a quantitative perspective sources ofbias include selection bias performance bias and attritionbias and validity is assessed by establishing the extent towhich a studyrsquos design and conduct address potential bias

This focus on lim iting bias to establish validity is antithet-ical to the philosophical foundations of qualitativeapproaches to inquiry Emden and Sandelowski suggest thatvalidity in quantitative terms measures those things that itpurports to render generalisable In social inquiry howeverthey argue that validity is perceived as criteria of rigour forqualitative research25

There is much dissent in the literature on the appropriate-ness of establishing criteria to assess the validity of qualita-

Balancing the evidence 51

tive research However Pearson takes the view that atransparent approach to appraising qualitative research iscentral to its ongoing credibility transferability and theoret-ical potential26 Denzin and Lincoln concur w ith this conclu-sion and highlight the need for development of a set ofvalidity criteria sensitive to the nature of the qualitativeresearch and its basis in subjectivity27 Popay et a l areunequivocal in their assertion that the development of stan-dards for assessing evidence from qualitative research is bothpossible and desirable28 They provide the follow ing as aguide to common standardsbull evidence of responsiveness to social context and flexibility

of designbull evidence of theoretical or purposeful samplingbull evidence of adequate descriptionbull evidence of data qualitybull evidence of theoretical and conceptual adequacy andbull potential for assessing typicality28

There is a grow ing amount of literature that exam ines theappraisal of qualitative studies and a large number of for-mats are available Draw ing upon this literature and anextensive process of development and piloting Averis andPearson describe a general set of criteria for appraising thevalidity of interpretative and critical research26 These criteriawere incorporated into the critical appraisal scale of the QARIsoftware which was developed by the project

The critical appraisal scale was piloted and refined in threesystematic review training workshops The checklist inAppendix III provides the essential framework for the criticalappraisal of interpretative and critical studies

Data extraction

Data extraction aims to reduce the findings of many studiesinto a single document and summarisebull methodsbull interventions andbull outcomes

Data extraction involves transferring data from the origi-nal paper using an approach agreed upon and standardisedfor the specific review An agreed format is essential tom inim ise error to provide a historical record of decisionsmade about the data in the review and to become the dataset for analysis and synthesis A data extraction instrumentdraw ing on the literature and input from a panel of expertswas developed extensively piloted refined and incorpo-rated into the QARI software (Appendix IV) Based on thestandard approach promoted by the Cochrane Collabora-tion and adopted by the JBI two reviewers are expected toindependently extract data and then confer

Meta-synthesis

The most complex problem in synthesising textual data isagreeing on and communicating techniques to compare thefindings of each study Meta-synthesis relates to the com-bining of separate elements to form a coherent whole Thisinvolves reasoning from the general to the particular usinga process of logical deduction An approach to the meta-synthesis of qualitative data which draws on the literatureand input from a panel of experts was developed for QARIThis involvesbull translating themes metaphors or conceptsbull transferring actual text or summarised text that illustrates

the theme metaphor or concept andbull re-categorising the data obtained to arrive at a

synthesisIn order to pursue this reviewers need to establish the

follow ing before carrying out data synthesisbull their own rules for setting up categoriesbull how to assign findings to categories andbull how to write narrative summaries for each category

The reviewers need to document these decisions and theirrationale in the systematic review report This process isincorporated into the QARI software

The analysis and synthesis of qualitative studies is com-monly termed meta-synthesis and like meta-analysis it isbased on processed data There are major differencesbetween the approach used to synthesise the findings ofRCT and the approach used for qualitative studies Realityfor the qualitative researcher and reviewer is viewed asmultiple and constructed and so undertaking meta-synthesis means that no two reviewers w ill produce exactlythe same results Although meta-synthesis provides only oneinterpretation it aims to capture the essence of the phenom-enon interest

When engaging in the synthesis of the results of qualita-tive studies differing research methods such as phenome-nology ethnography or grounded theory are not m ixed ina single synthesis of all qualitative studies

The aim of meta-synthesis is to portray an accurate inter-pretation of a phenomenon and to compare and contrastthe constructs of individual studies to reach consensus on anew construction of that phenomenon

Meta-synthesis involvesbull identifying findingsbull grouping findings into categories andbull grouping categories into synthesised findings

Findings Findings are conclusions reached by thereviewer(s) after exam ining the results of data analysis (egthemes metaphors) consisting of a statement that relates

52 A Pearson

two or more phenomena variables or circumstances thatmay inform practice

Categories Categories are groups of findings that reflectsim ilar relationships between sim ilar phenomena variablesor circumstances that may inform practice

Synthesised findings Synthesis refers to the combining ofseparate elements to form a coherent whole using logicaldeduction and reasoning from the general to the particularIn QARI a synthesised finding is defined as an overarchingdescription of a group of categorised findings that allow forthe generation of recommendations for practice

Categorising findingsIn order to pursue this reviewers need to establish their ownrules for the follow ing before carrying out data synthesisbull setting up categoriesbull assigning findings to categories andbull writing narrative summaries for each category

Reviewers need to document these decisions and theirrationale in the systematic review report This process isincorporated into the QARI software

The primary reviewer then categorises findings

Synthesised findingsWhen categorisation is complete the reviewers then studythe categories and synthesise these to form a set of synthe-sised findings

Levels of evidence

Current approaches to evaluating evidence utilise a hierar-chy of evidence designed to assess the validity of recom-mendations for clinical guidelines These approaches focuson the effectiveness of treatment and rank only quantitativeevidence according to the rigour of the research designedto lim it bias An approach to categorising the validity ofqualitative evidence which draws on the literature andinput from a panel of experts has been developed forQARI This approach is based on three levels of qualitativeevidence

Unequivoca l The evidence is beyond reasonable doubtand includes findings that are factual directly reportedobserved and not open to challenge

Credible The evidence while interpretative is plausible inlight of the data and theoretical framework Conclusions canbe logically inferred from the data but because the findingsare essentially interpretative these conclusions are open tochallenge

Unsupported Findings are not supported by the data andnone of the other level descriptors apply

These three levels of evidence are incorporated into theQARI software

Levels of applicability

There is little point in accumulating evidence to answer aquestion if it cannot then be used to benefit patients Evi-dence-based practice involves integration of the best avail-able evidence w ith clinical expertise When it comes todeciding whether or not to incorporate a particular activityor intervention into practice some or all of the follow ingconsiderations w ill be relevantbull Is it availablebull Is it affordablebull Is it applicable in the settingbull Would the patientclient be a w illing participant in the

implementation of the interventionbull Were the patients in the studystudies that provided the

evidence sufficiently sim ilar to your own to justify theimplementation of this particular intervention

bull What w ill be the potential benefits for the patientbull What is the potential harm to the patientbull Does this intervention allow for the individual patientrsquos

values and preferencesIn addition to the requirement to define levels of evidence

for practice there is also a need to establish levels of appli-cability The QARI project group developed the lsquoFeasibilityAppropriateness Meaningfulness and Effectivenessrsquo (FAME)scale26 a hierarchy of applicability of evidence that is incor-porated into the QARI software

The QARI levels of applicability of evidence are shown inTable 1

The QARI software

The QARI attempts to establish the results of non-quantita-tive research as appropriate evidence for health-care practi-tioners It was designed to create a system that would enablehealth scientists and health practitioners to review evidencefrom an inclusive position The purpose of the QARI devel-opmental process was to determ ine how evidence gener-ated through qualitative research could be systematicallyreviewed and identify how evidence of appropriatenessmeaningfulness and feasibility could augment evidence ofeffectiveness in evidence-based health care

The QARI software is designed to manage appraise anal-yse and synthesise textual data as part of a systematic reviewof evidence QARI has been designed as a web-based data-base and incorporates a critical appraisal scale data extrac-tion forms a data synthesis function and a reporting function

Balancing the evidence 53

The software has been internationally peer reviewed It hasbeen tested by a group of international systematic reviewersand is currently being piloted by evidence review groups inCanada Australia England and Scotland

The QARI software can be accessed by those who hold alicence by entering the JBI website (http wwwjoan-nabriggseduau) and then clicking on the SUMARI logo

An overview of how QARI is used is presented here todemonstrate the process of appraisal and meta-synthesis

When QARI is opened the main menu is always acrossthe top of the screen (Fig 1)

Reviews

Reviews are the projects to which Studies Extractions andFindings relate The reviews screen lists the reviews availableto the logged on reviewer (Fig 1)

Before a Reviewer can work on a project they must beassigned to the Review either when the Review is first cre-

Table 1 Qualitative Assessment and Review Instrument (QARI) levels of applicability of evidence

Feasibility Appropriateness Meaningfulness Effectiveness

Immediately practicable Acceptable and justifiable w ithin ethical guidelines

Provides a rationale for practice development

Effectiveness established to a degreethat merits application

Practicable w ith lim ited local training or modest additional resources

Acceptable after m inor revision Provides a rationale for local regional or national reform

Effectiveness established to a degree that suggests application

Practicable w ith extensive additional training or resources

Acceptable after major revision Provides a rationale for practice-relevant research

Effectiveness established to a degree that warrants consideration of applying the findings

Practicable w ith significant national reforms

Acceptable after development of new ethical guidelines

Provides a rationale for advocatingchange

Effectiveness established to a lim ited degree

Impracticable Ethically unacceptable Evidence unlikely to make sense to practitioners

Effectiveness not established

Figure 1 Reviews screen listing the reviews available to the logged on reviewer

Balancing the evidence 47

experiences and the nature and norms of the setting andculture in which the health care is being delivered these areall rich sources of evidence to draw upon in making clinicaldecisions

The dom inant orthodoxy of regarding the results of quan-titative research as evidence and all other know ledge assomething other than evidence does not reflect the under-standing of practice held by many clinicians This is notsurprising given the nature and meaning of evidence ineveryday life

Evidence in its most generic sense has been defined asbeing lsquothe available facts circumstances etc supporting orotherw ise a belief proposition etc or indicating whether athing is true or validrsquo3 Evidence in a legal sense has beendefined as being lsquoinformation given personally or drawnfrom a document etc and tending to prove a fact orproposition or statements or proofs adm issible as tes-timony in a law courtrsquo3

For philosophers evidence is understood to be informa-tion bearing on the truth or falsity of a proposition Accord-ing to Audi

A personrsquos evidence is generally taken to be all the informationa person has positive or negative relevant to a proposition Thenotion of evidence used in philosophy thus differs from theordinary notion according to which physical objects such as astrand of hair or a drop of blood counts as evidence Onersquosinformation about such objects could count as evidence in thephilosophical sense4

It is important from a philosophical standpoint to under-stand that the concept of evidence plays a key role in ourunderstanding of know ledge and rationality Traditionallylsquoone has know ledge only when one has a true belief basedon very strong evidencersquo4 Moreover for belief to be rationalit must be based on adequate evidence even when thatevidence is insufficient to ground know ledge

Evidence for health professionals

Some serious consideration has been given to the meaningof evidence in relation to the evidence-based health-caremovement D ixon-Woods et a l for example discuss therelevance of evidence elicited through qualitative researchin systematic reviews and Noblit and Hare describe howethnographic approaches to inquiry generate evidenceappropriate to practice56

According to Humphris the term lsquoevidence-basedrsquo inhealth care lsquoimplies the use and application of researchevidence as a basis on which to make health-care decisionsas opposed to decisions not based on evidencersquo7 Within themainstream health field led by medicine and medical sci-

ence research has been narrow ly confined to the empirico-analytical paradigm focusing on objectivity measurementand statistical significance This process of know ledge gen-eration involves testing a hypothesis or a set of hypothesesby deriving consequences from it and then testing whetherthose consequences hold true by experiment and observa-tion A theory of evidence assists here to the extent that itindicates what relationship should exist between the obser-vation reports and the hypotheses if those reports are toconstitute evidence for the hypotheses

There is some legitimacy in regarding the RCT and otherapproaches that focus on measurement and statistical anal-ysis as the most desirable approach to evidence generationwhen the question relates to cause-and-effect relationshipsHowever health professionals have broader evidence inter-ests that relate to the experience of health illness and healthcare Indeed it is not unusual w ithin the broad field of healthcare to find that the lsquobest availablersquo evidence on a giventopic cannot be reduced to a quantifiable value Expertopinion ndash whether it is expressed by an individual a learnedbody or by a group of experts in the form of a consensusguideline ndash draws on the experience of practitioners Well-designed qualitative research while often not properlyunderstood by researchers grounded in the physical sci-ences constitutes lsquogoodrsquo human science

Pearson argues for a pluralistic approach when consider-ing lsquowhat counts as evidencersquo for health-care practices andEvans and Pearson suggest reviews that include both (oreither) qualitative evidence and quantitative evidence are ofimportance to most practitioners89 However they go on tosuggest that lsquo optimal methods for review ing qualitativeresearch are still evolvingrsquo9

Evidence-based practice

Clinical effectiveness an evidence base for practice

The evidence-based practice movement currently focuses onthe effectiveness of interventions and activities and the termlsquosystematic reviewrsquo is now interpreted as a process thatsummarises and synthesises the results of experimental andother quantitative studies The results of descriptive obser-vational and interpretative inquiry are afforded little if anystatus in most systematic reviews Evans and Pearson aver

It can be argued that the approach of the systematic reviewshould be used for all summaries of the research ensuring thatthe care and rigour that was utilised by the primary researcher ismaintained by the reviewer However this concept has notreceived w ide acceptance and so the focus of most systematicreviews has remained predom inantly fixed on the random isedcontrolled trial9

48 A Pearson

Evans and Pearson go on to argue that the systematicreview process should be expanded to include findings gen-erated from all forms of rigorous research as well as thosefrom RCTs The systematic review process originated in thefield of organisational psychology from which medicalresearch took it and refined it

Writing in 1982 Salipente et a l clearly describe how theprocess was conceptualised to synthesise research groundedin any tradition10 Thus w ithin this context Evans and Pear-sonrsquos plea to lsquoexpandrsquo the rem it of systematic reviews couldbe better expressed as returning to the original nature of thesystematic review

The systematic review

The core of evidence-based practice is the systematic reviewof the literature on a particular condition intervention orissue The systematic review is essentially an analysis of all ofthe available literature (that is evidence) and a judgementof the effectiveness or otherw ise of a particular practiceCurrently the systematic review involves the follow ingsteps1 The development of a rigorous proposal or protocol is

vital for a high quality systematic review The reviewprotocol provides a predeterm ined plan to ensure scien-tific thoroughness and the m inim isation of potential biasIt also allows for periodic updating of the review ifnecessary

2 The protocol should state in detail the questions orhypotheses to be discussed in the review Questionsregarding the patients setting interventions and out-comes to be investigated should be specific

3 The protocol must describe the criteria that w ill be usedto select the literature The inclusion criteria shouldaddress the participants of the primary studies the inter-vention and the outcomes In addition to this it shouldalso specify what research methodologies w ill be consid-ered for inclusion in the review (eg RCTs clinical trialscase studies)

4 The protocol should provide a detailed strategy that w illbe used to identify all relevant literature w ithin a specifiedtime frame This should include databases and bibliogra-phies that w ill be searched and the search terms that w illbe used

5 Critical appraisal of the studies retrieved is important toassess the quality of the research m inim ising the risk ofan inconclusive review w ith excessive variation in studyquality The protocol must therefore describe how thequality of primary studies w ill be assessed and any exclu-sion criteria based on quality considerations

6 It is necessary to extract data from the primary researchregarding the participants the intervention the outcomemeasures and the results

7 Statistical analysis (meta analysis) may or may not be usedand w ill depend on the nature and quality of studiesincluded in the review Although it may not be possibleto state exactly what analysis w ill be undertaken thegeneral approach should be included in the protocolWhen statistical analysis is not possible current practiceis to develop a narrative summary

Pearson in arguing that evidence-based practice includesan interest in research on clinical effectiveness but is notconfined to this interest says

random ised trials are the gold standard for phenomena thatwe are interested in studying from a cause and effect perspectivebut clearly they are not the gold standard if we areinterested in how patients and nurses relate to eachother or if we are interested in how patients live through theexperience of radiotherapy when they have a life threateningillness We have yet to work out how to assess the quality ofalternative approaches to research other than the RCT11

He goes on to suggest that lsquo evidence-based practice isnot exclusively about effectiveness it is about basing prac-tice on the best available evidencersquo11

The diverse origins of problems in health care require abroad interpretation of what counts as valid evidence forpractice and the utilisation of a diverse range of researchmethodologies to generate appropriate evidence Method-ological approaches in this area need to be eclectic enoughto incorporate classical medical and scientific designs andthe emerging qualitative and action-orientated approachesfrom the humanities and social and behavioural sciencesThe development of interdisciplinary research and a greaterunderstanding of the relationship between medical nursingand allied health interventions are also fundamental to thecreation of research methodologies that are relevant andsensitive to the health needs of consumers

There is a small but grow ing amount of literature address-ing the role of qualitative research in evidence-basedpractice which recognises a need to move beyond theeffectiveness of interventions to consider their appropriate-ness and practical feasibility

Lemmer et a l in attempting to conduct a systematicreview in an area of health visiting focusing on the RCT asa lsquogold standardrsquo report on a paucity of trials in this fieldand argue that clinical complexity demands a need to inte-grate qualitative methods into systematic reviews Theyargue that lsquo the comprehensiveness and synthesis of asystematic review are more important to emphasise than

Balancing the evidence 49

whether the literature is outside the clinical rem it of anRCTrsquo12

The need to more fully integrate the results of qualitativeresearch into the systematic process is well stated by Popayand Williams who suggest that lsquo there are many propo-nents of evidence-based decision making w ithin healthcarewho cannot andor w ill not accept that qualitative researchhas an important part to play rsquo13

Popay and Williams assert that the results of qualitativeresearch do more than simply enhance those of quantitativestudies and suggest that qualitative research is capable ofgenerating evidence thatbull explores taken-for-granted practicesbull increases understanding of consumer and clinical

behaviourbull develops interventionsbull illum inates patientrsquos perceptions on qualityappropriate-

nessbull gives guidance to understanding organisational culture

and change management andbull evaluates complex policy initiatives13

Green and Britten stated that

Qualitative research may seem unscientific and anecdotal tomany medical scientists However as the critics of evidence basedmedicine are quick to point out medicine is more than theapplication of scientific rules14

Green and Britten go on to argue that qualitative researchfindings provide rigorous accounts of treatment regimens ineveryday contexts They also contend that there is anincreasing need w ithin the evidence-based practice arena toraise awareness of the fact that different research questionsrequire different kinds of research They are unequivocal intheir assertion that lsquogoodrsquo evidence goes further than theresults of meta-analysis of RCTs

The Cochrane Qualitative Methods Group established in2002 is currently exploring the scope for incorporatingqualitative research into Cochrane reviews There are still nointernationally reviewed guidelines for assessing the qualityof specific qualitative methods and no established proce-dures for ranking or rating qualitative research findingsreported in the literature There have been however a num-ber of attempts to synthesise (as a form of meta-analysis)the results of sim ilar qualitative studies and these are welldescribed by Sandelowski et a l these authors have alsodeveloped an in-depth theoretical approach to the system-atic metasynthesis of qualitative findings that maintain theintegrity of individual studies15 Draw ing on the work ofSandelowski et a l Popay and Williams and Lemmer et a lan approach to qualitative meta-analysis quality assessment

and the development of a quality rating scale for qualitativeresearch results could be used to add appropriateness andfeasibility dimensions to the current effectiveness-orientatedsystematic review process121315

There are signs that the evidence-based practice move-ment is beginning to develop a more comprehensive viewof evidence There are research initiatives attempting toconstruct approaches to assessing and synthesising theresults of interpretative and critical research so that theseforms of evidence can become an integral part of systematicreviews and thus inform practice Several authors havereported on the systematic review of evidence elicitedthrough interpretative and critical approaches to inquiry16ndash

18 An approach to the meta-synthesis of qualitative findingsand the problems associated w ith synthesising the findingsof studies that are essentially context-bound is described byJensen and Allen and Popay and Roen overview a w ide rangeof current initiatives focusing on methods to appraise andsynthesise qualitative research w ithin the framework of thesystematic review process1920

Developing sound and acceptable methodologies toappraise the findings of qualitative research and to synthe-sise the findings of two or more sim ilar studies is bothcomplex and challenging Interesting developments in thisarea have been reported by several authors1721ndash24 Suchevolving approaches to evidence-based practice represent agrow ing body of work supporting the appropriate use ofrelevant evidence in the systematic review process that w illhelp practitioners to perform well in practice and to use theirprofessional judgement in the use of appropriate evidence

Including qualitative data in systematic reviews

Over a period of two years a project utilising participatoryprocesses at three consensus workshops has explored thereview of qualitative evidence and an electronic packagehas been designed to enable reviewers to systematicallyreview qualitative evidence The purpose of this project wasto determ ine how evidence generated through qualitativeresearch could be systematically reviewed and to identifyhow evidence of appropriateness and feasibility could aug-ment evidence of effectiveness in evidence-based healthcare

Project design

The project consisted of four phasesIn Phase 1 acknow ledged experts in qualitative and

action-orientated approaches were invited to attend a con-

50 A Pearson

sensus workshop w ith a view to working w ith the researcherto develop instruments to evaluate and extract data from qualitative research reports

In Phase 2 acknow ledged experts in qualitative andaction-orientated approaches were invited to attend a con-sensus workshop w ith a view to working w ith the researcherto develop a systematic process of extracting and synthesis-ing data from qualitative research reports

In Phase 3 the researcher worked w ith a software devel-oper to develop an electronic system to review qualitativeevidence

In Phase 4 the software developed was piloted w ith theexpert groups and other health professionals attending sys-tematic review training workshops

Outcomes

At the workshops a group of leading Australian qualitativeresearchers were invited to participate in a consensus work-shop to consider how a systematic process of extracting andsynthesising qualitative data can occur to reflect a rigorousprocess equivalent to the existing processes applied to theresults of RCT and other quantitative research while main-taining sensitivity to the contextual nature of qualitativeresearch

More specifically participants were asked tobull design appropriate data extraction toolsbull design appropriate data synthesis tools andbull draft a position statement

N ine of those invited attended the workshops Workshopattendees were Professor Mary FitzGerald University ofNewcastle Associate Professor Jane Stein-Parbury Universityof Technology Professor Colin Holmes James Cook Univer-sity of Northern Queensland Professor M ichael C linton Cur-tin University Professor Desley Hegney University ofSouthern Queensland Dr Ken Walsh The University ofAdelaide Dr Karen Francis Charles Sturt University MrMatt Lew is La Trobe University and Ms Cathy Ward LaTrobe University

Consensus workshops

Expected outcomes were agreed upon at each workshop Itwas also agreed that the workshop outcomes would needto accommodate the nature of qualitative approaches toresearch rather than replicate existing formats The partici-pants emphasised the complexity of interpretative and crit-ical understandings of phenomena but were also aware ofthe need to ensure that outcomes would be practical andusable Participants were also m indful of the complexity ofthe types of research under consideration and wanted to

balance the utility of the outcomes w ith the complexity ofthe material

lsquoMeaningrsquo was identified as a unifying theme of qualita-tive research A list of methodological frameworks was gen-erated and these were then grouped in relation to theirorientation (Appendix I) Judgements were made about thevalue of each methodological position in relation to specificcriteria of meaningfulness appropriateness feasibility andeffectiveness

The participants recognised that they were working w itha w ide range of methodologies each w ith its own strengthsand weaknesses They agreed to develop a lsquomatrixrsquoapproach which was considered to be more appropriatethan a simple hierarchy of evidence With regard to thecentrality of contextual issues in interpretative and criticalresearch the group suggested that this be taken intoaccount by including statements in the summary about thescope of applicability

The group agreed that an overall matrix and the contextstatement should generate a summary statement describingthe strengths and weaknesses of the available evidence andthe associated levels of confidence

Workshop outcomes statement

Consensus was reached on an lsquooutcomes statementrsquo devel-oped collaboratively by the group (Appendix II)

Draw ing on this consensus summary statement proto-type components for a qualitative assessment and reviewinstrument were developed This was named the QualitativeAssessment and Review Instrument (QARI)

Appraising and synthesising qualitative data

Critical appraisal

The central concern in critically appraising experimental orquantifiable data is to lim it bias and thus establish the valid-ity of a study From a quantitative perspective sources ofbias include selection bias performance bias and attritionbias and validity is assessed by establishing the extent towhich a studyrsquos design and conduct address potential bias

This focus on lim iting bias to establish validity is antithet-ical to the philosophical foundations of qualitativeapproaches to inquiry Emden and Sandelowski suggest thatvalidity in quantitative terms measures those things that itpurports to render generalisable In social inquiry howeverthey argue that validity is perceived as criteria of rigour forqualitative research25

There is much dissent in the literature on the appropriate-ness of establishing criteria to assess the validity of qualita-

Balancing the evidence 51

tive research However Pearson takes the view that atransparent approach to appraising qualitative research iscentral to its ongoing credibility transferability and theoret-ical potential26 Denzin and Lincoln concur w ith this conclu-sion and highlight the need for development of a set ofvalidity criteria sensitive to the nature of the qualitativeresearch and its basis in subjectivity27 Popay et a l areunequivocal in their assertion that the development of stan-dards for assessing evidence from qualitative research is bothpossible and desirable28 They provide the follow ing as aguide to common standardsbull evidence of responsiveness to social context and flexibility

of designbull evidence of theoretical or purposeful samplingbull evidence of adequate descriptionbull evidence of data qualitybull evidence of theoretical and conceptual adequacy andbull potential for assessing typicality28

There is a grow ing amount of literature that exam ines theappraisal of qualitative studies and a large number of for-mats are available Draw ing upon this literature and anextensive process of development and piloting Averis andPearson describe a general set of criteria for appraising thevalidity of interpretative and critical research26 These criteriawere incorporated into the critical appraisal scale of the QARIsoftware which was developed by the project

The critical appraisal scale was piloted and refined in threesystematic review training workshops The checklist inAppendix III provides the essential framework for the criticalappraisal of interpretative and critical studies

Data extraction

Data extraction aims to reduce the findings of many studiesinto a single document and summarisebull methodsbull interventions andbull outcomes

Data extraction involves transferring data from the origi-nal paper using an approach agreed upon and standardisedfor the specific review An agreed format is essential tom inim ise error to provide a historical record of decisionsmade about the data in the review and to become the dataset for analysis and synthesis A data extraction instrumentdraw ing on the literature and input from a panel of expertswas developed extensively piloted refined and incorpo-rated into the QARI software (Appendix IV) Based on thestandard approach promoted by the Cochrane Collabora-tion and adopted by the JBI two reviewers are expected toindependently extract data and then confer

Meta-synthesis

The most complex problem in synthesising textual data isagreeing on and communicating techniques to compare thefindings of each study Meta-synthesis relates to the com-bining of separate elements to form a coherent whole Thisinvolves reasoning from the general to the particular usinga process of logical deduction An approach to the meta-synthesis of qualitative data which draws on the literatureand input from a panel of experts was developed for QARIThis involvesbull translating themes metaphors or conceptsbull transferring actual text or summarised text that illustrates

the theme metaphor or concept andbull re-categorising the data obtained to arrive at a

synthesisIn order to pursue this reviewers need to establish the

follow ing before carrying out data synthesisbull their own rules for setting up categoriesbull how to assign findings to categories andbull how to write narrative summaries for each category

The reviewers need to document these decisions and theirrationale in the systematic review report This process isincorporated into the QARI software

The analysis and synthesis of qualitative studies is com-monly termed meta-synthesis and like meta-analysis it isbased on processed data There are major differencesbetween the approach used to synthesise the findings ofRCT and the approach used for qualitative studies Realityfor the qualitative researcher and reviewer is viewed asmultiple and constructed and so undertaking meta-synthesis means that no two reviewers w ill produce exactlythe same results Although meta-synthesis provides only oneinterpretation it aims to capture the essence of the phenom-enon interest

When engaging in the synthesis of the results of qualita-tive studies differing research methods such as phenome-nology ethnography or grounded theory are not m ixed ina single synthesis of all qualitative studies

The aim of meta-synthesis is to portray an accurate inter-pretation of a phenomenon and to compare and contrastthe constructs of individual studies to reach consensus on anew construction of that phenomenon

Meta-synthesis involvesbull identifying findingsbull grouping findings into categories andbull grouping categories into synthesised findings

Findings Findings are conclusions reached by thereviewer(s) after exam ining the results of data analysis (egthemes metaphors) consisting of a statement that relates

52 A Pearson

two or more phenomena variables or circumstances thatmay inform practice

Categories Categories are groups of findings that reflectsim ilar relationships between sim ilar phenomena variablesor circumstances that may inform practice

Synthesised findings Synthesis refers to the combining ofseparate elements to form a coherent whole using logicaldeduction and reasoning from the general to the particularIn QARI a synthesised finding is defined as an overarchingdescription of a group of categorised findings that allow forthe generation of recommendations for practice

Categorising findingsIn order to pursue this reviewers need to establish their ownrules for the follow ing before carrying out data synthesisbull setting up categoriesbull assigning findings to categories andbull writing narrative summaries for each category

Reviewers need to document these decisions and theirrationale in the systematic review report This process isincorporated into the QARI software

The primary reviewer then categorises findings

Synthesised findingsWhen categorisation is complete the reviewers then studythe categories and synthesise these to form a set of synthe-sised findings

Levels of evidence

Current approaches to evaluating evidence utilise a hierar-chy of evidence designed to assess the validity of recom-mendations for clinical guidelines These approaches focuson the effectiveness of treatment and rank only quantitativeevidence according to the rigour of the research designedto lim it bias An approach to categorising the validity ofqualitative evidence which draws on the literature andinput from a panel of experts has been developed forQARI This approach is based on three levels of qualitativeevidence

Unequivoca l The evidence is beyond reasonable doubtand includes findings that are factual directly reportedobserved and not open to challenge

Credible The evidence while interpretative is plausible inlight of the data and theoretical framework Conclusions canbe logically inferred from the data but because the findingsare essentially interpretative these conclusions are open tochallenge

Unsupported Findings are not supported by the data andnone of the other level descriptors apply

These three levels of evidence are incorporated into theQARI software

Levels of applicability

There is little point in accumulating evidence to answer aquestion if it cannot then be used to benefit patients Evi-dence-based practice involves integration of the best avail-able evidence w ith clinical expertise When it comes todeciding whether or not to incorporate a particular activityor intervention into practice some or all of the follow ingconsiderations w ill be relevantbull Is it availablebull Is it affordablebull Is it applicable in the settingbull Would the patientclient be a w illing participant in the

implementation of the interventionbull Were the patients in the studystudies that provided the

evidence sufficiently sim ilar to your own to justify theimplementation of this particular intervention

bull What w ill be the potential benefits for the patientbull What is the potential harm to the patientbull Does this intervention allow for the individual patientrsquos

values and preferencesIn addition to the requirement to define levels of evidence

for practice there is also a need to establish levels of appli-cability The QARI project group developed the lsquoFeasibilityAppropriateness Meaningfulness and Effectivenessrsquo (FAME)scale26 a hierarchy of applicability of evidence that is incor-porated into the QARI software

The QARI levels of applicability of evidence are shown inTable 1

The QARI software

The QARI attempts to establish the results of non-quantita-tive research as appropriate evidence for health-care practi-tioners It was designed to create a system that would enablehealth scientists and health practitioners to review evidencefrom an inclusive position The purpose of the QARI devel-opmental process was to determ ine how evidence gener-ated through qualitative research could be systematicallyreviewed and identify how evidence of appropriatenessmeaningfulness and feasibility could augment evidence ofeffectiveness in evidence-based health care

The QARI software is designed to manage appraise anal-yse and synthesise textual data as part of a systematic reviewof evidence QARI has been designed as a web-based data-base and incorporates a critical appraisal scale data extrac-tion forms a data synthesis function and a reporting function

Balancing the evidence 53

The software has been internationally peer reviewed It hasbeen tested by a group of international systematic reviewersand is currently being piloted by evidence review groups inCanada Australia England and Scotland

The QARI software can be accessed by those who hold alicence by entering the JBI website (http wwwjoan-nabriggseduau) and then clicking on the SUMARI logo

An overview of how QARI is used is presented here todemonstrate the process of appraisal and meta-synthesis

When QARI is opened the main menu is always acrossthe top of the screen (Fig 1)

Reviews

Reviews are the projects to which Studies Extractions andFindings relate The reviews screen lists the reviews availableto the logged on reviewer (Fig 1)

Before a Reviewer can work on a project they must beassigned to the Review either when the Review is first cre-

Table 1 Qualitative Assessment and Review Instrument (QARI) levels of applicability of evidence

Feasibility Appropriateness Meaningfulness Effectiveness

Immediately practicable Acceptable and justifiable w ithin ethical guidelines

Provides a rationale for practice development

Effectiveness established to a degreethat merits application

Practicable w ith lim ited local training or modest additional resources

Acceptable after m inor revision Provides a rationale for local regional or national reform

Effectiveness established to a degree that suggests application

Practicable w ith extensive additional training or resources

Acceptable after major revision Provides a rationale for practice-relevant research

Effectiveness established to a degree that warrants consideration of applying the findings

Practicable w ith significant national reforms

Acceptable after development of new ethical guidelines

Provides a rationale for advocatingchange

Effectiveness established to a lim ited degree

Impracticable Ethically unacceptable Evidence unlikely to make sense to practitioners

Effectiveness not established

Figure 1 Reviews screen listing the reviews available to the logged on reviewer

48 A Pearson

Evans and Pearson go on to argue that the systematicreview process should be expanded to include findings gen-erated from all forms of rigorous research as well as thosefrom RCTs The systematic review process originated in thefield of organisational psychology from which medicalresearch took it and refined it

Writing in 1982 Salipente et a l clearly describe how theprocess was conceptualised to synthesise research groundedin any tradition10 Thus w ithin this context Evans and Pear-sonrsquos plea to lsquoexpandrsquo the rem it of systematic reviews couldbe better expressed as returning to the original nature of thesystematic review

The systematic review

The core of evidence-based practice is the systematic reviewof the literature on a particular condition intervention orissue The systematic review is essentially an analysis of all ofthe available literature (that is evidence) and a judgementof the effectiveness or otherw ise of a particular practiceCurrently the systematic review involves the follow ingsteps1 The development of a rigorous proposal or protocol is

vital for a high quality systematic review The reviewprotocol provides a predeterm ined plan to ensure scien-tific thoroughness and the m inim isation of potential biasIt also allows for periodic updating of the review ifnecessary

2 The protocol should state in detail the questions orhypotheses to be discussed in the review Questionsregarding the patients setting interventions and out-comes to be investigated should be specific

3 The protocol must describe the criteria that w ill be usedto select the literature The inclusion criteria shouldaddress the participants of the primary studies the inter-vention and the outcomes In addition to this it shouldalso specify what research methodologies w ill be consid-ered for inclusion in the review (eg RCTs clinical trialscase studies)

4 The protocol should provide a detailed strategy that w illbe used to identify all relevant literature w ithin a specifiedtime frame This should include databases and bibliogra-phies that w ill be searched and the search terms that w illbe used

5 Critical appraisal of the studies retrieved is important toassess the quality of the research m inim ising the risk ofan inconclusive review w ith excessive variation in studyquality The protocol must therefore describe how thequality of primary studies w ill be assessed and any exclu-sion criteria based on quality considerations

6 It is necessary to extract data from the primary researchregarding the participants the intervention the outcomemeasures and the results

7 Statistical analysis (meta analysis) may or may not be usedand w ill depend on the nature and quality of studiesincluded in the review Although it may not be possibleto state exactly what analysis w ill be undertaken thegeneral approach should be included in the protocolWhen statistical analysis is not possible current practiceis to develop a narrative summary

Pearson in arguing that evidence-based practice includesan interest in research on clinical effectiveness but is notconfined to this interest says

random ised trials are the gold standard for phenomena thatwe are interested in studying from a cause and effect perspectivebut clearly they are not the gold standard if we areinterested in how patients and nurses relate to eachother or if we are interested in how patients live through theexperience of radiotherapy when they have a life threateningillness We have yet to work out how to assess the quality ofalternative approaches to research other than the RCT11

He goes on to suggest that lsquo evidence-based practice isnot exclusively about effectiveness it is about basing prac-tice on the best available evidencersquo11

The diverse origins of problems in health care require abroad interpretation of what counts as valid evidence forpractice and the utilisation of a diverse range of researchmethodologies to generate appropriate evidence Method-ological approaches in this area need to be eclectic enoughto incorporate classical medical and scientific designs andthe emerging qualitative and action-orientated approachesfrom the humanities and social and behavioural sciencesThe development of interdisciplinary research and a greaterunderstanding of the relationship between medical nursingand allied health interventions are also fundamental to thecreation of research methodologies that are relevant andsensitive to the health needs of consumers

There is a small but grow ing amount of literature address-ing the role of qualitative research in evidence-basedpractice which recognises a need to move beyond theeffectiveness of interventions to consider their appropriate-ness and practical feasibility

Lemmer et a l in attempting to conduct a systematicreview in an area of health visiting focusing on the RCT asa lsquogold standardrsquo report on a paucity of trials in this fieldand argue that clinical complexity demands a need to inte-grate qualitative methods into systematic reviews Theyargue that lsquo the comprehensiveness and synthesis of asystematic review are more important to emphasise than

Balancing the evidence 49

whether the literature is outside the clinical rem it of anRCTrsquo12

The need to more fully integrate the results of qualitativeresearch into the systematic process is well stated by Popayand Williams who suggest that lsquo there are many propo-nents of evidence-based decision making w ithin healthcarewho cannot andor w ill not accept that qualitative researchhas an important part to play rsquo13

Popay and Williams assert that the results of qualitativeresearch do more than simply enhance those of quantitativestudies and suggest that qualitative research is capable ofgenerating evidence thatbull explores taken-for-granted practicesbull increases understanding of consumer and clinical

behaviourbull develops interventionsbull illum inates patientrsquos perceptions on qualityappropriate-

nessbull gives guidance to understanding organisational culture

and change management andbull evaluates complex policy initiatives13

Green and Britten stated that

Qualitative research may seem unscientific and anecdotal tomany medical scientists However as the critics of evidence basedmedicine are quick to point out medicine is more than theapplication of scientific rules14

Green and Britten go on to argue that qualitative researchfindings provide rigorous accounts of treatment regimens ineveryday contexts They also contend that there is anincreasing need w ithin the evidence-based practice arena toraise awareness of the fact that different research questionsrequire different kinds of research They are unequivocal intheir assertion that lsquogoodrsquo evidence goes further than theresults of meta-analysis of RCTs

The Cochrane Qualitative Methods Group established in2002 is currently exploring the scope for incorporatingqualitative research into Cochrane reviews There are still nointernationally reviewed guidelines for assessing the qualityof specific qualitative methods and no established proce-dures for ranking or rating qualitative research findingsreported in the literature There have been however a num-ber of attempts to synthesise (as a form of meta-analysis)the results of sim ilar qualitative studies and these are welldescribed by Sandelowski et a l these authors have alsodeveloped an in-depth theoretical approach to the system-atic metasynthesis of qualitative findings that maintain theintegrity of individual studies15 Draw ing on the work ofSandelowski et a l Popay and Williams and Lemmer et a lan approach to qualitative meta-analysis quality assessment

and the development of a quality rating scale for qualitativeresearch results could be used to add appropriateness andfeasibility dimensions to the current effectiveness-orientatedsystematic review process121315

There are signs that the evidence-based practice move-ment is beginning to develop a more comprehensive viewof evidence There are research initiatives attempting toconstruct approaches to assessing and synthesising theresults of interpretative and critical research so that theseforms of evidence can become an integral part of systematicreviews and thus inform practice Several authors havereported on the systematic review of evidence elicitedthrough interpretative and critical approaches to inquiry16ndash

18 An approach to the meta-synthesis of qualitative findingsand the problems associated w ith synthesising the findingsof studies that are essentially context-bound is described byJensen and Allen and Popay and Roen overview a w ide rangeof current initiatives focusing on methods to appraise andsynthesise qualitative research w ithin the framework of thesystematic review process1920

Developing sound and acceptable methodologies toappraise the findings of qualitative research and to synthe-sise the findings of two or more sim ilar studies is bothcomplex and challenging Interesting developments in thisarea have been reported by several authors1721ndash24 Suchevolving approaches to evidence-based practice represent agrow ing body of work supporting the appropriate use ofrelevant evidence in the systematic review process that w illhelp practitioners to perform well in practice and to use theirprofessional judgement in the use of appropriate evidence

Including qualitative data in systematic reviews

Over a period of two years a project utilising participatoryprocesses at three consensus workshops has explored thereview of qualitative evidence and an electronic packagehas been designed to enable reviewers to systematicallyreview qualitative evidence The purpose of this project wasto determ ine how evidence generated through qualitativeresearch could be systematically reviewed and to identifyhow evidence of appropriateness and feasibility could aug-ment evidence of effectiveness in evidence-based healthcare

Project design

The project consisted of four phasesIn Phase 1 acknow ledged experts in qualitative and

action-orientated approaches were invited to attend a con-

50 A Pearson

sensus workshop w ith a view to working w ith the researcherto develop instruments to evaluate and extract data from qualitative research reports

In Phase 2 acknow ledged experts in qualitative andaction-orientated approaches were invited to attend a con-sensus workshop w ith a view to working w ith the researcherto develop a systematic process of extracting and synthesis-ing data from qualitative research reports

In Phase 3 the researcher worked w ith a software devel-oper to develop an electronic system to review qualitativeevidence

In Phase 4 the software developed was piloted w ith theexpert groups and other health professionals attending sys-tematic review training workshops

Outcomes

At the workshops a group of leading Australian qualitativeresearchers were invited to participate in a consensus work-shop to consider how a systematic process of extracting andsynthesising qualitative data can occur to reflect a rigorousprocess equivalent to the existing processes applied to theresults of RCT and other quantitative research while main-taining sensitivity to the contextual nature of qualitativeresearch

More specifically participants were asked tobull design appropriate data extraction toolsbull design appropriate data synthesis tools andbull draft a position statement

N ine of those invited attended the workshops Workshopattendees were Professor Mary FitzGerald University ofNewcastle Associate Professor Jane Stein-Parbury Universityof Technology Professor Colin Holmes James Cook Univer-sity of Northern Queensland Professor M ichael C linton Cur-tin University Professor Desley Hegney University ofSouthern Queensland Dr Ken Walsh The University ofAdelaide Dr Karen Francis Charles Sturt University MrMatt Lew is La Trobe University and Ms Cathy Ward LaTrobe University

Consensus workshops

Expected outcomes were agreed upon at each workshop Itwas also agreed that the workshop outcomes would needto accommodate the nature of qualitative approaches toresearch rather than replicate existing formats The partici-pants emphasised the complexity of interpretative and crit-ical understandings of phenomena but were also aware ofthe need to ensure that outcomes would be practical andusable Participants were also m indful of the complexity ofthe types of research under consideration and wanted to

balance the utility of the outcomes w ith the complexity ofthe material

lsquoMeaningrsquo was identified as a unifying theme of qualita-tive research A list of methodological frameworks was gen-erated and these were then grouped in relation to theirorientation (Appendix I) Judgements were made about thevalue of each methodological position in relation to specificcriteria of meaningfulness appropriateness feasibility andeffectiveness

The participants recognised that they were working w itha w ide range of methodologies each w ith its own strengthsand weaknesses They agreed to develop a lsquomatrixrsquoapproach which was considered to be more appropriatethan a simple hierarchy of evidence With regard to thecentrality of contextual issues in interpretative and criticalresearch the group suggested that this be taken intoaccount by including statements in the summary about thescope of applicability

The group agreed that an overall matrix and the contextstatement should generate a summary statement describingthe strengths and weaknesses of the available evidence andthe associated levels of confidence

Workshop outcomes statement

Consensus was reached on an lsquooutcomes statementrsquo devel-oped collaboratively by the group (Appendix II)

Draw ing on this consensus summary statement proto-type components for a qualitative assessment and reviewinstrument were developed This was named the QualitativeAssessment and Review Instrument (QARI)

Appraising and synthesising qualitative data

Critical appraisal

The central concern in critically appraising experimental orquantifiable data is to lim it bias and thus establish the valid-ity of a study From a quantitative perspective sources ofbias include selection bias performance bias and attritionbias and validity is assessed by establishing the extent towhich a studyrsquos design and conduct address potential bias

This focus on lim iting bias to establish validity is antithet-ical to the philosophical foundations of qualitativeapproaches to inquiry Emden and Sandelowski suggest thatvalidity in quantitative terms measures those things that itpurports to render generalisable In social inquiry howeverthey argue that validity is perceived as criteria of rigour forqualitative research25

There is much dissent in the literature on the appropriate-ness of establishing criteria to assess the validity of qualita-

Balancing the evidence 51

tive research However Pearson takes the view that atransparent approach to appraising qualitative research iscentral to its ongoing credibility transferability and theoret-ical potential26 Denzin and Lincoln concur w ith this conclu-sion and highlight the need for development of a set ofvalidity criteria sensitive to the nature of the qualitativeresearch and its basis in subjectivity27 Popay et a l areunequivocal in their assertion that the development of stan-dards for assessing evidence from qualitative research is bothpossible and desirable28 They provide the follow ing as aguide to common standardsbull evidence of responsiveness to social context and flexibility

of designbull evidence of theoretical or purposeful samplingbull evidence of adequate descriptionbull evidence of data qualitybull evidence of theoretical and conceptual adequacy andbull potential for assessing typicality28

There is a grow ing amount of literature that exam ines theappraisal of qualitative studies and a large number of for-mats are available Draw ing upon this literature and anextensive process of development and piloting Averis andPearson describe a general set of criteria for appraising thevalidity of interpretative and critical research26 These criteriawere incorporated into the critical appraisal scale of the QARIsoftware which was developed by the project

The critical appraisal scale was piloted and refined in threesystematic review training workshops The checklist inAppendix III provides the essential framework for the criticalappraisal of interpretative and critical studies

Data extraction

Data extraction aims to reduce the findings of many studiesinto a single document and summarisebull methodsbull interventions andbull outcomes

Data extraction involves transferring data from the origi-nal paper using an approach agreed upon and standardisedfor the specific review An agreed format is essential tom inim ise error to provide a historical record of decisionsmade about the data in the review and to become the dataset for analysis and synthesis A data extraction instrumentdraw ing on the literature and input from a panel of expertswas developed extensively piloted refined and incorpo-rated into the QARI software (Appendix IV) Based on thestandard approach promoted by the Cochrane Collabora-tion and adopted by the JBI two reviewers are expected toindependently extract data and then confer

Meta-synthesis

The most complex problem in synthesising textual data isagreeing on and communicating techniques to compare thefindings of each study Meta-synthesis relates to the com-bining of separate elements to form a coherent whole Thisinvolves reasoning from the general to the particular usinga process of logical deduction An approach to the meta-synthesis of qualitative data which draws on the literatureand input from a panel of experts was developed for QARIThis involvesbull translating themes metaphors or conceptsbull transferring actual text or summarised text that illustrates

the theme metaphor or concept andbull re-categorising the data obtained to arrive at a

synthesisIn order to pursue this reviewers need to establish the

follow ing before carrying out data synthesisbull their own rules for setting up categoriesbull how to assign findings to categories andbull how to write narrative summaries for each category

The reviewers need to document these decisions and theirrationale in the systematic review report This process isincorporated into the QARI software

The analysis and synthesis of qualitative studies is com-monly termed meta-synthesis and like meta-analysis it isbased on processed data There are major differencesbetween the approach used to synthesise the findings ofRCT and the approach used for qualitative studies Realityfor the qualitative researcher and reviewer is viewed asmultiple and constructed and so undertaking meta-synthesis means that no two reviewers w ill produce exactlythe same results Although meta-synthesis provides only oneinterpretation it aims to capture the essence of the phenom-enon interest

When engaging in the synthesis of the results of qualita-tive studies differing research methods such as phenome-nology ethnography or grounded theory are not m ixed ina single synthesis of all qualitative studies

The aim of meta-synthesis is to portray an accurate inter-pretation of a phenomenon and to compare and contrastthe constructs of individual studies to reach consensus on anew construction of that phenomenon

Meta-synthesis involvesbull identifying findingsbull grouping findings into categories andbull grouping categories into synthesised findings

Findings Findings are conclusions reached by thereviewer(s) after exam ining the results of data analysis (egthemes metaphors) consisting of a statement that relates

52 A Pearson

two or more phenomena variables or circumstances thatmay inform practice

Categories Categories are groups of findings that reflectsim ilar relationships between sim ilar phenomena variablesor circumstances that may inform practice

Synthesised findings Synthesis refers to the combining ofseparate elements to form a coherent whole using logicaldeduction and reasoning from the general to the particularIn QARI a synthesised finding is defined as an overarchingdescription of a group of categorised findings that allow forthe generation of recommendations for practice

Categorising findingsIn order to pursue this reviewers need to establish their ownrules for the follow ing before carrying out data synthesisbull setting up categoriesbull assigning findings to categories andbull writing narrative summaries for each category

Reviewers need to document these decisions and theirrationale in the systematic review report This process isincorporated into the QARI software

The primary reviewer then categorises findings

Synthesised findingsWhen categorisation is complete the reviewers then studythe categories and synthesise these to form a set of synthe-sised findings

Levels of evidence

Current approaches to evaluating evidence utilise a hierar-chy of evidence designed to assess the validity of recom-mendations for clinical guidelines These approaches focuson the effectiveness of treatment and rank only quantitativeevidence according to the rigour of the research designedto lim it bias An approach to categorising the validity ofqualitative evidence which draws on the literature andinput from a panel of experts has been developed forQARI This approach is based on three levels of qualitativeevidence

Unequivoca l The evidence is beyond reasonable doubtand includes findings that are factual directly reportedobserved and not open to challenge

Credible The evidence while interpretative is plausible inlight of the data and theoretical framework Conclusions canbe logically inferred from the data but because the findingsare essentially interpretative these conclusions are open tochallenge

Unsupported Findings are not supported by the data andnone of the other level descriptors apply

These three levels of evidence are incorporated into theQARI software

Levels of applicability

There is little point in accumulating evidence to answer aquestion if it cannot then be used to benefit patients Evi-dence-based practice involves integration of the best avail-able evidence w ith clinical expertise When it comes todeciding whether or not to incorporate a particular activityor intervention into practice some or all of the follow ingconsiderations w ill be relevantbull Is it availablebull Is it affordablebull Is it applicable in the settingbull Would the patientclient be a w illing participant in the

implementation of the interventionbull Were the patients in the studystudies that provided the

evidence sufficiently sim ilar to your own to justify theimplementation of this particular intervention

bull What w ill be the potential benefits for the patientbull What is the potential harm to the patientbull Does this intervention allow for the individual patientrsquos

values and preferencesIn addition to the requirement to define levels of evidence

for practice there is also a need to establish levels of appli-cability The QARI project group developed the lsquoFeasibilityAppropriateness Meaningfulness and Effectivenessrsquo (FAME)scale26 a hierarchy of applicability of evidence that is incor-porated into the QARI software

The QARI levels of applicability of evidence are shown inTable 1

The QARI software

The QARI attempts to establish the results of non-quantita-tive research as appropriate evidence for health-care practi-tioners It was designed to create a system that would enablehealth scientists and health practitioners to review evidencefrom an inclusive position The purpose of the QARI devel-opmental process was to determ ine how evidence gener-ated through qualitative research could be systematicallyreviewed and identify how evidence of appropriatenessmeaningfulness and feasibility could augment evidence ofeffectiveness in evidence-based health care

The QARI software is designed to manage appraise anal-yse and synthesise textual data as part of a systematic reviewof evidence QARI has been designed as a web-based data-base and incorporates a critical appraisal scale data extrac-tion forms a data synthesis function and a reporting function

Balancing the evidence 53

The software has been internationally peer reviewed It hasbeen tested by a group of international systematic reviewersand is currently being piloted by evidence review groups inCanada Australia England and Scotland

The QARI software can be accessed by those who hold alicence by entering the JBI website (http wwwjoan-nabriggseduau) and then clicking on the SUMARI logo

An overview of how QARI is used is presented here todemonstrate the process of appraisal and meta-synthesis

When QARI is opened the main menu is always acrossthe top of the screen (Fig 1)

Reviews

Reviews are the projects to which Studies Extractions andFindings relate The reviews screen lists the reviews availableto the logged on reviewer (Fig 1)

Before a Reviewer can work on a project they must beassigned to the Review either when the Review is first cre-

Table 1 Qualitative Assessment and Review Instrument (QARI) levels of applicability of evidence

Feasibility Appropriateness Meaningfulness Effectiveness

Immediately practicable Acceptable and justifiable w ithin ethical guidelines

Provides a rationale for practice development

Effectiveness established to a degreethat merits application

Practicable w ith lim ited local training or modest additional resources

Acceptable after m inor revision Provides a rationale for local regional or national reform

Effectiveness established to a degree that suggests application

Practicable w ith extensive additional training or resources

Acceptable after major revision Provides a rationale for practice-relevant research

Effectiveness established to a degree that warrants consideration of applying the findings

Practicable w ith significant national reforms

Acceptable after development of new ethical guidelines

Provides a rationale for advocatingchange

Effectiveness established to a lim ited degree

Impracticable Ethically unacceptable Evidence unlikely to make sense to practitioners

Effectiveness not established

Figure 1 Reviews screen listing the reviews available to the logged on reviewer

Balancing the evidence 49

whether the literature is outside the clinical rem it of anRCTrsquo12

The need to more fully integrate the results of qualitativeresearch into the systematic process is well stated by Popayand Williams who suggest that lsquo there are many propo-nents of evidence-based decision making w ithin healthcarewho cannot andor w ill not accept that qualitative researchhas an important part to play rsquo13

Popay and Williams assert that the results of qualitativeresearch do more than simply enhance those of quantitativestudies and suggest that qualitative research is capable ofgenerating evidence thatbull explores taken-for-granted practicesbull increases understanding of consumer and clinical

behaviourbull develops interventionsbull illum inates patientrsquos perceptions on qualityappropriate-

nessbull gives guidance to understanding organisational culture

and change management andbull evaluates complex policy initiatives13

Green and Britten stated that

Qualitative research may seem unscientific and anecdotal tomany medical scientists However as the critics of evidence basedmedicine are quick to point out medicine is more than theapplication of scientific rules14

Green and Britten go on to argue that qualitative researchfindings provide rigorous accounts of treatment regimens ineveryday contexts They also contend that there is anincreasing need w ithin the evidence-based practice arena toraise awareness of the fact that different research questionsrequire different kinds of research They are unequivocal intheir assertion that lsquogoodrsquo evidence goes further than theresults of meta-analysis of RCTs

The Cochrane Qualitative Methods Group established in2002 is currently exploring the scope for incorporatingqualitative research into Cochrane reviews There are still nointernationally reviewed guidelines for assessing the qualityof specific qualitative methods and no established proce-dures for ranking or rating qualitative research findingsreported in the literature There have been however a num-ber of attempts to synthesise (as a form of meta-analysis)the results of sim ilar qualitative studies and these are welldescribed by Sandelowski et a l these authors have alsodeveloped an in-depth theoretical approach to the system-atic metasynthesis of qualitative findings that maintain theintegrity of individual studies15 Draw ing on the work ofSandelowski et a l Popay and Williams and Lemmer et a lan approach to qualitative meta-analysis quality assessment

and the development of a quality rating scale for qualitativeresearch results could be used to add appropriateness andfeasibility dimensions to the current effectiveness-orientatedsystematic review process121315

There are signs that the evidence-based practice move-ment is beginning to develop a more comprehensive viewof evidence There are research initiatives attempting toconstruct approaches to assessing and synthesising theresults of interpretative and critical research so that theseforms of evidence can become an integral part of systematicreviews and thus inform practice Several authors havereported on the systematic review of evidence elicitedthrough interpretative and critical approaches to inquiry16ndash

18 An approach to the meta-synthesis of qualitative findingsand the problems associated w ith synthesising the findingsof studies that are essentially context-bound is described byJensen and Allen and Popay and Roen overview a w ide rangeof current initiatives focusing on methods to appraise andsynthesise qualitative research w ithin the framework of thesystematic review process1920

Developing sound and acceptable methodologies toappraise the findings of qualitative research and to synthe-sise the findings of two or more sim ilar studies is bothcomplex and challenging Interesting developments in thisarea have been reported by several authors1721ndash24 Suchevolving approaches to evidence-based practice represent agrow ing body of work supporting the appropriate use ofrelevant evidence in the systematic review process that w illhelp practitioners to perform well in practice and to use theirprofessional judgement in the use of appropriate evidence

Including qualitative data in systematic reviews

Over a period of two years a project utilising participatoryprocesses at three consensus workshops has explored thereview of qualitative evidence and an electronic packagehas been designed to enable reviewers to systematicallyreview qualitative evidence The purpose of this project wasto determ ine how evidence generated through qualitativeresearch could be systematically reviewed and to identifyhow evidence of appropriateness and feasibility could aug-ment evidence of effectiveness in evidence-based healthcare

Project design

The project consisted of four phasesIn Phase 1 acknow ledged experts in qualitative and

action-orientated approaches were invited to attend a con-

50 A Pearson

sensus workshop w ith a view to working w ith the researcherto develop instruments to evaluate and extract data from qualitative research reports

In Phase 2 acknow ledged experts in qualitative andaction-orientated approaches were invited to attend a con-sensus workshop w ith a view to working w ith the researcherto develop a systematic process of extracting and synthesis-ing data from qualitative research reports

In Phase 3 the researcher worked w ith a software devel-oper to develop an electronic system to review qualitativeevidence

In Phase 4 the software developed was piloted w ith theexpert groups and other health professionals attending sys-tematic review training workshops

Outcomes

At the workshops a group of leading Australian qualitativeresearchers were invited to participate in a consensus work-shop to consider how a systematic process of extracting andsynthesising qualitative data can occur to reflect a rigorousprocess equivalent to the existing processes applied to theresults of RCT and other quantitative research while main-taining sensitivity to the contextual nature of qualitativeresearch

More specifically participants were asked tobull design appropriate data extraction toolsbull design appropriate data synthesis tools andbull draft a position statement

N ine of those invited attended the workshops Workshopattendees were Professor Mary FitzGerald University ofNewcastle Associate Professor Jane Stein-Parbury Universityof Technology Professor Colin Holmes James Cook Univer-sity of Northern Queensland Professor M ichael C linton Cur-tin University Professor Desley Hegney University ofSouthern Queensland Dr Ken Walsh The University ofAdelaide Dr Karen Francis Charles Sturt University MrMatt Lew is La Trobe University and Ms Cathy Ward LaTrobe University

Consensus workshops

Expected outcomes were agreed upon at each workshop Itwas also agreed that the workshop outcomes would needto accommodate the nature of qualitative approaches toresearch rather than replicate existing formats The partici-pants emphasised the complexity of interpretative and crit-ical understandings of phenomena but were also aware ofthe need to ensure that outcomes would be practical andusable Participants were also m indful of the complexity ofthe types of research under consideration and wanted to

balance the utility of the outcomes w ith the complexity ofthe material

lsquoMeaningrsquo was identified as a unifying theme of qualita-tive research A list of methodological frameworks was gen-erated and these were then grouped in relation to theirorientation (Appendix I) Judgements were made about thevalue of each methodological position in relation to specificcriteria of meaningfulness appropriateness feasibility andeffectiveness

The participants recognised that they were working w itha w ide range of methodologies each w ith its own strengthsand weaknesses They agreed to develop a lsquomatrixrsquoapproach which was considered to be more appropriatethan a simple hierarchy of evidence With regard to thecentrality of contextual issues in interpretative and criticalresearch the group suggested that this be taken intoaccount by including statements in the summary about thescope of applicability

The group agreed that an overall matrix and the contextstatement should generate a summary statement describingthe strengths and weaknesses of the available evidence andthe associated levels of confidence

Workshop outcomes statement

Consensus was reached on an lsquooutcomes statementrsquo devel-oped collaboratively by the group (Appendix II)

Draw ing on this consensus summary statement proto-type components for a qualitative assessment and reviewinstrument were developed This was named the QualitativeAssessment and Review Instrument (QARI)

Appraising and synthesising qualitative data

Critical appraisal

The central concern in critically appraising experimental orquantifiable data is to lim it bias and thus establish the valid-ity of a study From a quantitative perspective sources ofbias include selection bias performance bias and attritionbias and validity is assessed by establishing the extent towhich a studyrsquos design and conduct address potential bias

This focus on lim iting bias to establish validity is antithet-ical to the philosophical foundations of qualitativeapproaches to inquiry Emden and Sandelowski suggest thatvalidity in quantitative terms measures those things that itpurports to render generalisable In social inquiry howeverthey argue that validity is perceived as criteria of rigour forqualitative research25

There is much dissent in the literature on the appropriate-ness of establishing criteria to assess the validity of qualita-

Balancing the evidence 51

tive research However Pearson takes the view that atransparent approach to appraising qualitative research iscentral to its ongoing credibility transferability and theoret-ical potential26 Denzin and Lincoln concur w ith this conclu-sion and highlight the need for development of a set ofvalidity criteria sensitive to the nature of the qualitativeresearch and its basis in subjectivity27 Popay et a l areunequivocal in their assertion that the development of stan-dards for assessing evidence from qualitative research is bothpossible and desirable28 They provide the follow ing as aguide to common standardsbull evidence of responsiveness to social context and flexibility

of designbull evidence of theoretical or purposeful samplingbull evidence of adequate descriptionbull evidence of data qualitybull evidence of theoretical and conceptual adequacy andbull potential for assessing typicality28

There is a grow ing amount of literature that exam ines theappraisal of qualitative studies and a large number of for-mats are available Draw ing upon this literature and anextensive process of development and piloting Averis andPearson describe a general set of criteria for appraising thevalidity of interpretative and critical research26 These criteriawere incorporated into the critical appraisal scale of the QARIsoftware which was developed by the project

The critical appraisal scale was piloted and refined in threesystematic review training workshops The checklist inAppendix III provides the essential framework for the criticalappraisal of interpretative and critical studies

Data extraction

Data extraction aims to reduce the findings of many studiesinto a single document and summarisebull methodsbull interventions andbull outcomes

Data extraction involves transferring data from the origi-nal paper using an approach agreed upon and standardisedfor the specific review An agreed format is essential tom inim ise error to provide a historical record of decisionsmade about the data in the review and to become the dataset for analysis and synthesis A data extraction instrumentdraw ing on the literature and input from a panel of expertswas developed extensively piloted refined and incorpo-rated into the QARI software (Appendix IV) Based on thestandard approach promoted by the Cochrane Collabora-tion and adopted by the JBI two reviewers are expected toindependently extract data and then confer

Meta-synthesis

The most complex problem in synthesising textual data isagreeing on and communicating techniques to compare thefindings of each study Meta-synthesis relates to the com-bining of separate elements to form a coherent whole Thisinvolves reasoning from the general to the particular usinga process of logical deduction An approach to the meta-synthesis of qualitative data which draws on the literatureand input from a panel of experts was developed for QARIThis involvesbull translating themes metaphors or conceptsbull transferring actual text or summarised text that illustrates

the theme metaphor or concept andbull re-categorising the data obtained to arrive at a

synthesisIn order to pursue this reviewers need to establish the

follow ing before carrying out data synthesisbull their own rules for setting up categoriesbull how to assign findings to categories andbull how to write narrative summaries for each category

The reviewers need to document these decisions and theirrationale in the systematic review report This process isincorporated into the QARI software

The analysis and synthesis of qualitative studies is com-monly termed meta-synthesis and like meta-analysis it isbased on processed data There are major differencesbetween the approach used to synthesise the findings ofRCT and the approach used for qualitative studies Realityfor the qualitative researcher and reviewer is viewed asmultiple and constructed and so undertaking meta-synthesis means that no two reviewers w ill produce exactlythe same results Although meta-synthesis provides only oneinterpretation it aims to capture the essence of the phenom-enon interest

When engaging in the synthesis of the results of qualita-tive studies differing research methods such as phenome-nology ethnography or grounded theory are not m ixed ina single synthesis of all qualitative studies

The aim of meta-synthesis is to portray an accurate inter-pretation of a phenomenon and to compare and contrastthe constructs of individual studies to reach consensus on anew construction of that phenomenon

Meta-synthesis involvesbull identifying findingsbull grouping findings into categories andbull grouping categories into synthesised findings

Findings Findings are conclusions reached by thereviewer(s) after exam ining the results of data analysis (egthemes metaphors) consisting of a statement that relates

52 A Pearson

two or more phenomena variables or circumstances thatmay inform practice

Categories Categories are groups of findings that reflectsim ilar relationships between sim ilar phenomena variablesor circumstances that may inform practice

Synthesised findings Synthesis refers to the combining ofseparate elements to form a coherent whole using logicaldeduction and reasoning from the general to the particularIn QARI a synthesised finding is defined as an overarchingdescription of a group of categorised findings that allow forthe generation of recommendations for practice

Categorising findingsIn order to pursue this reviewers need to establish their ownrules for the follow ing before carrying out data synthesisbull setting up categoriesbull assigning findings to categories andbull writing narrative summaries for each category

Reviewers need to document these decisions and theirrationale in the systematic review report This process isincorporated into the QARI software

The primary reviewer then categorises findings

Synthesised findingsWhen categorisation is complete the reviewers then studythe categories and synthesise these to form a set of synthe-sised findings

Levels of evidence

Current approaches to evaluating evidence utilise a hierar-chy of evidence designed to assess the validity of recom-mendations for clinical guidelines These approaches focuson the effectiveness of treatment and rank only quantitativeevidence according to the rigour of the research designedto lim it bias An approach to categorising the validity ofqualitative evidence which draws on the literature andinput from a panel of experts has been developed forQARI This approach is based on three levels of qualitativeevidence

Unequivoca l The evidence is beyond reasonable doubtand includes findings that are factual directly reportedobserved and not open to challenge

Credible The evidence while interpretative is plausible inlight of the data and theoretical framework Conclusions canbe logically inferred from the data but because the findingsare essentially interpretative these conclusions are open tochallenge

Unsupported Findings are not supported by the data andnone of the other level descriptors apply

These three levels of evidence are incorporated into theQARI software

Levels of applicability

There is little point in accumulating evidence to answer aquestion if it cannot then be used to benefit patients Evi-dence-based practice involves integration of the best avail-able evidence w ith clinical expertise When it comes todeciding whether or not to incorporate a particular activityor intervention into practice some or all of the follow ingconsiderations w ill be relevantbull Is it availablebull Is it affordablebull Is it applicable in the settingbull Would the patientclient be a w illing participant in the

implementation of the interventionbull Were the patients in the studystudies that provided the

evidence sufficiently sim ilar to your own to justify theimplementation of this particular intervention

bull What w ill be the potential benefits for the patientbull What is the potential harm to the patientbull Does this intervention allow for the individual patientrsquos

values and preferencesIn addition to the requirement to define levels of evidence

for practice there is also a need to establish levels of appli-cability The QARI project group developed the lsquoFeasibilityAppropriateness Meaningfulness and Effectivenessrsquo (FAME)scale26 a hierarchy of applicability of evidence that is incor-porated into the QARI software

The QARI levels of applicability of evidence are shown inTable 1

The QARI software

The QARI attempts to establish the results of non-quantita-tive research as appropriate evidence for health-care practi-tioners It was designed to create a system that would enablehealth scientists and health practitioners to review evidencefrom an inclusive position The purpose of the QARI devel-opmental process was to determ ine how evidence gener-ated through qualitative research could be systematicallyreviewed and identify how evidence of appropriatenessmeaningfulness and feasibility could augment evidence ofeffectiveness in evidence-based health care

The QARI software is designed to manage appraise anal-yse and synthesise textual data as part of a systematic reviewof evidence QARI has been designed as a web-based data-base and incorporates a critical appraisal scale data extrac-tion forms a data synthesis function and a reporting function

Balancing the evidence 53

The software has been internationally peer reviewed It hasbeen tested by a group of international systematic reviewersand is currently being piloted by evidence review groups inCanada Australia England and Scotland

The QARI software can be accessed by those who hold alicence by entering the JBI website (http wwwjoan-nabriggseduau) and then clicking on the SUMARI logo

An overview of how QARI is used is presented here todemonstrate the process of appraisal and meta-synthesis

When QARI is opened the main menu is always acrossthe top of the screen (Fig 1)

Reviews

Reviews are the projects to which Studies Extractions andFindings relate The reviews screen lists the reviews availableto the logged on reviewer (Fig 1)

Before a Reviewer can work on a project they must beassigned to the Review either when the Review is first cre-

Table 1 Qualitative Assessment and Review Instrument (QARI) levels of applicability of evidence

Feasibility Appropriateness Meaningfulness Effectiveness

Immediately practicable Acceptable and justifiable w ithin ethical guidelines

Provides a rationale for practice development

Effectiveness established to a degreethat merits application

Practicable w ith lim ited local training or modest additional resources

Acceptable after m inor revision Provides a rationale for local regional or national reform

Effectiveness established to a degree that suggests application

Practicable w ith extensive additional training or resources

Acceptable after major revision Provides a rationale for practice-relevant research

Effectiveness established to a degree that warrants consideration of applying the findings

Practicable w ith significant national reforms

Acceptable after development of new ethical guidelines

Provides a rationale for advocatingchange

Effectiveness established to a lim ited degree

Impracticable Ethically unacceptable Evidence unlikely to make sense to practitioners

Effectiveness not established

Figure 1 Reviews screen listing the reviews available to the logged on reviewer

50 A Pearson

sensus workshop w ith a view to working w ith the researcherto develop instruments to evaluate and extract data from qualitative research reports

In Phase 2 acknow ledged experts in qualitative andaction-orientated approaches were invited to attend a con-sensus workshop w ith a view to working w ith the researcherto develop a systematic process of extracting and synthesis-ing data from qualitative research reports

In Phase 3 the researcher worked w ith a software devel-oper to develop an electronic system to review qualitativeevidence

In Phase 4 the software developed was piloted w ith theexpert groups and other health professionals attending sys-tematic review training workshops

Outcomes

At the workshops a group of leading Australian qualitativeresearchers were invited to participate in a consensus work-shop to consider how a systematic process of extracting andsynthesising qualitative data can occur to reflect a rigorousprocess equivalent to the existing processes applied to theresults of RCT and other quantitative research while main-taining sensitivity to the contextual nature of qualitativeresearch

More specifically participants were asked tobull design appropriate data extraction toolsbull design appropriate data synthesis tools andbull draft a position statement

N ine of those invited attended the workshops Workshopattendees were Professor Mary FitzGerald University ofNewcastle Associate Professor Jane Stein-Parbury Universityof Technology Professor Colin Holmes James Cook Univer-sity of Northern Queensland Professor M ichael C linton Cur-tin University Professor Desley Hegney University ofSouthern Queensland Dr Ken Walsh The University ofAdelaide Dr Karen Francis Charles Sturt University MrMatt Lew is La Trobe University and Ms Cathy Ward LaTrobe University

Consensus workshops

Expected outcomes were agreed upon at each workshop Itwas also agreed that the workshop outcomes would needto accommodate the nature of qualitative approaches toresearch rather than replicate existing formats The partici-pants emphasised the complexity of interpretative and crit-ical understandings of phenomena but were also aware ofthe need to ensure that outcomes would be practical andusable Participants were also m indful of the complexity ofthe types of research under consideration and wanted to

balance the utility of the outcomes w ith the complexity ofthe material

lsquoMeaningrsquo was identified as a unifying theme of qualita-tive research A list of methodological frameworks was gen-erated and these were then grouped in relation to theirorientation (Appendix I) Judgements were made about thevalue of each methodological position in relation to specificcriteria of meaningfulness appropriateness feasibility andeffectiveness

The participants recognised that they were working w itha w ide range of methodologies each w ith its own strengthsand weaknesses They agreed to develop a lsquomatrixrsquoapproach which was considered to be more appropriatethan a simple hierarchy of evidence With regard to thecentrality of contextual issues in interpretative and criticalresearch the group suggested that this be taken intoaccount by including statements in the summary about thescope of applicability

The group agreed that an overall matrix and the contextstatement should generate a summary statement describingthe strengths and weaknesses of the available evidence andthe associated levels of confidence

Workshop outcomes statement

Consensus was reached on an lsquooutcomes statementrsquo devel-oped collaboratively by the group (Appendix II)

Draw ing on this consensus summary statement proto-type components for a qualitative assessment and reviewinstrument were developed This was named the QualitativeAssessment and Review Instrument (QARI)

Appraising and synthesising qualitative data

Critical appraisal

The central concern in critically appraising experimental orquantifiable data is to lim it bias and thus establish the valid-ity of a study From a quantitative perspective sources ofbias include selection bias performance bias and attritionbias and validity is assessed by establishing the extent towhich a studyrsquos design and conduct address potential bias

This focus on lim iting bias to establish validity is antithet-ical to the philosophical foundations of qualitativeapproaches to inquiry Emden and Sandelowski suggest thatvalidity in quantitative terms measures those things that itpurports to render generalisable In social inquiry howeverthey argue that validity is perceived as criteria of rigour forqualitative research25

There is much dissent in the literature on the appropriate-ness of establishing criteria to assess the validity of qualita-

Balancing the evidence 51

tive research However Pearson takes the view that atransparent approach to appraising qualitative research iscentral to its ongoing credibility transferability and theoret-ical potential26 Denzin and Lincoln concur w ith this conclu-sion and highlight the need for development of a set ofvalidity criteria sensitive to the nature of the qualitativeresearch and its basis in subjectivity27 Popay et a l areunequivocal in their assertion that the development of stan-dards for assessing evidence from qualitative research is bothpossible and desirable28 They provide the follow ing as aguide to common standardsbull evidence of responsiveness to social context and flexibility

of designbull evidence of theoretical or purposeful samplingbull evidence of adequate descriptionbull evidence of data qualitybull evidence of theoretical and conceptual adequacy andbull potential for assessing typicality28

There is a grow ing amount of literature that exam ines theappraisal of qualitative studies and a large number of for-mats are available Draw ing upon this literature and anextensive process of development and piloting Averis andPearson describe a general set of criteria for appraising thevalidity of interpretative and critical research26 These criteriawere incorporated into the critical appraisal scale of the QARIsoftware which was developed by the project

The critical appraisal scale was piloted and refined in threesystematic review training workshops The checklist inAppendix III provides the essential framework for the criticalappraisal of interpretative and critical studies

Data extraction

Data extraction aims to reduce the findings of many studiesinto a single document and summarisebull methodsbull interventions andbull outcomes

Data extraction involves transferring data from the origi-nal paper using an approach agreed upon and standardisedfor the specific review An agreed format is essential tom inim ise error to provide a historical record of decisionsmade about the data in the review and to become the dataset for analysis and synthesis A data extraction instrumentdraw ing on the literature and input from a panel of expertswas developed extensively piloted refined and incorpo-rated into the QARI software (Appendix IV) Based on thestandard approach promoted by the Cochrane Collabora-tion and adopted by the JBI two reviewers are expected toindependently extract data and then confer

Meta-synthesis

The most complex problem in synthesising textual data isagreeing on and communicating techniques to compare thefindings of each study Meta-synthesis relates to the com-bining of separate elements to form a coherent whole Thisinvolves reasoning from the general to the particular usinga process of logical deduction An approach to the meta-synthesis of qualitative data which draws on the literatureand input from a panel of experts was developed for QARIThis involvesbull translating themes metaphors or conceptsbull transferring actual text or summarised text that illustrates

the theme metaphor or concept andbull re-categorising the data obtained to arrive at a

synthesisIn order to pursue this reviewers need to establish the

follow ing before carrying out data synthesisbull their own rules for setting up categoriesbull how to assign findings to categories andbull how to write narrative summaries for each category

The reviewers need to document these decisions and theirrationale in the systematic review report This process isincorporated into the QARI software

The analysis and synthesis of qualitative studies is com-monly termed meta-synthesis and like meta-analysis it isbased on processed data There are major differencesbetween the approach used to synthesise the findings ofRCT and the approach used for qualitative studies Realityfor the qualitative researcher and reviewer is viewed asmultiple and constructed and so undertaking meta-synthesis means that no two reviewers w ill produce exactlythe same results Although meta-synthesis provides only oneinterpretation it aims to capture the essence of the phenom-enon interest

When engaging in the synthesis of the results of qualita-tive studies differing research methods such as phenome-nology ethnography or grounded theory are not m ixed ina single synthesis of all qualitative studies

The aim of meta-synthesis is to portray an accurate inter-pretation of a phenomenon and to compare and contrastthe constructs of individual studies to reach consensus on anew construction of that phenomenon

Meta-synthesis involvesbull identifying findingsbull grouping findings into categories andbull grouping categories into synthesised findings

Findings Findings are conclusions reached by thereviewer(s) after exam ining the results of data analysis (egthemes metaphors) consisting of a statement that relates

52 A Pearson

two or more phenomena variables or circumstances thatmay inform practice

Categories Categories are groups of findings that reflectsim ilar relationships between sim ilar phenomena variablesor circumstances that may inform practice

Synthesised findings Synthesis refers to the combining ofseparate elements to form a coherent whole using logicaldeduction and reasoning from the general to the particularIn QARI a synthesised finding is defined as an overarchingdescription of a group of categorised findings that allow forthe generation of recommendations for practice

Categorising findingsIn order to pursue this reviewers need to establish their ownrules for the follow ing before carrying out data synthesisbull setting up categoriesbull assigning findings to categories andbull writing narrative summaries for each category

Reviewers need to document these decisions and theirrationale in the systematic review report This process isincorporated into the QARI software

The primary reviewer then categorises findings

Synthesised findingsWhen categorisation is complete the reviewers then studythe categories and synthesise these to form a set of synthe-sised findings

Levels of evidence

Current approaches to evaluating evidence utilise a hierar-chy of evidence designed to assess the validity of recom-mendations for clinical guidelines These approaches focuson the effectiveness of treatment and rank only quantitativeevidence according to the rigour of the research designedto lim it bias An approach to categorising the validity ofqualitative evidence which draws on the literature andinput from a panel of experts has been developed forQARI This approach is based on three levels of qualitativeevidence

Unequivoca l The evidence is beyond reasonable doubtand includes findings that are factual directly reportedobserved and not open to challenge

Credible The evidence while interpretative is plausible inlight of the data and theoretical framework Conclusions canbe logically inferred from the data but because the findingsare essentially interpretative these conclusions are open tochallenge

Unsupported Findings are not supported by the data andnone of the other level descriptors apply

These three levels of evidence are incorporated into theQARI software

Levels of applicability

There is little point in accumulating evidence to answer aquestion if it cannot then be used to benefit patients Evi-dence-based practice involves integration of the best avail-able evidence w ith clinical expertise When it comes todeciding whether or not to incorporate a particular activityor intervention into practice some or all of the follow ingconsiderations w ill be relevantbull Is it availablebull Is it affordablebull Is it applicable in the settingbull Would the patientclient be a w illing participant in the

implementation of the interventionbull Were the patients in the studystudies that provided the

evidence sufficiently sim ilar to your own to justify theimplementation of this particular intervention

bull What w ill be the potential benefits for the patientbull What is the potential harm to the patientbull Does this intervention allow for the individual patientrsquos

values and preferencesIn addition to the requirement to define levels of evidence

for practice there is also a need to establish levels of appli-cability The QARI project group developed the lsquoFeasibilityAppropriateness Meaningfulness and Effectivenessrsquo (FAME)scale26 a hierarchy of applicability of evidence that is incor-porated into the QARI software

The QARI levels of applicability of evidence are shown inTable 1

The QARI software

The QARI attempts to establish the results of non-quantita-tive research as appropriate evidence for health-care practi-tioners It was designed to create a system that would enablehealth scientists and health practitioners to review evidencefrom an inclusive position The purpose of the QARI devel-opmental process was to determ ine how evidence gener-ated through qualitative research could be systematicallyreviewed and identify how evidence of appropriatenessmeaningfulness and feasibility could augment evidence ofeffectiveness in evidence-based health care

The QARI software is designed to manage appraise anal-yse and synthesise textual data as part of a systematic reviewof evidence QARI has been designed as a web-based data-base and incorporates a critical appraisal scale data extrac-tion forms a data synthesis function and a reporting function

Balancing the evidence 53

The software has been internationally peer reviewed It hasbeen tested by a group of international systematic reviewersand is currently being piloted by evidence review groups inCanada Australia England and Scotland

The QARI software can be accessed by those who hold alicence by entering the JBI website (http wwwjoan-nabriggseduau) and then clicking on the SUMARI logo

An overview of how QARI is used is presented here todemonstrate the process of appraisal and meta-synthesis

When QARI is opened the main menu is always acrossthe top of the screen (Fig 1)

Reviews

Reviews are the projects to which Studies Extractions andFindings relate The reviews screen lists the reviews availableto the logged on reviewer (Fig 1)

Before a Reviewer can work on a project they must beassigned to the Review either when the Review is first cre-

Table 1 Qualitative Assessment and Review Instrument (QARI) levels of applicability of evidence

Feasibility Appropriateness Meaningfulness Effectiveness

Immediately practicable Acceptable and justifiable w ithin ethical guidelines

Provides a rationale for practice development

Effectiveness established to a degreethat merits application

Practicable w ith lim ited local training or modest additional resources

Acceptable after m inor revision Provides a rationale for local regional or national reform

Effectiveness established to a degree that suggests application

Practicable w ith extensive additional training or resources

Acceptable after major revision Provides a rationale for practice-relevant research

Effectiveness established to a degree that warrants consideration of applying the findings

Practicable w ith significant national reforms

Acceptable after development of new ethical guidelines

Provides a rationale for advocatingchange

Effectiveness established to a lim ited degree

Impracticable Ethically unacceptable Evidence unlikely to make sense to practitioners

Effectiveness not established

Figure 1 Reviews screen listing the reviews available to the logged on reviewer

Balancing the evidence 51

tive research However Pearson takes the view that atransparent approach to appraising qualitative research iscentral to its ongoing credibility transferability and theoret-ical potential26 Denzin and Lincoln concur w ith this conclu-sion and highlight the need for development of a set ofvalidity criteria sensitive to the nature of the qualitativeresearch and its basis in subjectivity27 Popay et a l areunequivocal in their assertion that the development of stan-dards for assessing evidence from qualitative research is bothpossible and desirable28 They provide the follow ing as aguide to common standardsbull evidence of responsiveness to social context and flexibility

of designbull evidence of theoretical or purposeful samplingbull evidence of adequate descriptionbull evidence of data qualitybull evidence of theoretical and conceptual adequacy andbull potential for assessing typicality28

There is a grow ing amount of literature that exam ines theappraisal of qualitative studies and a large number of for-mats are available Draw ing upon this literature and anextensive process of development and piloting Averis andPearson describe a general set of criteria for appraising thevalidity of interpretative and critical research26 These criteriawere incorporated into the critical appraisal scale of the QARIsoftware which was developed by the project

The critical appraisal scale was piloted and refined in threesystematic review training workshops The checklist inAppendix III provides the essential framework for the criticalappraisal of interpretative and critical studies

Data extraction

Data extraction aims to reduce the findings of many studiesinto a single document and summarisebull methodsbull interventions andbull outcomes

Data extraction involves transferring data from the origi-nal paper using an approach agreed upon and standardisedfor the specific review An agreed format is essential tom inim ise error to provide a historical record of decisionsmade about the data in the review and to become the dataset for analysis and synthesis A data extraction instrumentdraw ing on the literature and input from a panel of expertswas developed extensively piloted refined and incorpo-rated into the QARI software (Appendix IV) Based on thestandard approach promoted by the Cochrane Collabora-tion and adopted by the JBI two reviewers are expected toindependently extract data and then confer

Meta-synthesis

The most complex problem in synthesising textual data isagreeing on and communicating techniques to compare thefindings of each study Meta-synthesis relates to the com-bining of separate elements to form a coherent whole Thisinvolves reasoning from the general to the particular usinga process of logical deduction An approach to the meta-synthesis of qualitative data which draws on the literatureand input from a panel of experts was developed for QARIThis involvesbull translating themes metaphors or conceptsbull transferring actual text or summarised text that illustrates

the theme metaphor or concept andbull re-categorising the data obtained to arrive at a

synthesisIn order to pursue this reviewers need to establish the

follow ing before carrying out data synthesisbull their own rules for setting up categoriesbull how to assign findings to categories andbull how to write narrative summaries for each category

The reviewers need to document these decisions and theirrationale in the systematic review report This process isincorporated into the QARI software

The analysis and synthesis of qualitative studies is com-monly termed meta-synthesis and like meta-analysis it isbased on processed data There are major differencesbetween the approach used to synthesise the findings ofRCT and the approach used for qualitative studies Realityfor the qualitative researcher and reviewer is viewed asmultiple and constructed and so undertaking meta-synthesis means that no two reviewers w ill produce exactlythe same results Although meta-synthesis provides only oneinterpretation it aims to capture the essence of the phenom-enon interest

When engaging in the synthesis of the results of qualita-tive studies differing research methods such as phenome-nology ethnography or grounded theory are not m ixed ina single synthesis of all qualitative studies

The aim of meta-synthesis is to portray an accurate inter-pretation of a phenomenon and to compare and contrastthe constructs of individual studies to reach consensus on anew construction of that phenomenon

Meta-synthesis involvesbull identifying findingsbull grouping findings into categories andbull grouping categories into synthesised findings

Findings Findings are conclusions reached by thereviewer(s) after exam ining the results of data analysis (egthemes metaphors) consisting of a statement that relates

52 A Pearson

two or more phenomena variables or circumstances thatmay inform practice

Categories Categories are groups of findings that reflectsim ilar relationships between sim ilar phenomena variablesor circumstances that may inform practice

Synthesised findings Synthesis refers to the combining ofseparate elements to form a coherent whole using logicaldeduction and reasoning from the general to the particularIn QARI a synthesised finding is defined as an overarchingdescription of a group of categorised findings that allow forthe generation of recommendations for practice

Categorising findingsIn order to pursue this reviewers need to establish their ownrules for the follow ing before carrying out data synthesisbull setting up categoriesbull assigning findings to categories andbull writing narrative summaries for each category

Reviewers need to document these decisions and theirrationale in the systematic review report This process isincorporated into the QARI software

The primary reviewer then categorises findings

Synthesised findingsWhen categorisation is complete the reviewers then studythe categories and synthesise these to form a set of synthe-sised findings

Levels of evidence

Current approaches to evaluating evidence utilise a hierar-chy of evidence designed to assess the validity of recom-mendations for clinical guidelines These approaches focuson the effectiveness of treatment and rank only quantitativeevidence according to the rigour of the research designedto lim it bias An approach to categorising the validity ofqualitative evidence which draws on the literature andinput from a panel of experts has been developed forQARI This approach is based on three levels of qualitativeevidence

Unequivoca l The evidence is beyond reasonable doubtand includes findings that are factual directly reportedobserved and not open to challenge

Credible The evidence while interpretative is plausible inlight of the data and theoretical framework Conclusions canbe logically inferred from the data but because the findingsare essentially interpretative these conclusions are open tochallenge

Unsupported Findings are not supported by the data andnone of the other level descriptors apply

These three levels of evidence are incorporated into theQARI software

Levels of applicability

There is little point in accumulating evidence to answer aquestion if it cannot then be used to benefit patients Evi-dence-based practice involves integration of the best avail-able evidence w ith clinical expertise When it comes todeciding whether or not to incorporate a particular activityor intervention into practice some or all of the follow ingconsiderations w ill be relevantbull Is it availablebull Is it affordablebull Is it applicable in the settingbull Would the patientclient be a w illing participant in the

implementation of the interventionbull Were the patients in the studystudies that provided the

evidence sufficiently sim ilar to your own to justify theimplementation of this particular intervention

bull What w ill be the potential benefits for the patientbull What is the potential harm to the patientbull Does this intervention allow for the individual patientrsquos

values and preferencesIn addition to the requirement to define levels of evidence

for practice there is also a need to establish levels of appli-cability The QARI project group developed the lsquoFeasibilityAppropriateness Meaningfulness and Effectivenessrsquo (FAME)scale26 a hierarchy of applicability of evidence that is incor-porated into the QARI software

The QARI levels of applicability of evidence are shown inTable 1

The QARI software

The QARI attempts to establish the results of non-quantita-tive research as appropriate evidence for health-care practi-tioners It was designed to create a system that would enablehealth scientists and health practitioners to review evidencefrom an inclusive position The purpose of the QARI devel-opmental process was to determ ine how evidence gener-ated through qualitative research could be systematicallyreviewed and identify how evidence of appropriatenessmeaningfulness and feasibility could augment evidence ofeffectiveness in evidence-based health care

The QARI software is designed to manage appraise anal-yse and synthesise textual data as part of a systematic reviewof evidence QARI has been designed as a web-based data-base and incorporates a critical appraisal scale data extrac-tion forms a data synthesis function and a reporting function

Balancing the evidence 53

The software has been internationally peer reviewed It hasbeen tested by a group of international systematic reviewersand is currently being piloted by evidence review groups inCanada Australia England and Scotland

The QARI software can be accessed by those who hold alicence by entering the JBI website (http wwwjoan-nabriggseduau) and then clicking on the SUMARI logo

An overview of how QARI is used is presented here todemonstrate the process of appraisal and meta-synthesis

When QARI is opened the main menu is always acrossthe top of the screen (Fig 1)

Reviews

Reviews are the projects to which Studies Extractions andFindings relate The reviews screen lists the reviews availableto the logged on reviewer (Fig 1)

Before a Reviewer can work on a project they must beassigned to the Review either when the Review is first cre-

Table 1 Qualitative Assessment and Review Instrument (QARI) levels of applicability of evidence

Feasibility Appropriateness Meaningfulness Effectiveness

Immediately practicable Acceptable and justifiable w ithin ethical guidelines

Provides a rationale for practice development

Effectiveness established to a degreethat merits application

Practicable w ith lim ited local training or modest additional resources

Acceptable after m inor revision Provides a rationale for local regional or national reform

Effectiveness established to a degree that suggests application

Practicable w ith extensive additional training or resources

Acceptable after major revision Provides a rationale for practice-relevant research

Effectiveness established to a degree that warrants consideration of applying the findings

Practicable w ith significant national reforms

Acceptable after development of new ethical guidelines

Provides a rationale for advocatingchange

Effectiveness established to a lim ited degree

Impracticable Ethically unacceptable Evidence unlikely to make sense to practitioners

Effectiveness not established

Figure 1 Reviews screen listing the reviews available to the logged on reviewer

52 A Pearson

two or more phenomena variables or circumstances thatmay inform practice

Categories Categories are groups of findings that reflectsim ilar relationships between sim ilar phenomena variablesor circumstances that may inform practice

Synthesised findings Synthesis refers to the combining ofseparate elements to form a coherent whole using logicaldeduction and reasoning from the general to the particularIn QARI a synthesised finding is defined as an overarchingdescription of a group of categorised findings that allow forthe generation of recommendations for practice

Categorising findingsIn order to pursue this reviewers need to establish their ownrules for the follow ing before carrying out data synthesisbull setting up categoriesbull assigning findings to categories andbull writing narrative summaries for each category

Reviewers need to document these decisions and theirrationale in the systematic review report This process isincorporated into the QARI software

The primary reviewer then categorises findings

Synthesised findingsWhen categorisation is complete the reviewers then studythe categories and synthesise these to form a set of synthe-sised findings

Levels of evidence

Current approaches to evaluating evidence utilise a hierar-chy of evidence designed to assess the validity of recom-mendations for clinical guidelines These approaches focuson the effectiveness of treatment and rank only quantitativeevidence according to the rigour of the research designedto lim it bias An approach to categorising the validity ofqualitative evidence which draws on the literature andinput from a panel of experts has been developed forQARI This approach is based on three levels of qualitativeevidence

Unequivoca l The evidence is beyond reasonable doubtand includes findings that are factual directly reportedobserved and not open to challenge

Credible The evidence while interpretative is plausible inlight of the data and theoretical framework Conclusions canbe logically inferred from the data but because the findingsare essentially interpretative these conclusions are open tochallenge

Unsupported Findings are not supported by the data andnone of the other level descriptors apply

These three levels of evidence are incorporated into theQARI software

Levels of applicability

There is little point in accumulating evidence to answer aquestion if it cannot then be used to benefit patients Evi-dence-based practice involves integration of the best avail-able evidence w ith clinical expertise When it comes todeciding whether or not to incorporate a particular activityor intervention into practice some or all of the follow ingconsiderations w ill be relevantbull Is it availablebull Is it affordablebull Is it applicable in the settingbull Would the patientclient be a w illing participant in the

implementation of the interventionbull Were the patients in the studystudies that provided the

evidence sufficiently sim ilar to your own to justify theimplementation of this particular intervention

bull What w ill be the potential benefits for the patientbull What is the potential harm to the patientbull Does this intervention allow for the individual patientrsquos

values and preferencesIn addition to the requirement to define levels of evidence

for practice there is also a need to establish levels of appli-cability The QARI project group developed the lsquoFeasibilityAppropriateness Meaningfulness and Effectivenessrsquo (FAME)scale26 a hierarchy of applicability of evidence that is incor-porated into the QARI software

The QARI levels of applicability of evidence are shown inTable 1

The QARI software

The QARI attempts to establish the results of non-quantita-tive research as appropriate evidence for health-care practi-tioners It was designed to create a system that would enablehealth scientists and health practitioners to review evidencefrom an inclusive position The purpose of the QARI devel-opmental process was to determ ine how evidence gener-ated through qualitative research could be systematicallyreviewed and identify how evidence of appropriatenessmeaningfulness and feasibility could augment evidence ofeffectiveness in evidence-based health care

The QARI software is designed to manage appraise anal-yse and synthesise textual data as part of a systematic reviewof evidence QARI has been designed as a web-based data-base and incorporates a critical appraisal scale data extrac-tion forms a data synthesis function and a reporting function

Balancing the evidence 53

The software has been internationally peer reviewed It hasbeen tested by a group of international systematic reviewersand is currently being piloted by evidence review groups inCanada Australia England and Scotland

The QARI software can be accessed by those who hold alicence by entering the JBI website (http wwwjoan-nabriggseduau) and then clicking on the SUMARI logo

An overview of how QARI is used is presented here todemonstrate the process of appraisal and meta-synthesis

When QARI is opened the main menu is always acrossthe top of the screen (Fig 1)

Reviews

Reviews are the projects to which Studies Extractions andFindings relate The reviews screen lists the reviews availableto the logged on reviewer (Fig 1)

Before a Reviewer can work on a project they must beassigned to the Review either when the Review is first cre-

Table 1 Qualitative Assessment and Review Instrument (QARI) levels of applicability of evidence

Feasibility Appropriateness Meaningfulness Effectiveness

Immediately practicable Acceptable and justifiable w ithin ethical guidelines

Provides a rationale for practice development

Effectiveness established to a degreethat merits application

Practicable w ith lim ited local training or modest additional resources

Acceptable after m inor revision Provides a rationale for local regional or national reform

Effectiveness established to a degree that suggests application

Practicable w ith extensive additional training or resources

Acceptable after major revision Provides a rationale for practice-relevant research

Effectiveness established to a degree that warrants consideration of applying the findings

Practicable w ith significant national reforms

Acceptable after development of new ethical guidelines

Provides a rationale for advocatingchange

Effectiveness established to a lim ited degree

Impracticable Ethically unacceptable Evidence unlikely to make sense to practitioners

Effectiveness not established

Figure 1 Reviews screen listing the reviews available to the logged on reviewer

Balancing the evidence 53

The software has been internationally peer reviewed It hasbeen tested by a group of international systematic reviewersand is currently being piloted by evidence review groups inCanada Australia England and Scotland

The QARI software can be accessed by those who hold alicence by entering the JBI website (http wwwjoan-nabriggseduau) and then clicking on the SUMARI logo

An overview of how QARI is used is presented here todemonstrate the process of appraisal and meta-synthesis

When QARI is opened the main menu is always acrossthe top of the screen (Fig 1)

Reviews

Reviews are the projects to which Studies Extractions andFindings relate The reviews screen lists the reviews availableto the logged on reviewer (Fig 1)

Before a Reviewer can work on a project they must beassigned to the Review either when the Review is first cre-

Table 1 Qualitative Assessment and Review Instrument (QARI) levels of applicability of evidence

Feasibility Appropriateness Meaningfulness Effectiveness

Immediately practicable Acceptable and justifiable w ithin ethical guidelines

Provides a rationale for practice development

Effectiveness established to a degreethat merits application

Practicable w ith lim ited local training or modest additional resources

Acceptable after m inor revision Provides a rationale for local regional or national reform

Effectiveness established to a degree that suggests application

Practicable w ith extensive additional training or resources

Acceptable after major revision Provides a rationale for practice-relevant research

Effectiveness established to a degree that warrants consideration of applying the findings

Practicable w ith significant national reforms

Acceptable after development of new ethical guidelines

Provides a rationale for advocatingchange

Effectiveness established to a lim ited degree

Impracticable Ethically unacceptable Evidence unlikely to make sense to practitioners

Effectiveness not established

Figure 1 Reviews screen listing the reviews available to the logged on reviewer

54 A Pearson

ated or when it is edited later The Primary Reviewer mustappoint a Secondary Reviewer when a new review is addedA Study is related to a Review Extractions and Findings arewritten for a single Study

The Studies screen allows users to view add edit anddelete Studies (Fig 2) A Study goes through a process ofbeing Assessed and Extracted before Findings are madeagainst it Assessment of a Study determ ines whether aStudy is to be lsquoincludedrsquo or lsquoexcludedrsquo For a study to beused in the Review it must be assessed Both the Primaryand Secondary Reviewer perform the assessment processindependently

Assessment

If a Study is found to be lsquoexcludedrsquo a reason for the exclusionmust be entered During this process the Primary Reviewerwould review each Study to determ ine a final Assessmentstatus (Fig 3) If a Studyrsquos Assessment status is in dispute thePrimary Reviewer would need to resolve any conflicts

In the case of both Assessments on the Study beinglsquoexcludedrsquo a final exclusion reason would need to be cre-ated This reason would default to the original exclusionreason given by the Primary Reviewer but could be modifiedbefore being saved

Once any assessment conflicts are resolved the PrimaryReviewer would complete the final assessment The Studyrsquosstatus would then be updated to lsquoextractionrsquo and theReviewrsquos status to lsquoopenrsquo

Filtering

This function w ill filter the view so that only unassessedstudies are displayed The lsquoAssessmentrsquo column indicateswhat is required to complete the assessment

None No assessment has been preformed by the primaryor secondary reviewer

Primary The primary assessment has not been completedbut secondary assessment has been completed

Secondary The secondary assessment has not been com-pleted but primary assessment has been completed

Fina l Final assessment by the primary reviewer is requiredIncluded The study has been fully assessed and is included

in the reviewExcluded The study has been fully assessed and is

excluded from the review

Initial assessment

To add an initial Assessment the reviewer clicks the AuthorrsquosName on the Studies Screen and then clicks the ltselectgt

Figure 2 View ing the studies

Balancing the evidence 55

button When lsquoAssessmentrsquo is clicked in the menu to the leftof the screen the Assessment Summary screen w ill be dis-played If the Primary Reviewer clicks the ltAdd Primarygtbutton the Assessment Edit screen w ill appear The reviewerthen answers each of the 10 questions by selecting from thedrop-down menus to the right of the questions (see Assess-ment Edit screen at Fig 3) The reviewer then includes orexcludes the Study by using the drop-down menu at thebottom of the questions If the study is excluded an expla-nation is entered in the reason field

The Secondary Reviewer can now assess the study TheSecondary Reviewer w ill follow the same steps as the PrimaryReviewer but click the ltAdd Secondarygt button

Final assessment

Once both initial assessments have been completed thePrimary Reviewer can perform the final assessment

When both assessments are designated lsquoincludedrsquo finalanswers for the 10 questions must be decided as theresponses may not be the same When the final assessmentis to lsquoexcludersquo the study the Exclude Reason fields must becombined

Data extraction is then performed on included studies(Fig 4) The Primary Reviewer can add their Extraction on aStudy An Extraction must be done on a Study before theFindings can be associated to it

A methodology usually covers the theoretical foundationsof the research A list of methodologies appears in AppendixI This list is not exhaustive and where possible more detailsshould be added (eg ethnography may be critical orfem inist)

Method is the way that the data is collected and a list ofmethods again not exhaustive is provided below It isimportant to be as specific as possible For example if inter-

Figure 3 Assessment edit screen

56 A Pearson

view is selected it is important to specify what type ofinterview open-ended sem istructured face-to-face ortelephonebull interviewbull media analysisbull field notesbull discourse analysisbull observationbull surveybull questionnaire

An intervention is a planned change made to the researchsituation by the researcher as part of the research projectFor example an intervention could be serving lunch at10 am in a nursing home or providing an education inter-vention However there w ill not necessarily be an interven-tion in qualitative research and this field may either refer toan activity or phenomenon or be left incomplete

Setting and Context refers to the specific location wherethe research is conducted For example the setting could

be a nursing home a hospital or a dementia specific wardin a subacute hospital Some research w ill have no settingat all (eg discourse analysis)

The Geographical Context refers to the specific locationof the research For example Poland Austria or rural NewZealand

The Cultural Context refers to the cultural features of thestudy setting such as time period (eg 16th century) ethnicgroupings (eg indigenous Australians) age groupings (egolder people living in the community) or socio-econom icgroups (eg professional) These data should be as specificas possible and m ight also identify employment lifestylelevel of functionality and gender factors as well as partici-pation rate Information entered in this field should relate tothe inclusion and exclusion criteria of the research Ambig-uous terms or group names should also be defined in thissection (eg a carer is a personal care attendant)

Data analysis refers to the techniques utilised to analysethe data A list of examples is provided below This list is not

Figure 4 Extraction details screen

Balancing the evidence 57

exhaustive and should be supplemented w ith specific infor-mation where appropriatebull named software programsbull contextual analysisbull comparative analysisbull thematic analysisbull discourse analysisbull content analysis

A Reviewer can add Findings to a Study but only after anExtraction is completed on that Study (Fig 5) If a Studyrsquosstatus changes to lsquoexcludedrsquo then the related Findings w illalso be excluded (but not deleted)

Once all of the information on a Review is collected in theform of Findings and Extractions the Findings can be allo-cated to user-defined categories (Fig 6) To develop cate-gories ltCategorisegt is selected in the main menu A screenw ill appear that lists all of the Findings that have beencreated for the review A category should be established forthose Findings that can be naturally grouped

Once all of the Findings have been lsquocategorisedrsquo thecategories can then be designated to user-defined Synthe-sised Findings (Fig 7) To develop Synthesised FindingsltSynthesisgt is selected in the main menu A screen w illappear that lists all of the findings that have been created

for the review This w ill be grouped by the categories allo-cated to the findings A Synthesised Finding should beestablished for those categories that can be naturallygrouped

Discussion

This project set out to elicit whether or not it would bepossible to integrate qualitative research findings into thesystematic review process and if it was possible to developa system to do so The outcome is a prototype systemdesigned to enable systematic reviews to follow a rigorousprocess in appraising and synthesising qualitative data Thisdevelopment augments a larger project currently inprogress a System for the Unified Management of theAssessment and Review of Information (SUMARI) SUMARIis designed to enable reviewers to incorporate a w ider rangeof findings than those currently accommodated w ithin anemerging review orthodoxy The QARI forms one module ofSUMARI

The Comprehensive Systematic Review (CSR) is predi-cated on the view that the results of well-designed researchstudies ndash grounded in any methodological position ndash pro-vide more rigorous evidence than anecdotes or personal

Figure 5 Add findings to a study

58 A Pearson

opinion but that in the absence of such results opinionderiving from experience and expertise can still legitimatelybe regarded as the lsquobest availablersquo evidence

The CSR is an approach to evidence review that enablesreviewers to consider evidence of Feasibility Appropriate-ness Meaningfulness and Effectiveness in the form of afocused review of one two or more evidence types

SUMARI

SUMARI has been developed to enable systematic reviewersto take an inclusive view of what counts as evidenceSUMARI is a developing software package designed to assisthealth and other researchers and practitioners to conductsystematic reviews of evidence of Feasibility Appropriate-ness Meaningfulness and Effectiveness and to conducteconom ic evaluations of activities and interventions Thepackage consists of five modules (Fig 8)

Module 1 Comprehensive Review Management System (CReMS)This module includes the review protocol search results anda reporting function It is designed to manage a systematicreview and captures the results generated through the fouranalytical modules and formats them into a final report

Module 2 Qua litative Assessment and Review Instrument (QARI)This module is designed to facilitate critical appraisal dataextraction and synthesis of the findings of qualitative studies

Module 3 Meta Ana lysis of Statistics Assessment and Review Instrument (MAStARI)This module is designed to conduct the meta-analysis of theresults of comparable cohort time series and descriptivestudies using a number of statistical approaches

Module 4 Narrative Opinion and Text Assessment and Review Instrument (N OTARI)This module is designed to facilitate critical appraisal dataextraction and synthesis of expert opinion texts and reports

Module 5 Ana lysis of Cost Technology and Utilisation Assessment and Review Instrument (ACTUARI)This module is designed to facilitate critical appraisal dataextraction and synthesis of econom ic data

Using the package

The CReMS module is web based and when downloadedon the userrsquos server can be accessed on the web by those

Figure 6 Finding details screen

Balancing the evidence 59

authorised by the user It can be used as a stand-aloneprogram or in conjunction w ith other SUMARI modules

Each of the other SUMARI modules are also web basedand are designed to interface w ith CReMS and all othermodules Reviewers who w ish to utilise the functions of aspecific module can also use them as stand-alone programs

The total package is designed so that each module inter-acts w ith the others and a reviewer can at the point in the

review when critical appraisal data extraction and data syn-thesismeta-analysis is reached select a pathway to manageRCT data non-RCT quantitative data qualitative data tex-tual data from opinion papers or reports or econom ic dataA single focus review (eg a review of effectiveness) wouldfollow the RCT data pathway only and extract and analyseonly results from RCTs A review w ith more than one focuscan select any number of pathways For example a reviewof effectiveness and feasibility may enter data from actionresearch and evaluative studies into the QARI pathway anddata from reports of learned bodies into the N OTARI path-way as well as RCT results

Conclusion

This paper reports on a system that presents a practicalapproach to developing implementing and evaluating prac-

Figure 7 Allocating synthesised findings to categories

Figure 8 Modules in the System for the Unified Management ofthe Assessment and Review of Information (SUMARI)

60 A Pearson

tice based on lsquoevidencersquo in its broadest sense In addition toexam ining the concept of clinical effectiveness and theCochrane Collaboration approach to the meta-analysis ofquantitative research findings other non-quantitative formsof evidence and how they can be used as appropriatesources of evidence for practice have also been consideredThis has been done w ith a view to providing a practicalprocess using the QARI system which w ill enable the com-prehensive review of qualitative evidence for clinical prac-tice This project continues to evolve w ith the primary aimof re-conceptualising the concept of evidence for practicebecause health-care practices are often far more complexthan they immediately appear

The international interest in evidence-based practice aris-ing largely out of the work of the Cochrane Collaborationis likely to accelerate given the global concerns aboutimproving health care increasing the effectiveness andappropriateness of health interventions and containing thecosts of delivering health services Although the meta-analysis of the results of research into effectiveness is nowhighly refined there is still much work to be done beforethis can be seen to be m irrored in regard to the results ofqualitative research findings A number of research groupsare exam ining and developing ways to advance the incor-poration of qualitative evidence into systematic reviews TheQARI software program is currently being used by groups inCanada Scotland England Thailand and Australia and QARIlicenses are now available A number of completed reviewsare currently being prepared for publication and it is antic-ipated that feedback from the readers of these reviews thegroups currently using QARI and new QARI licenses w ill leadto further work and to the development of a more fullyrefined methodology and software program

Acknowledgements

The author and The Joanna Briggs Institute acknow ledge theparticipation of the follow ing members of the QARI devel-opment group and the contribution they have made to thedevelopment of QARIbull Professor Mary FitzGerald University of Newcastle

Newcastle Australiabull Associate Professor Jane Stein-Parbury University of Tech-

nology Sydney Australiabull Professor Colin Holmes James Cook University of North-

ern Queensland Townsville Australiabull Professor Desley Hegney University of Southern

Queensland Toowoomba and The University ofQueensland Brisbane Australia

bull Professor Ken Walsh Victoria University of WellingtonWellington New Zealand

bull Professor Karen Francis Monash University MelbourneAustralia

bull Mr Matt Lew is La Trobe University Melbourne Australiaand

bull Ms Cathy Ward La Trobe University MelbourneAustralia

References1 Sackett DL Rosenberg WM Gray JA Haynes RB Richardson

WS Evidence-based medicine what it is and what it isnrsquot BrMed J 1996 312 71ndash2

2 FitzGerald M The practical implications of a critique of tradi-tional science Int J Nurs Prac 1995 1 2ndash11

3 Pearsall J Trumble B eds Oxford English Reference D ictionaryOxford Oxford University Press 1995 487

4 Audi R The Cambridge D ictionary of Philosophy CambridgeCambridge University Press 1995

5 D ixon-Woods M Fitzpatrick R Roberts K Including qualitativeresearch in systematic reviews opportunities and problems JEva l Clin Pract 2001 7 125ndash33

6 Noblit G Hare R Meta-Ethnography Synthesising Qua litativeStudies Newbury Park Sage 1988

7 Humphris D Types of evidence In Harmer S Collinson G edsAchieving Evidence-Based Practice a Handbook for PractitionersLondon Bailliere Tindall 1999 13ndash39

8 Pearson A Evidence-based nursing quality through researchIn Nay R Garratt S eds Nursing O lder People Issues andInnovations Sydney Maclennan amp Petty 1999 338ndash52

9 Evans D Pearson A Systematic reviews gatekeepers of nursingknow ledge J Clin Nurs 2001 10 593ndash9

10 Salipante P Notz W Bigelow J A matrix approach to literaturereviews In Staw BM Cumm ings LL eds Research in Organi-zationa l Behavior Greenw ich JAI Press 1982 321ndash48

11 Pearson A Excellence in care Future dimensions for effectivenursing NT Res 1998 3 25ndash7

12 Lemmer B Grellier R Steven J Systematic review of nonran-dom and qualitative research literature exploring and uncov-ering an evidence base for health visiting and decision makingQua litative Hea lth Res 1999 9 315ndash28

13 Popay J Williams G Qualitative research and evidence-basedhealthcare J Roya l Soc Med 1998 91 32ndash7

14 Green J Britten N Qualitative research and evidence basedmedicine BMJ 1998 316 1230ndash32

15 Sandelowski M Docherty S Emden C Qualitative metasynthe-sis issues and techniques Res Nurs Hea lth 1997 20 365ndash71

16 Barroso J Powell-Cope G Metasynthesis of qualitative researchon living w ith HIV infection Qua litative Hea lth Res 2000 10340ndash53

17 Campbell R Pound P Pope C et a l Evaluating meta-ethnogra-phy a synthesis of qualitative research on lay experiences ofdiabetes and diabetes care Soc Sci Med 2003 56 671ndash84

18 Rees R Harden A Shepherd J Brunton G O liver S O akley AYoung People and Physica l Activity A Systematic Review ofResearch on Barriers and Facilitators London EPPI-Centre Uni-versity of London 2001 Available from http eppiioeacuk

19 Jensen L Allen M Meta-synthesis of qualitative findings Qua l-itative Hea lth Res 1996 6 553ndash60

Balancing the evidence 55

button When lsquoAssessmentrsquo is clicked in the menu to the leftof the screen the Assessment Summary screen w ill be dis-played If the Primary Reviewer clicks the ltAdd Primarygtbutton the Assessment Edit screen w ill appear The reviewerthen answers each of the 10 questions by selecting from thedrop-down menus to the right of the questions (see Assess-ment Edit screen at Fig 3) The reviewer then includes orexcludes the Study by using the drop-down menu at thebottom of the questions If the study is excluded an expla-nation is entered in the reason field

The Secondary Reviewer can now assess the study TheSecondary Reviewer w ill follow the same steps as the PrimaryReviewer but click the ltAdd Secondarygt button

Final assessment

Once both initial assessments have been completed thePrimary Reviewer can perform the final assessment

When both assessments are designated lsquoincludedrsquo finalanswers for the 10 questions must be decided as theresponses may not be the same When the final assessmentis to lsquoexcludersquo the study the Exclude Reason fields must becombined

Data extraction is then performed on included studies(Fig 4) The Primary Reviewer can add their Extraction on aStudy An Extraction must be done on a Study before theFindings can be associated to it

A methodology usually covers the theoretical foundationsof the research A list of methodologies appears in AppendixI This list is not exhaustive and where possible more detailsshould be added (eg ethnography may be critical orfem inist)

Method is the way that the data is collected and a list ofmethods again not exhaustive is provided below It isimportant to be as specific as possible For example if inter-

Figure 3 Assessment edit screen

56 A Pearson

view is selected it is important to specify what type ofinterview open-ended sem istructured face-to-face ortelephonebull interviewbull media analysisbull field notesbull discourse analysisbull observationbull surveybull questionnaire

An intervention is a planned change made to the researchsituation by the researcher as part of the research projectFor example an intervention could be serving lunch at10 am in a nursing home or providing an education inter-vention However there w ill not necessarily be an interven-tion in qualitative research and this field may either refer toan activity or phenomenon or be left incomplete

Setting and Context refers to the specific location wherethe research is conducted For example the setting could

be a nursing home a hospital or a dementia specific wardin a subacute hospital Some research w ill have no settingat all (eg discourse analysis)

The Geographical Context refers to the specific locationof the research For example Poland Austria or rural NewZealand

The Cultural Context refers to the cultural features of thestudy setting such as time period (eg 16th century) ethnicgroupings (eg indigenous Australians) age groupings (egolder people living in the community) or socio-econom icgroups (eg professional) These data should be as specificas possible and m ight also identify employment lifestylelevel of functionality and gender factors as well as partici-pation rate Information entered in this field should relate tothe inclusion and exclusion criteria of the research Ambig-uous terms or group names should also be defined in thissection (eg a carer is a personal care attendant)

Data analysis refers to the techniques utilised to analysethe data A list of examples is provided below This list is not

Figure 4 Extraction details screen

Balancing the evidence 57

exhaustive and should be supplemented w ith specific infor-mation where appropriatebull named software programsbull contextual analysisbull comparative analysisbull thematic analysisbull discourse analysisbull content analysis

A Reviewer can add Findings to a Study but only after anExtraction is completed on that Study (Fig 5) If a Studyrsquosstatus changes to lsquoexcludedrsquo then the related Findings w illalso be excluded (but not deleted)

Once all of the information on a Review is collected in theform of Findings and Extractions the Findings can be allo-cated to user-defined categories (Fig 6) To develop cate-gories ltCategorisegt is selected in the main menu A screenw ill appear that lists all of the Findings that have beencreated for the review A category should be established forthose Findings that can be naturally grouped

Once all of the Findings have been lsquocategorisedrsquo thecategories can then be designated to user-defined Synthe-sised Findings (Fig 7) To develop Synthesised FindingsltSynthesisgt is selected in the main menu A screen w illappear that lists all of the findings that have been created

for the review This w ill be grouped by the categories allo-cated to the findings A Synthesised Finding should beestablished for those categories that can be naturallygrouped

Discussion

This project set out to elicit whether or not it would bepossible to integrate qualitative research findings into thesystematic review process and if it was possible to developa system to do so The outcome is a prototype systemdesigned to enable systematic reviews to follow a rigorousprocess in appraising and synthesising qualitative data Thisdevelopment augments a larger project currently inprogress a System for the Unified Management of theAssessment and Review of Information (SUMARI) SUMARIis designed to enable reviewers to incorporate a w ider rangeof findings than those currently accommodated w ithin anemerging review orthodoxy The QARI forms one module ofSUMARI

The Comprehensive Systematic Review (CSR) is predi-cated on the view that the results of well-designed researchstudies ndash grounded in any methodological position ndash pro-vide more rigorous evidence than anecdotes or personal

Figure 5 Add findings to a study

58 A Pearson

opinion but that in the absence of such results opinionderiving from experience and expertise can still legitimatelybe regarded as the lsquobest availablersquo evidence

The CSR is an approach to evidence review that enablesreviewers to consider evidence of Feasibility Appropriate-ness Meaningfulness and Effectiveness in the form of afocused review of one two or more evidence types

SUMARI

SUMARI has been developed to enable systematic reviewersto take an inclusive view of what counts as evidenceSUMARI is a developing software package designed to assisthealth and other researchers and practitioners to conductsystematic reviews of evidence of Feasibility Appropriate-ness Meaningfulness and Effectiveness and to conducteconom ic evaluations of activities and interventions Thepackage consists of five modules (Fig 8)

Module 1 Comprehensive Review Management System (CReMS)This module includes the review protocol search results anda reporting function It is designed to manage a systematicreview and captures the results generated through the fouranalytical modules and formats them into a final report

Module 2 Qua litative Assessment and Review Instrument (QARI)This module is designed to facilitate critical appraisal dataextraction and synthesis of the findings of qualitative studies

Module 3 Meta Ana lysis of Statistics Assessment and Review Instrument (MAStARI)This module is designed to conduct the meta-analysis of theresults of comparable cohort time series and descriptivestudies using a number of statistical approaches

Module 4 Narrative Opinion and Text Assessment and Review Instrument (N OTARI)This module is designed to facilitate critical appraisal dataextraction and synthesis of expert opinion texts and reports

Module 5 Ana lysis of Cost Technology and Utilisation Assessment and Review Instrument (ACTUARI)This module is designed to facilitate critical appraisal dataextraction and synthesis of econom ic data

Using the package

The CReMS module is web based and when downloadedon the userrsquos server can be accessed on the web by those

Figure 6 Finding details screen

Balancing the evidence 59

authorised by the user It can be used as a stand-aloneprogram or in conjunction w ith other SUMARI modules

Each of the other SUMARI modules are also web basedand are designed to interface w ith CReMS and all othermodules Reviewers who w ish to utilise the functions of aspecific module can also use them as stand-alone programs

The total package is designed so that each module inter-acts w ith the others and a reviewer can at the point in the

review when critical appraisal data extraction and data syn-thesismeta-analysis is reached select a pathway to manageRCT data non-RCT quantitative data qualitative data tex-tual data from opinion papers or reports or econom ic dataA single focus review (eg a review of effectiveness) wouldfollow the RCT data pathway only and extract and analyseonly results from RCTs A review w ith more than one focuscan select any number of pathways For example a reviewof effectiveness and feasibility may enter data from actionresearch and evaluative studies into the QARI pathway anddata from reports of learned bodies into the N OTARI path-way as well as RCT results

Conclusion

This paper reports on a system that presents a practicalapproach to developing implementing and evaluating prac-

Figure 7 Allocating synthesised findings to categories

Figure 8 Modules in the System for the Unified Management ofthe Assessment and Review of Information (SUMARI)

60 A Pearson

tice based on lsquoevidencersquo in its broadest sense In addition toexam ining the concept of clinical effectiveness and theCochrane Collaboration approach to the meta-analysis ofquantitative research findings other non-quantitative formsof evidence and how they can be used as appropriatesources of evidence for practice have also been consideredThis has been done w ith a view to providing a practicalprocess using the QARI system which w ill enable the com-prehensive review of qualitative evidence for clinical prac-tice This project continues to evolve w ith the primary aimof re-conceptualising the concept of evidence for practicebecause health-care practices are often far more complexthan they immediately appear

The international interest in evidence-based practice aris-ing largely out of the work of the Cochrane Collaborationis likely to accelerate given the global concerns aboutimproving health care increasing the effectiveness andappropriateness of health interventions and containing thecosts of delivering health services Although the meta-analysis of the results of research into effectiveness is nowhighly refined there is still much work to be done beforethis can be seen to be m irrored in regard to the results ofqualitative research findings A number of research groupsare exam ining and developing ways to advance the incor-poration of qualitative evidence into systematic reviews TheQARI software program is currently being used by groups inCanada Scotland England Thailand and Australia and QARIlicenses are now available A number of completed reviewsare currently being prepared for publication and it is antic-ipated that feedback from the readers of these reviews thegroups currently using QARI and new QARI licenses w ill leadto further work and to the development of a more fullyrefined methodology and software program

Acknowledgements

The author and The Joanna Briggs Institute acknow ledge theparticipation of the follow ing members of the QARI devel-opment group and the contribution they have made to thedevelopment of QARIbull Professor Mary FitzGerald University of Newcastle

Newcastle Australiabull Associate Professor Jane Stein-Parbury University of Tech-

nology Sydney Australiabull Professor Colin Holmes James Cook University of North-

ern Queensland Townsville Australiabull Professor Desley Hegney University of Southern

Queensland Toowoomba and The University ofQueensland Brisbane Australia

bull Professor Ken Walsh Victoria University of WellingtonWellington New Zealand

bull Professor Karen Francis Monash University MelbourneAustralia

bull Mr Matt Lew is La Trobe University Melbourne Australiaand

bull Ms Cathy Ward La Trobe University MelbourneAustralia

References1 Sackett DL Rosenberg WM Gray JA Haynes RB Richardson

WS Evidence-based medicine what it is and what it isnrsquot BrMed J 1996 312 71ndash2

2 FitzGerald M The practical implications of a critique of tradi-tional science Int J Nurs Prac 1995 1 2ndash11

3 Pearsall J Trumble B eds Oxford English Reference D ictionaryOxford Oxford University Press 1995 487

4 Audi R The Cambridge D ictionary of Philosophy CambridgeCambridge University Press 1995

5 D ixon-Woods M Fitzpatrick R Roberts K Including qualitativeresearch in systematic reviews opportunities and problems JEva l Clin Pract 2001 7 125ndash33

6 Noblit G Hare R Meta-Ethnography Synthesising Qua litativeStudies Newbury Park Sage 1988

7 Humphris D Types of evidence In Harmer S Collinson G edsAchieving Evidence-Based Practice a Handbook for PractitionersLondon Bailliere Tindall 1999 13ndash39

8 Pearson A Evidence-based nursing quality through researchIn Nay R Garratt S eds Nursing O lder People Issues andInnovations Sydney Maclennan amp Petty 1999 338ndash52

9 Evans D Pearson A Systematic reviews gatekeepers of nursingknow ledge J Clin Nurs 2001 10 593ndash9

10 Salipante P Notz W Bigelow J A matrix approach to literaturereviews In Staw BM Cumm ings LL eds Research in Organi-zationa l Behavior Greenw ich JAI Press 1982 321ndash48

11 Pearson A Excellence in care Future dimensions for effectivenursing NT Res 1998 3 25ndash7

12 Lemmer B Grellier R Steven J Systematic review of nonran-dom and qualitative research literature exploring and uncov-ering an evidence base for health visiting and decision makingQua litative Hea lth Res 1999 9 315ndash28

13 Popay J Williams G Qualitative research and evidence-basedhealthcare J Roya l Soc Med 1998 91 32ndash7

14 Green J Britten N Qualitative research and evidence basedmedicine BMJ 1998 316 1230ndash32

15 Sandelowski M Docherty S Emden C Qualitative metasynthe-sis issues and techniques Res Nurs Hea lth 1997 20 365ndash71

16 Barroso J Powell-Cope G Metasynthesis of qualitative researchon living w ith HIV infection Qua litative Hea lth Res 2000 10340ndash53

17 Campbell R Pound P Pope C et a l Evaluating meta-ethnogra-phy a synthesis of qualitative research on lay experiences ofdiabetes and diabetes care Soc Sci Med 2003 56 671ndash84

18 Rees R Harden A Shepherd J Brunton G O liver S O akley AYoung People and Physica l Activity A Systematic Review ofResearch on Barriers and Facilitators London EPPI-Centre Uni-versity of London 2001 Available from http eppiioeacuk

19 Jensen L Allen M Meta-synthesis of qualitative findings Qua l-itative Hea lth Res 1996 6 553ndash60

56 A Pearson

view is selected it is important to specify what type ofinterview open-ended sem istructured face-to-face ortelephonebull interviewbull media analysisbull field notesbull discourse analysisbull observationbull surveybull questionnaire

An intervention is a planned change made to the researchsituation by the researcher as part of the research projectFor example an intervention could be serving lunch at10 am in a nursing home or providing an education inter-vention However there w ill not necessarily be an interven-tion in qualitative research and this field may either refer toan activity or phenomenon or be left incomplete

Setting and Context refers to the specific location wherethe research is conducted For example the setting could

be a nursing home a hospital or a dementia specific wardin a subacute hospital Some research w ill have no settingat all (eg discourse analysis)

The Geographical Context refers to the specific locationof the research For example Poland Austria or rural NewZealand

The Cultural Context refers to the cultural features of thestudy setting such as time period (eg 16th century) ethnicgroupings (eg indigenous Australians) age groupings (egolder people living in the community) or socio-econom icgroups (eg professional) These data should be as specificas possible and m ight also identify employment lifestylelevel of functionality and gender factors as well as partici-pation rate Information entered in this field should relate tothe inclusion and exclusion criteria of the research Ambig-uous terms or group names should also be defined in thissection (eg a carer is a personal care attendant)

Data analysis refers to the techniques utilised to analysethe data A list of examples is provided below This list is not

Figure 4 Extraction details screen

Balancing the evidence 57

exhaustive and should be supplemented w ith specific infor-mation where appropriatebull named software programsbull contextual analysisbull comparative analysisbull thematic analysisbull discourse analysisbull content analysis

A Reviewer can add Findings to a Study but only after anExtraction is completed on that Study (Fig 5) If a Studyrsquosstatus changes to lsquoexcludedrsquo then the related Findings w illalso be excluded (but not deleted)

Once all of the information on a Review is collected in theform of Findings and Extractions the Findings can be allo-cated to user-defined categories (Fig 6) To develop cate-gories ltCategorisegt is selected in the main menu A screenw ill appear that lists all of the Findings that have beencreated for the review A category should be established forthose Findings that can be naturally grouped

Once all of the Findings have been lsquocategorisedrsquo thecategories can then be designated to user-defined Synthe-sised Findings (Fig 7) To develop Synthesised FindingsltSynthesisgt is selected in the main menu A screen w illappear that lists all of the findings that have been created

for the review This w ill be grouped by the categories allo-cated to the findings A Synthesised Finding should beestablished for those categories that can be naturallygrouped

Discussion

This project set out to elicit whether or not it would bepossible to integrate qualitative research findings into thesystematic review process and if it was possible to developa system to do so The outcome is a prototype systemdesigned to enable systematic reviews to follow a rigorousprocess in appraising and synthesising qualitative data Thisdevelopment augments a larger project currently inprogress a System for the Unified Management of theAssessment and Review of Information (SUMARI) SUMARIis designed to enable reviewers to incorporate a w ider rangeof findings than those currently accommodated w ithin anemerging review orthodoxy The QARI forms one module ofSUMARI

The Comprehensive Systematic Review (CSR) is predi-cated on the view that the results of well-designed researchstudies ndash grounded in any methodological position ndash pro-vide more rigorous evidence than anecdotes or personal

Figure 5 Add findings to a study

58 A Pearson

opinion but that in the absence of such results opinionderiving from experience and expertise can still legitimatelybe regarded as the lsquobest availablersquo evidence

The CSR is an approach to evidence review that enablesreviewers to consider evidence of Feasibility Appropriate-ness Meaningfulness and Effectiveness in the form of afocused review of one two or more evidence types

SUMARI

SUMARI has been developed to enable systematic reviewersto take an inclusive view of what counts as evidenceSUMARI is a developing software package designed to assisthealth and other researchers and practitioners to conductsystematic reviews of evidence of Feasibility Appropriate-ness Meaningfulness and Effectiveness and to conducteconom ic evaluations of activities and interventions Thepackage consists of five modules (Fig 8)

Module 1 Comprehensive Review Management System (CReMS)This module includes the review protocol search results anda reporting function It is designed to manage a systematicreview and captures the results generated through the fouranalytical modules and formats them into a final report

Module 2 Qua litative Assessment and Review Instrument (QARI)This module is designed to facilitate critical appraisal dataextraction and synthesis of the findings of qualitative studies

Module 3 Meta Ana lysis of Statistics Assessment and Review Instrument (MAStARI)This module is designed to conduct the meta-analysis of theresults of comparable cohort time series and descriptivestudies using a number of statistical approaches

Module 4 Narrative Opinion and Text Assessment and Review Instrument (N OTARI)This module is designed to facilitate critical appraisal dataextraction and synthesis of expert opinion texts and reports

Module 5 Ana lysis of Cost Technology and Utilisation Assessment and Review Instrument (ACTUARI)This module is designed to facilitate critical appraisal dataextraction and synthesis of econom ic data

Using the package

The CReMS module is web based and when downloadedon the userrsquos server can be accessed on the web by those

Figure 6 Finding details screen

Balancing the evidence 59

authorised by the user It can be used as a stand-aloneprogram or in conjunction w ith other SUMARI modules

Each of the other SUMARI modules are also web basedand are designed to interface w ith CReMS and all othermodules Reviewers who w ish to utilise the functions of aspecific module can also use them as stand-alone programs

The total package is designed so that each module inter-acts w ith the others and a reviewer can at the point in the

review when critical appraisal data extraction and data syn-thesismeta-analysis is reached select a pathway to manageRCT data non-RCT quantitative data qualitative data tex-tual data from opinion papers or reports or econom ic dataA single focus review (eg a review of effectiveness) wouldfollow the RCT data pathway only and extract and analyseonly results from RCTs A review w ith more than one focuscan select any number of pathways For example a reviewof effectiveness and feasibility may enter data from actionresearch and evaluative studies into the QARI pathway anddata from reports of learned bodies into the N OTARI path-way as well as RCT results

Conclusion

This paper reports on a system that presents a practicalapproach to developing implementing and evaluating prac-

Figure 7 Allocating synthesised findings to categories

Figure 8 Modules in the System for the Unified Management ofthe Assessment and Review of Information (SUMARI)

60 A Pearson

tice based on lsquoevidencersquo in its broadest sense In addition toexam ining the concept of clinical effectiveness and theCochrane Collaboration approach to the meta-analysis ofquantitative research findings other non-quantitative formsof evidence and how they can be used as appropriatesources of evidence for practice have also been consideredThis has been done w ith a view to providing a practicalprocess using the QARI system which w ill enable the com-prehensive review of qualitative evidence for clinical prac-tice This project continues to evolve w ith the primary aimof re-conceptualising the concept of evidence for practicebecause health-care practices are often far more complexthan they immediately appear

The international interest in evidence-based practice aris-ing largely out of the work of the Cochrane Collaborationis likely to accelerate given the global concerns aboutimproving health care increasing the effectiveness andappropriateness of health interventions and containing thecosts of delivering health services Although the meta-analysis of the results of research into effectiveness is nowhighly refined there is still much work to be done beforethis can be seen to be m irrored in regard to the results ofqualitative research findings A number of research groupsare exam ining and developing ways to advance the incor-poration of qualitative evidence into systematic reviews TheQARI software program is currently being used by groups inCanada Scotland England Thailand and Australia and QARIlicenses are now available A number of completed reviewsare currently being prepared for publication and it is antic-ipated that feedback from the readers of these reviews thegroups currently using QARI and new QARI licenses w ill leadto further work and to the development of a more fullyrefined methodology and software program

Acknowledgements

The author and The Joanna Briggs Institute acknow ledge theparticipation of the follow ing members of the QARI devel-opment group and the contribution they have made to thedevelopment of QARIbull Professor Mary FitzGerald University of Newcastle

Newcastle Australiabull Associate Professor Jane Stein-Parbury University of Tech-

nology Sydney Australiabull Professor Colin Holmes James Cook University of North-

ern Queensland Townsville Australiabull Professor Desley Hegney University of Southern

Queensland Toowoomba and The University ofQueensland Brisbane Australia

bull Professor Ken Walsh Victoria University of WellingtonWellington New Zealand

bull Professor Karen Francis Monash University MelbourneAustralia

bull Mr Matt Lew is La Trobe University Melbourne Australiaand

bull Ms Cathy Ward La Trobe University MelbourneAustralia

References1 Sackett DL Rosenberg WM Gray JA Haynes RB Richardson

WS Evidence-based medicine what it is and what it isnrsquot BrMed J 1996 312 71ndash2

2 FitzGerald M The practical implications of a critique of tradi-tional science Int J Nurs Prac 1995 1 2ndash11

3 Pearsall J Trumble B eds Oxford English Reference D ictionaryOxford Oxford University Press 1995 487

4 Audi R The Cambridge D ictionary of Philosophy CambridgeCambridge University Press 1995

5 D ixon-Woods M Fitzpatrick R Roberts K Including qualitativeresearch in systematic reviews opportunities and problems JEva l Clin Pract 2001 7 125ndash33

6 Noblit G Hare R Meta-Ethnography Synthesising Qua litativeStudies Newbury Park Sage 1988

7 Humphris D Types of evidence In Harmer S Collinson G edsAchieving Evidence-Based Practice a Handbook for PractitionersLondon Bailliere Tindall 1999 13ndash39

8 Pearson A Evidence-based nursing quality through researchIn Nay R Garratt S eds Nursing O lder People Issues andInnovations Sydney Maclennan amp Petty 1999 338ndash52

9 Evans D Pearson A Systematic reviews gatekeepers of nursingknow ledge J Clin Nurs 2001 10 593ndash9

10 Salipante P Notz W Bigelow J A matrix approach to literaturereviews In Staw BM Cumm ings LL eds Research in Organi-zationa l Behavior Greenw ich JAI Press 1982 321ndash48

11 Pearson A Excellence in care Future dimensions for effectivenursing NT Res 1998 3 25ndash7

12 Lemmer B Grellier R Steven J Systematic review of nonran-dom and qualitative research literature exploring and uncov-ering an evidence base for health visiting and decision makingQua litative Hea lth Res 1999 9 315ndash28

13 Popay J Williams G Qualitative research and evidence-basedhealthcare J Roya l Soc Med 1998 91 32ndash7

14 Green J Britten N Qualitative research and evidence basedmedicine BMJ 1998 316 1230ndash32

15 Sandelowski M Docherty S Emden C Qualitative metasynthe-sis issues and techniques Res Nurs Hea lth 1997 20 365ndash71

16 Barroso J Powell-Cope G Metasynthesis of qualitative researchon living w ith HIV infection Qua litative Hea lth Res 2000 10340ndash53

17 Campbell R Pound P Pope C et a l Evaluating meta-ethnogra-phy a synthesis of qualitative research on lay experiences ofdiabetes and diabetes care Soc Sci Med 2003 56 671ndash84

18 Rees R Harden A Shepherd J Brunton G O liver S O akley AYoung People and Physica l Activity A Systematic Review ofResearch on Barriers and Facilitators London EPPI-Centre Uni-versity of London 2001 Available from http eppiioeacuk

19 Jensen L Allen M Meta-synthesis of qualitative findings Qua l-itative Hea lth Res 1996 6 553ndash60

Balancing the evidence 57

exhaustive and should be supplemented w ith specific infor-mation where appropriatebull named software programsbull contextual analysisbull comparative analysisbull thematic analysisbull discourse analysisbull content analysis

A Reviewer can add Findings to a Study but only after anExtraction is completed on that Study (Fig 5) If a Studyrsquosstatus changes to lsquoexcludedrsquo then the related Findings w illalso be excluded (but not deleted)

Once all of the information on a Review is collected in theform of Findings and Extractions the Findings can be allo-cated to user-defined categories (Fig 6) To develop cate-gories ltCategorisegt is selected in the main menu A screenw ill appear that lists all of the Findings that have beencreated for the review A category should be established forthose Findings that can be naturally grouped

Once all of the Findings have been lsquocategorisedrsquo thecategories can then be designated to user-defined Synthe-sised Findings (Fig 7) To develop Synthesised FindingsltSynthesisgt is selected in the main menu A screen w illappear that lists all of the findings that have been created

for the review This w ill be grouped by the categories allo-cated to the findings A Synthesised Finding should beestablished for those categories that can be naturallygrouped

Discussion

This project set out to elicit whether or not it would bepossible to integrate qualitative research findings into thesystematic review process and if it was possible to developa system to do so The outcome is a prototype systemdesigned to enable systematic reviews to follow a rigorousprocess in appraising and synthesising qualitative data Thisdevelopment augments a larger project currently inprogress a System for the Unified Management of theAssessment and Review of Information (SUMARI) SUMARIis designed to enable reviewers to incorporate a w ider rangeof findings than those currently accommodated w ithin anemerging review orthodoxy The QARI forms one module ofSUMARI

The Comprehensive Systematic Review (CSR) is predi-cated on the view that the results of well-designed researchstudies ndash grounded in any methodological position ndash pro-vide more rigorous evidence than anecdotes or personal

Figure 5 Add findings to a study

58 A Pearson

opinion but that in the absence of such results opinionderiving from experience and expertise can still legitimatelybe regarded as the lsquobest availablersquo evidence

The CSR is an approach to evidence review that enablesreviewers to consider evidence of Feasibility Appropriate-ness Meaningfulness and Effectiveness in the form of afocused review of one two or more evidence types

SUMARI

SUMARI has been developed to enable systematic reviewersto take an inclusive view of what counts as evidenceSUMARI is a developing software package designed to assisthealth and other researchers and practitioners to conductsystematic reviews of evidence of Feasibility Appropriate-ness Meaningfulness and Effectiveness and to conducteconom ic evaluations of activities and interventions Thepackage consists of five modules (Fig 8)

Module 1 Comprehensive Review Management System (CReMS)This module includes the review protocol search results anda reporting function It is designed to manage a systematicreview and captures the results generated through the fouranalytical modules and formats them into a final report

Module 2 Qua litative Assessment and Review Instrument (QARI)This module is designed to facilitate critical appraisal dataextraction and synthesis of the findings of qualitative studies

Module 3 Meta Ana lysis of Statistics Assessment and Review Instrument (MAStARI)This module is designed to conduct the meta-analysis of theresults of comparable cohort time series and descriptivestudies using a number of statistical approaches

Module 4 Narrative Opinion and Text Assessment and Review Instrument (N OTARI)This module is designed to facilitate critical appraisal dataextraction and synthesis of expert opinion texts and reports

Module 5 Ana lysis of Cost Technology and Utilisation Assessment and Review Instrument (ACTUARI)This module is designed to facilitate critical appraisal dataextraction and synthesis of econom ic data

Using the package

The CReMS module is web based and when downloadedon the userrsquos server can be accessed on the web by those

Figure 6 Finding details screen

Balancing the evidence 59

authorised by the user It can be used as a stand-aloneprogram or in conjunction w ith other SUMARI modules

Each of the other SUMARI modules are also web basedand are designed to interface w ith CReMS and all othermodules Reviewers who w ish to utilise the functions of aspecific module can also use them as stand-alone programs

The total package is designed so that each module inter-acts w ith the others and a reviewer can at the point in the

review when critical appraisal data extraction and data syn-thesismeta-analysis is reached select a pathway to manageRCT data non-RCT quantitative data qualitative data tex-tual data from opinion papers or reports or econom ic dataA single focus review (eg a review of effectiveness) wouldfollow the RCT data pathway only and extract and analyseonly results from RCTs A review w ith more than one focuscan select any number of pathways For example a reviewof effectiveness and feasibility may enter data from actionresearch and evaluative studies into the QARI pathway anddata from reports of learned bodies into the N OTARI path-way as well as RCT results

Conclusion

This paper reports on a system that presents a practicalapproach to developing implementing and evaluating prac-

Figure 7 Allocating synthesised findings to categories

Figure 8 Modules in the System for the Unified Management ofthe Assessment and Review of Information (SUMARI)

60 A Pearson

tice based on lsquoevidencersquo in its broadest sense In addition toexam ining the concept of clinical effectiveness and theCochrane Collaboration approach to the meta-analysis ofquantitative research findings other non-quantitative formsof evidence and how they can be used as appropriatesources of evidence for practice have also been consideredThis has been done w ith a view to providing a practicalprocess using the QARI system which w ill enable the com-prehensive review of qualitative evidence for clinical prac-tice This project continues to evolve w ith the primary aimof re-conceptualising the concept of evidence for practicebecause health-care practices are often far more complexthan they immediately appear

The international interest in evidence-based practice aris-ing largely out of the work of the Cochrane Collaborationis likely to accelerate given the global concerns aboutimproving health care increasing the effectiveness andappropriateness of health interventions and containing thecosts of delivering health services Although the meta-analysis of the results of research into effectiveness is nowhighly refined there is still much work to be done beforethis can be seen to be m irrored in regard to the results ofqualitative research findings A number of research groupsare exam ining and developing ways to advance the incor-poration of qualitative evidence into systematic reviews TheQARI software program is currently being used by groups inCanada Scotland England Thailand and Australia and QARIlicenses are now available A number of completed reviewsare currently being prepared for publication and it is antic-ipated that feedback from the readers of these reviews thegroups currently using QARI and new QARI licenses w ill leadto further work and to the development of a more fullyrefined methodology and software program

Acknowledgements

The author and The Joanna Briggs Institute acknow ledge theparticipation of the follow ing members of the QARI devel-opment group and the contribution they have made to thedevelopment of QARIbull Professor Mary FitzGerald University of Newcastle

Newcastle Australiabull Associate Professor Jane Stein-Parbury University of Tech-

nology Sydney Australiabull Professor Colin Holmes James Cook University of North-

ern Queensland Townsville Australiabull Professor Desley Hegney University of Southern

Queensland Toowoomba and The University ofQueensland Brisbane Australia

bull Professor Ken Walsh Victoria University of WellingtonWellington New Zealand

bull Professor Karen Francis Monash University MelbourneAustralia

bull Mr Matt Lew is La Trobe University Melbourne Australiaand

bull Ms Cathy Ward La Trobe University MelbourneAustralia

References1 Sackett DL Rosenberg WM Gray JA Haynes RB Richardson

WS Evidence-based medicine what it is and what it isnrsquot BrMed J 1996 312 71ndash2

2 FitzGerald M The practical implications of a critique of tradi-tional science Int J Nurs Prac 1995 1 2ndash11

3 Pearsall J Trumble B eds Oxford English Reference D ictionaryOxford Oxford University Press 1995 487

4 Audi R The Cambridge D ictionary of Philosophy CambridgeCambridge University Press 1995

5 D ixon-Woods M Fitzpatrick R Roberts K Including qualitativeresearch in systematic reviews opportunities and problems JEva l Clin Pract 2001 7 125ndash33

6 Noblit G Hare R Meta-Ethnography Synthesising Qua litativeStudies Newbury Park Sage 1988

7 Humphris D Types of evidence In Harmer S Collinson G edsAchieving Evidence-Based Practice a Handbook for PractitionersLondon Bailliere Tindall 1999 13ndash39

8 Pearson A Evidence-based nursing quality through researchIn Nay R Garratt S eds Nursing O lder People Issues andInnovations Sydney Maclennan amp Petty 1999 338ndash52

9 Evans D Pearson A Systematic reviews gatekeepers of nursingknow ledge J Clin Nurs 2001 10 593ndash9

10 Salipante P Notz W Bigelow J A matrix approach to literaturereviews In Staw BM Cumm ings LL eds Research in Organi-zationa l Behavior Greenw ich JAI Press 1982 321ndash48

11 Pearson A Excellence in care Future dimensions for effectivenursing NT Res 1998 3 25ndash7

12 Lemmer B Grellier R Steven J Systematic review of nonran-dom and qualitative research literature exploring and uncov-ering an evidence base for health visiting and decision makingQua litative Hea lth Res 1999 9 315ndash28

13 Popay J Williams G Qualitative research and evidence-basedhealthcare J Roya l Soc Med 1998 91 32ndash7

14 Green J Britten N Qualitative research and evidence basedmedicine BMJ 1998 316 1230ndash32

15 Sandelowski M Docherty S Emden C Qualitative metasynthe-sis issues and techniques Res Nurs Hea lth 1997 20 365ndash71

16 Barroso J Powell-Cope G Metasynthesis of qualitative researchon living w ith HIV infection Qua litative Hea lth Res 2000 10340ndash53

17 Campbell R Pound P Pope C et a l Evaluating meta-ethnogra-phy a synthesis of qualitative research on lay experiences ofdiabetes and diabetes care Soc Sci Med 2003 56 671ndash84

18 Rees R Harden A Shepherd J Brunton G O liver S O akley AYoung People and Physica l Activity A Systematic Review ofResearch on Barriers and Facilitators London EPPI-Centre Uni-versity of London 2001 Available from http eppiioeacuk

19 Jensen L Allen M Meta-synthesis of qualitative findings Qua l-itative Hea lth Res 1996 6 553ndash60

58 A Pearson

opinion but that in the absence of such results opinionderiving from experience and expertise can still legitimatelybe regarded as the lsquobest availablersquo evidence

The CSR is an approach to evidence review that enablesreviewers to consider evidence of Feasibility Appropriate-ness Meaningfulness and Effectiveness in the form of afocused review of one two or more evidence types

SUMARI

SUMARI has been developed to enable systematic reviewersto take an inclusive view of what counts as evidenceSUMARI is a developing software package designed to assisthealth and other researchers and practitioners to conductsystematic reviews of evidence of Feasibility Appropriate-ness Meaningfulness and Effectiveness and to conducteconom ic evaluations of activities and interventions Thepackage consists of five modules (Fig 8)

Module 1 Comprehensive Review Management System (CReMS)This module includes the review protocol search results anda reporting function It is designed to manage a systematicreview and captures the results generated through the fouranalytical modules and formats them into a final report

Module 2 Qua litative Assessment and Review Instrument (QARI)This module is designed to facilitate critical appraisal dataextraction and synthesis of the findings of qualitative studies

Module 3 Meta Ana lysis of Statistics Assessment and Review Instrument (MAStARI)This module is designed to conduct the meta-analysis of theresults of comparable cohort time series and descriptivestudies using a number of statistical approaches

Module 4 Narrative Opinion and Text Assessment and Review Instrument (N OTARI)This module is designed to facilitate critical appraisal dataextraction and synthesis of expert opinion texts and reports

Module 5 Ana lysis of Cost Technology and Utilisation Assessment and Review Instrument (ACTUARI)This module is designed to facilitate critical appraisal dataextraction and synthesis of econom ic data

Using the package

The CReMS module is web based and when downloadedon the userrsquos server can be accessed on the web by those

Figure 6 Finding details screen

Balancing the evidence 59

authorised by the user It can be used as a stand-aloneprogram or in conjunction w ith other SUMARI modules

Each of the other SUMARI modules are also web basedand are designed to interface w ith CReMS and all othermodules Reviewers who w ish to utilise the functions of aspecific module can also use them as stand-alone programs

The total package is designed so that each module inter-acts w ith the others and a reviewer can at the point in the

review when critical appraisal data extraction and data syn-thesismeta-analysis is reached select a pathway to manageRCT data non-RCT quantitative data qualitative data tex-tual data from opinion papers or reports or econom ic dataA single focus review (eg a review of effectiveness) wouldfollow the RCT data pathway only and extract and analyseonly results from RCTs A review w ith more than one focuscan select any number of pathways For example a reviewof effectiveness and feasibility may enter data from actionresearch and evaluative studies into the QARI pathway anddata from reports of learned bodies into the N OTARI path-way as well as RCT results

Conclusion

This paper reports on a system that presents a practicalapproach to developing implementing and evaluating prac-

Figure 7 Allocating synthesised findings to categories

Figure 8 Modules in the System for the Unified Management ofthe Assessment and Review of Information (SUMARI)

60 A Pearson

tice based on lsquoevidencersquo in its broadest sense In addition toexam ining the concept of clinical effectiveness and theCochrane Collaboration approach to the meta-analysis ofquantitative research findings other non-quantitative formsof evidence and how they can be used as appropriatesources of evidence for practice have also been consideredThis has been done w ith a view to providing a practicalprocess using the QARI system which w ill enable the com-prehensive review of qualitative evidence for clinical prac-tice This project continues to evolve w ith the primary aimof re-conceptualising the concept of evidence for practicebecause health-care practices are often far more complexthan they immediately appear

The international interest in evidence-based practice aris-ing largely out of the work of the Cochrane Collaborationis likely to accelerate given the global concerns aboutimproving health care increasing the effectiveness andappropriateness of health interventions and containing thecosts of delivering health services Although the meta-analysis of the results of research into effectiveness is nowhighly refined there is still much work to be done beforethis can be seen to be m irrored in regard to the results ofqualitative research findings A number of research groupsare exam ining and developing ways to advance the incor-poration of qualitative evidence into systematic reviews TheQARI software program is currently being used by groups inCanada Scotland England Thailand and Australia and QARIlicenses are now available A number of completed reviewsare currently being prepared for publication and it is antic-ipated that feedback from the readers of these reviews thegroups currently using QARI and new QARI licenses w ill leadto further work and to the development of a more fullyrefined methodology and software program

Acknowledgements

The author and The Joanna Briggs Institute acknow ledge theparticipation of the follow ing members of the QARI devel-opment group and the contribution they have made to thedevelopment of QARIbull Professor Mary FitzGerald University of Newcastle

Newcastle Australiabull Associate Professor Jane Stein-Parbury University of Tech-

nology Sydney Australiabull Professor Colin Holmes James Cook University of North-

ern Queensland Townsville Australiabull Professor Desley Hegney University of Southern

Queensland Toowoomba and The University ofQueensland Brisbane Australia

bull Professor Ken Walsh Victoria University of WellingtonWellington New Zealand

bull Professor Karen Francis Monash University MelbourneAustralia

bull Mr Matt Lew is La Trobe University Melbourne Australiaand

bull Ms Cathy Ward La Trobe University MelbourneAustralia

References1 Sackett DL Rosenberg WM Gray JA Haynes RB Richardson

WS Evidence-based medicine what it is and what it isnrsquot BrMed J 1996 312 71ndash2

2 FitzGerald M The practical implications of a critique of tradi-tional science Int J Nurs Prac 1995 1 2ndash11

3 Pearsall J Trumble B eds Oxford English Reference D ictionaryOxford Oxford University Press 1995 487

4 Audi R The Cambridge D ictionary of Philosophy CambridgeCambridge University Press 1995

5 D ixon-Woods M Fitzpatrick R Roberts K Including qualitativeresearch in systematic reviews opportunities and problems JEva l Clin Pract 2001 7 125ndash33

6 Noblit G Hare R Meta-Ethnography Synthesising Qua litativeStudies Newbury Park Sage 1988

7 Humphris D Types of evidence In Harmer S Collinson G edsAchieving Evidence-Based Practice a Handbook for PractitionersLondon Bailliere Tindall 1999 13ndash39

8 Pearson A Evidence-based nursing quality through researchIn Nay R Garratt S eds Nursing O lder People Issues andInnovations Sydney Maclennan amp Petty 1999 338ndash52

9 Evans D Pearson A Systematic reviews gatekeepers of nursingknow ledge J Clin Nurs 2001 10 593ndash9

10 Salipante P Notz W Bigelow J A matrix approach to literaturereviews In Staw BM Cumm ings LL eds Research in Organi-zationa l Behavior Greenw ich JAI Press 1982 321ndash48

11 Pearson A Excellence in care Future dimensions for effectivenursing NT Res 1998 3 25ndash7

12 Lemmer B Grellier R Steven J Systematic review of nonran-dom and qualitative research literature exploring and uncov-ering an evidence base for health visiting and decision makingQua litative Hea lth Res 1999 9 315ndash28

13 Popay J Williams G Qualitative research and evidence-basedhealthcare J Roya l Soc Med 1998 91 32ndash7

14 Green J Britten N Qualitative research and evidence basedmedicine BMJ 1998 316 1230ndash32

15 Sandelowski M Docherty S Emden C Qualitative metasynthe-sis issues and techniques Res Nurs Hea lth 1997 20 365ndash71

16 Barroso J Powell-Cope G Metasynthesis of qualitative researchon living w ith HIV infection Qua litative Hea lth Res 2000 10340ndash53

17 Campbell R Pound P Pope C et a l Evaluating meta-ethnogra-phy a synthesis of qualitative research on lay experiences ofdiabetes and diabetes care Soc Sci Med 2003 56 671ndash84

18 Rees R Harden A Shepherd J Brunton G O liver S O akley AYoung People and Physica l Activity A Systematic Review ofResearch on Barriers and Facilitators London EPPI-Centre Uni-versity of London 2001 Available from http eppiioeacuk

19 Jensen L Allen M Meta-synthesis of qualitative findings Qua l-itative Hea lth Res 1996 6 553ndash60

Balancing the evidence 59

authorised by the user It can be used as a stand-aloneprogram or in conjunction w ith other SUMARI modules

Each of the other SUMARI modules are also web basedand are designed to interface w ith CReMS and all othermodules Reviewers who w ish to utilise the functions of aspecific module can also use them as stand-alone programs

The total package is designed so that each module inter-acts w ith the others and a reviewer can at the point in the

review when critical appraisal data extraction and data syn-thesismeta-analysis is reached select a pathway to manageRCT data non-RCT quantitative data qualitative data tex-tual data from opinion papers or reports or econom ic dataA single focus review (eg a review of effectiveness) wouldfollow the RCT data pathway only and extract and analyseonly results from RCTs A review w ith more than one focuscan select any number of pathways For example a reviewof effectiveness and feasibility may enter data from actionresearch and evaluative studies into the QARI pathway anddata from reports of learned bodies into the N OTARI path-way as well as RCT results

Conclusion

This paper reports on a system that presents a practicalapproach to developing implementing and evaluating prac-

Figure 7 Allocating synthesised findings to categories

Figure 8 Modules in the System for the Unified Management ofthe Assessment and Review of Information (SUMARI)

60 A Pearson

tice based on lsquoevidencersquo in its broadest sense In addition toexam ining the concept of clinical effectiveness and theCochrane Collaboration approach to the meta-analysis ofquantitative research findings other non-quantitative formsof evidence and how they can be used as appropriatesources of evidence for practice have also been consideredThis has been done w ith a view to providing a practicalprocess using the QARI system which w ill enable the com-prehensive review of qualitative evidence for clinical prac-tice This project continues to evolve w ith the primary aimof re-conceptualising the concept of evidence for practicebecause health-care practices are often far more complexthan they immediately appear

The international interest in evidence-based practice aris-ing largely out of the work of the Cochrane Collaborationis likely to accelerate given the global concerns aboutimproving health care increasing the effectiveness andappropriateness of health interventions and containing thecosts of delivering health services Although the meta-analysis of the results of research into effectiveness is nowhighly refined there is still much work to be done beforethis can be seen to be m irrored in regard to the results ofqualitative research findings A number of research groupsare exam ining and developing ways to advance the incor-poration of qualitative evidence into systematic reviews TheQARI software program is currently being used by groups inCanada Scotland England Thailand and Australia and QARIlicenses are now available A number of completed reviewsare currently being prepared for publication and it is antic-ipated that feedback from the readers of these reviews thegroups currently using QARI and new QARI licenses w ill leadto further work and to the development of a more fullyrefined methodology and software program

Acknowledgements

The author and The Joanna Briggs Institute acknow ledge theparticipation of the follow ing members of the QARI devel-opment group and the contribution they have made to thedevelopment of QARIbull Professor Mary FitzGerald University of Newcastle

Newcastle Australiabull Associate Professor Jane Stein-Parbury University of Tech-

nology Sydney Australiabull Professor Colin Holmes James Cook University of North-

ern Queensland Townsville Australiabull Professor Desley Hegney University of Southern

Queensland Toowoomba and The University ofQueensland Brisbane Australia

bull Professor Ken Walsh Victoria University of WellingtonWellington New Zealand

bull Professor Karen Francis Monash University MelbourneAustralia

bull Mr Matt Lew is La Trobe University Melbourne Australiaand

bull Ms Cathy Ward La Trobe University MelbourneAustralia

References1 Sackett DL Rosenberg WM Gray JA Haynes RB Richardson

WS Evidence-based medicine what it is and what it isnrsquot BrMed J 1996 312 71ndash2

2 FitzGerald M The practical implications of a critique of tradi-tional science Int J Nurs Prac 1995 1 2ndash11

3 Pearsall J Trumble B eds Oxford English Reference D ictionaryOxford Oxford University Press 1995 487

4 Audi R The Cambridge D ictionary of Philosophy CambridgeCambridge University Press 1995

5 D ixon-Woods M Fitzpatrick R Roberts K Including qualitativeresearch in systematic reviews opportunities and problems JEva l Clin Pract 2001 7 125ndash33

6 Noblit G Hare R Meta-Ethnography Synthesising Qua litativeStudies Newbury Park Sage 1988

7 Humphris D Types of evidence In Harmer S Collinson G edsAchieving Evidence-Based Practice a Handbook for PractitionersLondon Bailliere Tindall 1999 13ndash39

8 Pearson A Evidence-based nursing quality through researchIn Nay R Garratt S eds Nursing O lder People Issues andInnovations Sydney Maclennan amp Petty 1999 338ndash52

9 Evans D Pearson A Systematic reviews gatekeepers of nursingknow ledge J Clin Nurs 2001 10 593ndash9

10 Salipante P Notz W Bigelow J A matrix approach to literaturereviews In Staw BM Cumm ings LL eds Research in Organi-zationa l Behavior Greenw ich JAI Press 1982 321ndash48

11 Pearson A Excellence in care Future dimensions for effectivenursing NT Res 1998 3 25ndash7

12 Lemmer B Grellier R Steven J Systematic review of nonran-dom and qualitative research literature exploring and uncov-ering an evidence base for health visiting and decision makingQua litative Hea lth Res 1999 9 315ndash28

13 Popay J Williams G Qualitative research and evidence-basedhealthcare J Roya l Soc Med 1998 91 32ndash7

14 Green J Britten N Qualitative research and evidence basedmedicine BMJ 1998 316 1230ndash32

15 Sandelowski M Docherty S Emden C Qualitative metasynthe-sis issues and techniques Res Nurs Hea lth 1997 20 365ndash71

16 Barroso J Powell-Cope G Metasynthesis of qualitative researchon living w ith HIV infection Qua litative Hea lth Res 2000 10340ndash53

17 Campbell R Pound P Pope C et a l Evaluating meta-ethnogra-phy a synthesis of qualitative research on lay experiences ofdiabetes and diabetes care Soc Sci Med 2003 56 671ndash84

18 Rees R Harden A Shepherd J Brunton G O liver S O akley AYoung People and Physica l Activity A Systematic Review ofResearch on Barriers and Facilitators London EPPI-Centre Uni-versity of London 2001 Available from http eppiioeacuk

19 Jensen L Allen M Meta-synthesis of qualitative findings Qua l-itative Hea lth Res 1996 6 553ndash60

60 A Pearson

tice based on lsquoevidencersquo in its broadest sense In addition toexam ining the concept of clinical effectiveness and theCochrane Collaboration approach to the meta-analysis ofquantitative research findings other non-quantitative formsof evidence and how they can be used as appropriatesources of evidence for practice have also been consideredThis has been done w ith a view to providing a practicalprocess using the QARI system which w ill enable the com-prehensive review of qualitative evidence for clinical prac-tice This project continues to evolve w ith the primary aimof re-conceptualising the concept of evidence for practicebecause health-care practices are often far more complexthan they immediately appear

The international interest in evidence-based practice aris-ing largely out of the work of the Cochrane Collaborationis likely to accelerate given the global concerns aboutimproving health care increasing the effectiveness andappropriateness of health interventions and containing thecosts of delivering health services Although the meta-analysis of the results of research into effectiveness is nowhighly refined there is still much work to be done beforethis can be seen to be m irrored in regard to the results ofqualitative research findings A number of research groupsare exam ining and developing ways to advance the incor-poration of qualitative evidence into systematic reviews TheQARI software program is currently being used by groups inCanada Scotland England Thailand and Australia and QARIlicenses are now available A number of completed reviewsare currently being prepared for publication and it is antic-ipated that feedback from the readers of these reviews thegroups currently using QARI and new QARI licenses w ill leadto further work and to the development of a more fullyrefined methodology and software program

Acknowledgements

The author and The Joanna Briggs Institute acknow ledge theparticipation of the follow ing members of the QARI devel-opment group and the contribution they have made to thedevelopment of QARIbull Professor Mary FitzGerald University of Newcastle

Newcastle Australiabull Associate Professor Jane Stein-Parbury University of Tech-

nology Sydney Australiabull Professor Colin Holmes James Cook University of North-

ern Queensland Townsville Australiabull Professor Desley Hegney University of Southern

Queensland Toowoomba and The University ofQueensland Brisbane Australia

bull Professor Ken Walsh Victoria University of WellingtonWellington New Zealand

bull Professor Karen Francis Monash University MelbourneAustralia

bull Mr Matt Lew is La Trobe University Melbourne Australiaand

bull Ms Cathy Ward La Trobe University MelbourneAustralia

References1 Sackett DL Rosenberg WM Gray JA Haynes RB Richardson

WS Evidence-based medicine what it is and what it isnrsquot BrMed J 1996 312 71ndash2

2 FitzGerald M The practical implications of a critique of tradi-tional science Int J Nurs Prac 1995 1 2ndash11

3 Pearsall J Trumble B eds Oxford English Reference D ictionaryOxford Oxford University Press 1995 487

4 Audi R The Cambridge D ictionary of Philosophy CambridgeCambridge University Press 1995

5 D ixon-Woods M Fitzpatrick R Roberts K Including qualitativeresearch in systematic reviews opportunities and problems JEva l Clin Pract 2001 7 125ndash33

6 Noblit G Hare R Meta-Ethnography Synthesising Qua litativeStudies Newbury Park Sage 1988

7 Humphris D Types of evidence In Harmer S Collinson G edsAchieving Evidence-Based Practice a Handbook for PractitionersLondon Bailliere Tindall 1999 13ndash39

8 Pearson A Evidence-based nursing quality through researchIn Nay R Garratt S eds Nursing O lder People Issues andInnovations Sydney Maclennan amp Petty 1999 338ndash52

9 Evans D Pearson A Systematic reviews gatekeepers of nursingknow ledge J Clin Nurs 2001 10 593ndash9

10 Salipante P Notz W Bigelow J A matrix approach to literaturereviews In Staw BM Cumm ings LL eds Research in Organi-zationa l Behavior Greenw ich JAI Press 1982 321ndash48

11 Pearson A Excellence in care Future dimensions for effectivenursing NT Res 1998 3 25ndash7

12 Lemmer B Grellier R Steven J Systematic review of nonran-dom and qualitative research literature exploring and uncov-ering an evidence base for health visiting and decision makingQua litative Hea lth Res 1999 9 315ndash28

13 Popay J Williams G Qualitative research and evidence-basedhealthcare J Roya l Soc Med 1998 91 32ndash7

14 Green J Britten N Qualitative research and evidence basedmedicine BMJ 1998 316 1230ndash32

15 Sandelowski M Docherty S Emden C Qualitative metasynthe-sis issues and techniques Res Nurs Hea lth 1997 20 365ndash71

16 Barroso J Powell-Cope G Metasynthesis of qualitative researchon living w ith HIV infection Qua litative Hea lth Res 2000 10340ndash53

17 Campbell R Pound P Pope C et a l Evaluating meta-ethnogra-phy a synthesis of qualitative research on lay experiences ofdiabetes and diabetes care Soc Sci Med 2003 56 671ndash84

18 Rees R Harden A Shepherd J Brunton G O liver S O akley AYoung People and Physica l Activity A Systematic Review ofResearch on Barriers and Facilitators London EPPI-Centre Uni-versity of London 2001 Available from http eppiioeacuk

19 Jensen L Allen M Meta-synthesis of qualitative findings Qua l-itative Hea lth Res 1996 6 553ndash60

Balancing the evidence 61

20 Popay J Roen K Synthesis of evidence from research usingdiverse study designs a review of selected methodologicalwork Hea lth Care Reports 2003 1 1ndash24

21 Patterson B Thorne S Canam C et a l Meta-Study of Qua litativeHea lth Research Thousand O aks Sage 2001

22 Schreiber R Crooks D Stern P Qualitative meta-analysis InMorse J ed Completing a Qua litative Project Deta ils and D ia-logue Thousand O aks Sage 1997 311ndash27

23 Estabrooks C Field P Morse J Aggregating qualitative findingsan approach to theory development Qua litative Hea lth Res1994 4 503ndash11

24 Spencer L Richie J Lew is J et a l Qua lity in Qua litative Eva luationA Framework for Assessing Research London Government ChiefSocial Researcherrsquos O ffice 2003 Available from http wwwpolicyhubgovuk

25 Emden C Sandelowski M The good the bad and the relativePart two Goodness and the criterion problem in qualitativeresearch Int J Nurs Prac 1999 5 2ndash7

26 Averis A Pearson A Filling the gaps identifying nursingresearch priorities through the analysis of completed systematicreviews JBI Reports 2003 1 49ndash126

27 Denzin N Lincoln Y Handbook of Qua litative Research Thou-sand O aks Sage 1994

28 Popay J Rogers A Williams G Rationale and standards for thesystematic review of qualitative literature in health servicesresearch Qua litative Hea lth Res 1998 8 341ndash51

29 Evans D Pearson A Systematic reviews of qualitative researchClin Effectiveness Nurs 2001 5 1ndash7

Appendix I

Categorisation of methodological frameworks

ActionDescriptionbull Ethnographybull Grounded Theorybull Action Researchbull Case Studiesbull Descriptivebull Programme Evaluation

Subjectivity (structures of consciousness)bull Phenomenology

bull Ethnomethodologybull Hermeneuticbull Phenomenography

Ana lytica lbull ConceptualAnalyticalbull Historicalbull D iscourse analysisbull Biographical textualnarrativebull Culturalmedia analysisbull Deconstructive analysis

Appendix II

Group outcomes statement

1 In light of the emphasis placed on evidence-basedapproaches in contemporary health care practitionersare increasingly required to recognise and assim ilatethe body of research literature relevant to their area ofpractice Up to this point the emphasis has been ondeterm ining effectiveness w ith particular reference toquantitative research This consideration should alsoextend to issues relating to Feasibility AppropriatenessMeaningfulness and Effectiveness this extension ofinquiry requires that qualitative research should also beconsidered Qualitative research yields distinct benefitsthat do not stem from quantitative research this needsto be recognised in the context of a truly systematic andextensive review of the literature

2 We have observed that there is currently a largeamount of qualitative research available that is notbeing systematically utilised to inform practice Untilnow there has not been a process to incorporate thisresearch into the development of clinical guidelines

Consequently a large amount of potentially importantdata has been ignored

3 The challenges that were before the group referred toa range of issues specific to creating a concerted syn-thesis of qualitative research Recent work by Evans andPearson identifies the importance of incorporating bothqualitative and quantitative research in systematicreviews but suggests that there are potentially a rangeof issues that w ill arise from this development29 Thenature of quantitative research makes a synthesis of dataa relatively straightforward task Data in the forms ofmeans and standard deviations can be extracted fromthe research and entered into statistical tools allow ingfor a meta analysis to be conducted which in turn resultsin a clear indication of the effectiveness of the interven-tion This reference back to primary data cannot beconducted for qualitative research as the nature of thedata precludes this Essentially the aim of this processis to provide a system of evaluating the quality of qual-itative research and to synthesise the body of research

62 A Pearson

4 We envisage a tool ndash the Qualitative Assessment andReview Instrument (QARI) The model underlying QARIprovides a systematic review process that m irrors thatundertaken for quantitative research while being sensi-tive to the nature of qualitative data The model recog-nises the value of qualitative research yet provides amechanism to categorise the quality of original studiesand the applicability of the findings to practice A seriesof findings and concom itant narrative descriptions w illbe elicited from individual studies The model views themajority of these research findings as fitting into fouroverarching categories Political Professional Subjectiveand C linical

5 Political issues are those pertaining to the power rela-tionships between people people and ideas people andorganisations and how these relate to society includingsocio-econom ic concerns (Belonging to or taking aside in politics relating to a personrsquos or an organisationrsquosstatus or influence) Professional matters include ethicallegal and regulatory concerns and issues relating to theorganisations monitoring these aspects which are rele-vant to practice

6 Subjective issues pertain to internal states personalexperience opinions values thoughts beliefs and inter-pretations This category is distinct from factual report-ing of past events

7 C linical concerns are related to care and treatment Thismay refer to equipment staffing or issues relating toclinical context

8 It is recognised that a situation may arise where a givenfinding (narrative description) may fall into the domainof more than one category In such cases it is recom-mended that the finding be addressed under each cat-egory deemed to be appropriate These cases need tobe verified by the second reviewer

9 The questions addressed by qualitative research differfrom those addressed by quantitative research differentperspectives are also provided Currently there is nosystematic approach to incorporate these into reviewsof evidence-based practice Qualitative research isimportant because it incorporates a service userrsquos voiceinto the process of formulating evidence-based practiceAt present the only way this userrsquos voice is heard is viainterest groups

10 There are currently no means to systematically reviewqualitative research We believe best practice shouldreflect the whole range of evidence available providedthat this evidence is subjected to appropriate appraisalWe have reached consensus on a protocol for criticallyand systematically appraising a body of qualitativeresearch that leads to summary statements and recom-mendations Computer software w ill be developed tosupport this process

Appendix III

Qualitative findings critical appraisal scale

Criteria Yes No Unclear

1 There is congruity between the stated philosophical perspective and the research methodology

2 There is congruity between the research methodology and the research question or objectives

3 There is congruity between the research methodology and the methods used to collect data

4 There is congruity between the research methodology and the representation and analysis of data

5 There is congruity between the research methodology and the interpretation of results

6 There is a statement locating the researcher culturally or theoretically

7 The influence of the researcher on the research and vice-versa is addressed

8 Participants and their voices are adequately represented

9 The research is ethical according to current criteria or for recent studies there is evidence of ethicalapproval by an appropriate body

10 Conclusions drawn in the research report appear to flow from the analysis or interpretation of thedata

Total

Balancing the evidence 63

Reviewerrsquos comments

ndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndash

ndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndash

Appendix IV

Qualitative data extraction tool

Author ______________________ Record number _______ Journal ______________________ Year ________ Reviewer _____________________ Method Methodology Data analysis Setting amp Context Geographical context Cultural context Participants Number Description Interventions

64 A Pearson

Findings Narrative description Qualitative evidence rating (123)

Authorsrsquo conclusions

Comments

Balancing the evidence 63

Reviewerrsquos comments

ndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndash

ndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndashndash

Appendix IV

Qualitative data extraction tool

Author ______________________ Record number _______ Journal ______________________ Year ________ Reviewer _____________________ Method Methodology Data analysis Setting amp Context Geographical context Cultural context Participants Number Description Interventions

64 A Pearson

Findings Narrative description Qualitative evidence rating (123)

Authorsrsquo conclusions

Comments

64 A Pearson

Findings Narrative description Qualitative evidence rating (123)

Authorsrsquo conclusions

Comments