Pollock et al. Systematic Reviews (2015) 4:55 DOI 10.1186/s13643-015-0023-5

RESEARCH Open Access

User involvement in a Cochrane systematicreview: using structured methods to enhance theclinical relevance, usefulness and usability of asystematic review updateAlex Pollock1*, Pauline Campbell1, Gillian Baer2, Pei Ling Choo3, Jacqui Morris4 and Anne Forster5

Abstract

Background: This paper describes the structured methods used to involve patients, carers and health professionalsin an update of a Cochrane systematic review relating to physiotherapy after stroke and explores the perceivedimpact of involvement.

Methods: We sought funding and ethical approval for our user involvement. We recruited a stakeholder groupcomprising stroke survivors, carers, physiotherapists and educators and held three pre-planned meetings during thecourse of updating a Cochrane systematic review. Within these meetings, we used formal group consensusmethods, based on nominal group techniques, to reach consensus decisions on key issues relating to the structureand methods of the review.

Results: The stakeholder group comprised 13 people, including stroke survivors, carers and physiotherapists with arange of different experience, and either 12 or 13 participated in each meeting. At meeting 1, there was consensusthat methods of categorising interventions that were used in the original Cochrane review were no longerappropriate or clinically relevant (11/13 participants disagreed or strongly disagreed with previous categories) andthat international trials (which had not fitted into the original method of categorisation) ought to be includedwithin the review (12/12 participants agreed or strongly agreed these should be included). At meeting 2, the groupmembers reached consensus over 27 clearly defined treatment components, which were to be used to categoriseinterventions within the review (12/12 agreed or strongly agreed), and at meeting 3, they agreed on the keymessages emerging from the completed review. All participants strongly agreed that the views of the groupimpacted on the review update, that the review benefited from the involvement of the stakeholder group, and thatthey believed other Cochrane reviews would benefit from the involvement of similar stakeholder groups.

Conclusions: We involved a stakeholder group in the update of a Cochrane systematic review, using clearlydescribed structured methods to reach consensus decisions. The involvement of stakeholders impactedsubstantially on the review, with the inclusion of international studies, and changes to classification of treatments,comparisons and subgroup comparisons explored within the meta-analysis. We argue that the structured approachwhich we adopted has implications for other systematic reviews.

Keywords: Cochrane review, User involvement, Stakeholders, Consensus methods, Nominal group technique,Physiotherapy, Stroke

* Correspondence: alex.pollock@gcu.ac.uk1Nursing, Midwifery and Allied Health Professions Research Unit (NMAHP RU),Glasgow Caledonian University, Cowcaddens Road, Glasgow G4 0BA, UKFull list of author information is available at the end of the article

© 2015 Pollock et al.; licensee BioMed Central. This is an Open Access article distributed under the terms of the CreativeCommons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, andreproduction in any medium, provided the original work is properly credited. The Creative Commons Public DomainDedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article,unless otherwise stated.

Pollock et al. Systematic Reviews (2015) 4:55 Page 2 of 11

BackgroundThe act of professionals and consumersa working togetherto produce and share knowledge has been an explicit prin-cipal of the Cochrane Collaboration since it first began in1993 [1,2]. A variety of rationales have been proposed tosupport this active involvement of people with a healthcondition in systematic reviews, but the principal argu-ments are that this involvement is beneficial to the quality,relevance and impact of health research [3,4]. The activeinvolvement of people with a health condition has beenproposed as a way to enhance the perceived usefulness ofsystematic review evidence, addressing barriers to the up-take of synthesised research evidence [5]. However, despitewidespread acceptance of these arguments, which aredriving national strategies to ensure involvement of peopleaffected by health conditions in all research activities [6],there remains a lack of high-quality evidence demonstrat-ing the impact of involvement on research activity and up-take of evidence [3]. Arguably, this lack of evidence hascontributed to the considerable inconsistencies in theextent of user involvement within Cochrane reviews,despite over 20 years of consumer involvement withinthe Cochrane Collaboration [7].Where there has been user involvementa within indi-

vidual Cochrane reviews, the approaches to involvementhave also varied considerably [7]. A review of user in-volvement in systematic reviews confirmed that a widerange of different approaches to involvement has beenimplemented [8]. This review found that the most com-monly used approaches are consultation with a group ofpeople at a one-off workshop or at key stages in the re-view process, or involvement of individual people asmembers of a review team, although other approachessuch as email consultation and using a Delphi processhave also been used [8].Challenges to the development of effective methods of

involvement within individual reviews are compoundedby poor description of involvement within many reviews[8] and by limited evaluation of the impact of involvement[3]. However, based on an exploration of case examples,Boote proposes a number of strategies to facilitate effect-ive involvement within systematic reviews [8]. These in-clude budgeting for the costs of involvement; having areview team member with lead responsibility for involve-ment; providing training, briefing notes and backgroundinformation to people involved; and the use of structuredmethods of involvement (such as the nominal group tech-nique or Delphi process) at key stages of the reviewprocess.In order to enhance the clinical relevance of a Cochrane

systematic review relating to physiotherapy after stroke,we aimed to use structured methods, based on Boote’ssuggested strategies [8], to involve patients, carers andhealth professionals in an update of this review.

Cochrane systematic reviewStroke is a leading cause of death and disability and iswithin the top ten causes of long-term physical disabilityin many Western nations [9-11]. The most common andwidely recognised impairment caused by stroke is motorimpairment, with paralysis of some parts of the body,difficulties with various physical functions and limita-tions in mobility [12]. Consequently, the ultimate goal ofphysical rehabilitation is most commonly the improve-ment of function and mobility [13]. Over the years, vari-ous approaches to physical rehabilitation have beendeveloped, according to different ideas about how peoplerecover after a stroke. Considerable debate continuesamong physiotherapistsb about the relative benefits ofdifferent approaches [14,15].Our Cochrane systematic review explored whether

physical rehabilitation approaches are effective in recoveryof function and mobility in people with stroke and if anyone physical rehabilitation approach was more effectivethan any other approach. It was first published in 2003[16] and updated in 2007 [17]. However, within the 2007version, a number of limitations were identified, and thereview authors recommended that these should be ad-dressed within any subsequent updates [17]. These limita-tions related to the methods of defining and categorisingphysical rehabilitation approaches within the review,which were based on historical development of ap-proaches within Western nations. The first concern wasthat the methods used within the review to define and cat-egorise interventions were no longer relevant to currentclinical practice. The second concern was whether phys-ical rehabilitation interventions described in foreign-language publications (many of which were listed as‘awaiting assessment’ in the 2007 version) were relevantfor inclusion and - if they were - how these internationalapproaches could best be categorised within the review.Prior to the next update [18], we therefore planned to

amend the structure and format (without expanding thescope) of this review in order to address these limita-tions and produce a review which had international clin-ical relevance. We planned to explore the approachesdescribed in the foreign-language publications and clarifywhether it was clinically relevant and useful to synthesiseevidence relating to these international approaches withinthis review. While it was clear from the 2007 version thatamendments were required to address limitations withinthe review, the nature of these amendments was not clear.There were therefore a number of important decisions tobe made which could substantially affect the structure,scope and methods of the review. These decisions wouldsignificantly impact on how the evidence was synthesisedwithin the updated review, with potential solutions varyingfrom amending the classification system with the existingreview to having two separate reviews. Key decisions and

Pollock et al. Systematic Reviews (2015) 4:55 Page 3 of 11

potential solutions are illustrated in Figure 1. We soughtto address these issues and make decisions through in-volvement of a group of stroke survivors, carers and phys-iotherapists and educators (‘stakeholder group’) in thereview update.This paper describes the structured methods of involve-

ment of this stakeholder group in the updated Cochranereview and explores the perceived impact of involvement.The methods and results of the review have been fullypresented within the Cochrane Library [18].

MethodsFigure 2 provides an overview of the key stages of thestructured methods of involvement in relation to the up-dating of the Cochrane review.

Resourcing the projectWe sought funding to support the stakeholder group in-volvement in this project. We costed for the direct ex-penses associated with involvement (that is, travel,subsistence), but not for any funding to pay for the timeof group members. This decision was based on our priorexperience of working with stroke survivors and carers,where we found that people were generally happy to be

Figure 1 Possible amendments to Cochrane review identified within

involved in stroke research without compensation fortheir time. Furthermore, we considered that the involve-ment of physiotherapists and educators ought to providethese individuals with an opportunity for professionaldevelopment and be carried out during their employedworking hours (through negotiation with relevant man-agers). We costed for a part-time researcher (for12 months) who would play the lead role in manage-ment of the stakeholder group, as well as contribute tothe tasks involved in updating the review (includingrunning searches, identifying and appraising trials,extracting and entering data).

Ethical approval and consentThere has been some debate relating to the necessity togain ethical approval for the involvement of people withinthe process of healthcare research (that is, as opposed toparticipants in research) [8,19]. However, as we planned toaudio-record group meetings and use this qualitative dataas a record of group discussion and decisions, we did seekethical approval for the stakeholder group involvementwithin this project (from Glasgow Caledonian UniversitySchool of Health and Life Sciences ethics committee). We

project plan.

Figure 2 Overview of structured methods of involvement during review update.

Pollock et al. Systematic Reviews (2015) 4:55 Page 4 of 11

gained signed informed consent from all group partici-pants, specifically relating to the collection of audio data.

Identifying and recruiting stakeholder group membersWe planned to recruit a stakeholder group comprisingstroke survivors, carers, physiotherapists and educators.As this review was focussed on interventions that aregenerally exclusively delivered by physiotherapists, wedecided that physiotherapists should be the only healthprofessionals involved. We planned to purposefully se-lect physiotherapists to ensure a variety of grades, yearsof experience, post-graduate courses and geographicalwork base. We planned to recruit between 8 and 12group members as previous studies using the nominalgroup technique have recommended between 5 and 9participants [20], and we predicted that some groupmembers may not attend all meetings so we wished toover-recruit.We formed a two-page role description [see Additional

file 1], which detailed the aim and purpose of the stake-holder group, what was required in terms of skills, ex-perience and knowledge and the time commitment

(including the dates of all meetings), what the potentialbenefits of involvement could be and confirmation ofre-imbursement of any travel expenses. Separate ver-sions were written for stroke survivors/carers and phys-iotherapists/educators. We circulated this description,with a request for interested people to contact us via emailusing established professional, charity and patient-supportnetworks across Scotland. We limited recruitment topeople living in Scotland due to the requirement to attendmeetings within Central Scotland and our limited avail-able travel budget. Physiotherapists who responded wereasked to complete a simple questionnaire to provide de-tails of their current position, clinical experience and rele-vant post-graduate training. This information was used toselect physiotherapists for inclusion within the group, ac-cording to our sampling strategy, in order to give repre-sentation of the criteria listed above.

Stakeholder group meetingsWe pre-planned three stakeholder group meetings dur-ing the course of the 12-month project. The content,structure and format of these three meetings, and details

Pollock et al. Systematic Reviews (2015) 4:55 Page 5 of 11

of the statements which were discussed and voted on,are illustrated in a table [see Additional file 2]. Meetings1 and 2 were held at the start of the project, prior to thereview update being carried out. The objectives of meeting1 were to explore and reach consensus on how physicalrehabilitation approaches should be categorised, includingdecisions over whether international approaches shouldbe included, and - if so - how these should be categorisedwithin the review. Discussions were focused around pre-determined statements on which consensus decisionswere desired. The objectives for meeting 2 were to exploreand agree on specific strategies to update and amend thecurrent Cochrane systematic review, based on the deci-sions made during meeting 1. Meeting 3 was held at theend of the project when the results of the review updatewere complete, and the objective of this meeting was toexplore the perceived clinical implications of the findings.During meeting 1, the stakeholder group also brieflydebated how the results of the review should be dis-seminated to NHS Scotland practitioners and howevidence-based practice could be encouraged, and anaction plan for dissemination activities was drafted.This dissemination plan was revisited in meetings 2and 3, and plans for local dissemination across NHSScotland discussed and agreed.Meeting ‘ground rules’ were discussed and agreed by all

group members at the start of each meeting. These in-cluded turning off mobile phones and electronic devices,raising a hand when wanting to speak, no person speakingfor more than 2 min at any one time, respecting each

Introduction of top

• Members introduced• Purpose & procedure of• Meeting rules agreed

Voting on st

• Silence / no discussion • Voting slips completed canonymously

• Time for quiet reflection

Vote counting

• Voting slips gathered in and counted in front of the group

• Agreement (or disagreement) noted• Where consensus - move to next topic• If disagreement - progress to generation of new statement

Figure 3 Illustration of key stages of nominal group technique.

other’s opinions and not talking over other people. Eachmeeting had a designated facilitator and timekeeper. Wesent relevant background papers prior to each meeting,and researchers provided brief presentations of key mater-ial at the start of each meeting.

Consensus methodsWe used formal group consensus methods to reach con-sensus decisions within the stakeholder group meetings,as such methods are recognised to be advantageouswhen subjective judgements need to be organised [21].These consensus methods were based on nominal grouptechniques (see Figure 3), as this method enables thepooling of decisions and judgements from a group of in-formed experts, leading to votes on a range of optionsuntil ultimately group consensus is reached [20,22]. Thestructure of each meeting involved a set time period fordiscussion around a statement or issue, followed by vot-ing. The statements for meeting 1 were pre-determinedby the research team, while statements for subsequentmeetings evolved and were agreed through group discus-sion. No group discussion occurred during the voting,which was completed anonymously on a paper slip. Forthe statement which was discussed, each individual mem-ber ranked their agreement with the statement on a scaleof 1 (strongly agree) to 5 (strongly disagree) and providedwritten comments on their reason if they wished [seeAdditional file 3 for a sample voting slip]. Immediatelyafter voting, the paper slips were gathered in, counted andpresented to the group. If there was a lack of consensus in

ic / statement

meeting explained

Group members discuss topic / statement

• Ideas discussed by all members• Facilitator ensures everyone gets fair chance to speak

• No judgements passed• Discussion ceases after fixed time period

atement

during votingonfidentially &

and consideration

Pollock et al. Systematic Reviews (2015) 4:55 Page 6 of 11

the responses, we planned further discussion around asecond statement (the wording of which would have beenagreed by the group) and subsequent rounds of voting asrequired; however, no second rounds of discussion or vot-ing were required during either of the meetings. Writtencomments were tabulated, with reference to the scoremaintained.Some of the review authors attended the stakeholder

group meetings and contributed to discussions, as it wasfelt their attendance was important in order to answerany questions that the stakeholder group may have relat-ing to the existing review and Cochrane review methods.However, review authors did not participate in the vot-ing process.Analysis involved determining the proportion of re-

spondents agreeing with each statement within eachround of voting. In addition, consensus decision meet-ings were audio-recorded and transcribed verbatim tofacilitate further analysis of qualitative data relating tothe themes discussed around specific statements.

Additional contact with stakeholder group membersIn addition to the three planned meetings, contact wasmade with stakeholder group members by email as re-quired throughout the project. Researchers contactedmembers to seek feedback or comments on specific issuesrelevant to the stage of the review update, and a variety ofmethods were used to gain feedback. Additional file 4 pro-vides an overview of the range of methods used to gainfeedback and involvement. These methods included theuse of a number of feedback forms and the use of votingaround specific issues or decisions (using Doodle polls).

Evaluation of stakeholder groupAt the end of meeting 3, we circulated a brief evaluationform requesting written feedback on the process of in-volvement and the perceived impact of involvement. Wealso had an informal group discussion around the per-ceived impact of involvement, which was audio-recordedand transcribed.

Table 1 Results of voting at stakeholder group meeting 1

Totalnumber ofresponses

Statement A. The current categories are appropriate andclinically relevant

n = 13a

%100

Statement B. These international trials should be includedin our review of physiotherapy treatment approaches

n = 12a

%100

Statement C. The interventions studied in these trials aresimilar to one another

n = 12a

%100aThirteen participants attended the first half of meeting 1 and voted on statementparticipants voted on statements B and C.

ResultsWe gained funding to support this project from theScottish Government’s Health Directorate, Chief ScientistOffice (ref: CZG/2/544). The project was approved by theGlasgow Caledonian University School of Health and LifeSciences ethics committee (Reference: HLS12/40).Thirteen people were recruited onto the stakeholder

group. This comprised three stroke survivors, one carerand nine physiotherapists. Only four stroke survivors/carers volunteered to participate; all four joined thegroup. The nine physiotherapists represented a widerange of clinical experience, geographical locations andpost-graduate experience. An additional four physio-therapy educators/researchers, who were all authors onthe Cochrane review, attended and contributed to thestakeholder group meetings in order to clarify issues(when required) during discussion but were excludedfrom the voting process.

Meeting 1The results of the voting in meeting 1 are presented inTable 1. There was generally a consensus that the cat-egories which had been used in the 2007 version of thereview were no longer appropriate or clinically relevant(statement A). There was unanimous agreement that theinternational trials (which had not fitted into the previ-ous method of categorisation) ought to be includedwithin the review (statement B). Discussion highlightedthe need to ensure that the review did not exclude evi-dence based on geographical boundaries or norms ofpractice. Participants recognised challenges relating toclear descriptions of the physical rehabilitation ap-proaches and achieving clinically relevant and usefulcategorisations of these. There was less clear agreementregarding the similarity of content of the physical re-habilitation approaches investigated by the internationaltrials which had been presented (statement C), but dis-cussion and comments clearly highlighted a commonperception of some similarities between the interven-tions described in the trials.

Stronglyagree

Agree Neither agreeor disagree

Disagree Stronglydisagree

1 2 3 4 5

0 1 1 9 2

8 8 69 15

6 6 0 0 0

50 50

2 7 3 0 0

17 58 25

A. One participant then had to leave the meeting, and the remaining 12

Pollock et al. Systematic Reviews (2015) 4:55 Page 7 of 11

Discussion at the end of meeting 1 confirmed the po-tential benefits of defining ‘treatment components’, andat the end of this meeting, the group members asked theresearchers to complete additional tasks. This involvedthe systematic exploration of the treatment componentsreported within the intervention descriptions of all thetrials that had been included within the 2007 version ofthe review. This led to the production of materials whichwere sent out to meeting participants prior to meeting 2[see Additional file 2 for more details].In addition, although the stakeholder group members

were not specifically asked to debate this, the importanceof exploring particular subgroups came up repeatedlywithin the discussion. For example, discussion emphasisedthe importance of the amount (or ‘dose’) of interventionprovided and the stage of the rehabilitation process of par-ticipants. These discussions enabled the researchers toidentify subgroups which were perceived to be importantby the group and to subsequently amend the protocol toreflect these agreed subgroups.

Meeting 2Prior to meeting 2, the stakeholder group members weresent descriptions of the treatment components from the30 trials, which were either included or awaiting assess-ment in the 2007 version of the review. Group memberswere asked to consider and send feedback relating tohow the interventions within individual studies could becategorised and how the treatment components de-scribed as part of these interventions might be grouped.We received feedback from ten individual people, whichwas collated, and presented (anonymously) to the wholegroup at the start of meeting 2. This presentation clearlyhighlighted where there had been agreement between re-sponses and where there had been disagreement. Follow-ing these presentations, the group members were askedto decide how to progress toward reaching agreementover the categorisation and definition of treatment com-ponents within the interventions.Group members decided to work together to produce

agreed descriptions and definitions of the individual treat-ment components which make up different approaches tophysical rehabilitation and to reach agreement on how

Table 2 Results of voting at stakeholder group meeting 2

Tonures

Statement A. The new categories are appropriate and clinically relevant n =

%1

Statement B. The stated names are appropriate and clinically relevant n =

%1aTwelve participants attended and voted during meeting 2.

to categorise these individual treatment components.Through a process of group discussion, 27 individualtreatment components were defined and grouped under7 categories [see Additional file 5]. The wording of eachof these definitions was agreed through group discus-sion, with the researcher typing these into a projectedspreadsheet during the discussion. Voting was carriedout after all the definitions and categories had been ver-bally agreed, to ensure that there was consensus overthese. This voting demonstrated unanimous agreementwith the new categories, and the names given to thesecategories (see Table 2).

Meeting 3At meeting 3, the results of the review update were pre-sented by the researchers, with the data grouped accord-ing to the treatment components and categories agreedduring meeting 2. After the presentation of each meta-analysis within the updated review, the group membersdebated the clinical implications of the results. At theend of the meeting, the group revisited these clinical im-plications and discussed and reached agreement on thekey messages arising from the review and the implica-tions for physiotherapists, stroke survivors and carers.These agreed key messages were later incorporated intothe final version of the written Cochrane review, withacknowledgement that group members had agreed onthese during the stakeholder group meeting.

Additional contact with stakeholder group membersFollowing meeting 3, contact was maintained with thegroup members via email. In particular, contact was madein relation to:

� Document summarising Cochrane review findings.A draft document was circulated to group members,and feedback was provided by email.

� Decision to change title of Cochrane review. Duringthe Cochrane peer review process, a suggestion wasmade that the title ought to be changed (from‘physical treatment approaches’ to ‘physicalrehabilitation approaches’ or to ‘physical therapyapproaches’). A Doodle poll was used, enabling all

talmber ofponses

Stronglyagree

Agree Neither agreeor disagree

Disagree Stronglydisagree

1 2 3 4 5

12a 2 10 0 0 0

00 17 83

12a 2 10 0 0 0

00 17 83

Pollock et al. Systematic Reviews (2015) 4:55 Page 8 of 11

group members to ‘vote’ on the desired title,resulting in clear agreement around a preferred title.

� Collaboration on delivery of a Focussed Symposiumat the Annual United Kingdom and IrelandCochrane Symposium. All group members wereasked if they would like to collaborate on a proposalto run a symposium session. Four membersvolunteered for this, resulting in a successfulworkshop proposal, which was later delivered bytwo researchers and four stakeholder groupmembers at the Cochrane Symposium [23].

EvaluationNine group members completed an evaluation form atthe end of meeting 3, with all respondents stronglyagreeing that the views of the group impacted on the re-view update, that the review benefitted from the involve-ment of the stakeholder group and that they believedother Cochrane reviews would benefit from the involve-ment of similar stakeholder groups. The full results fromthe written evaluation are provided in Additional file 6.A small number of group members highlighted specificdifficulties relating to involvement, including issuesaround the timing and travel to meetings, and the repre-sentativeness of the group members; however, the major-ity of the feedback was positive and particularlyemphasised that group members felt their opinions werevalued [‘I feel that the opinions of the stakeholder groupwere greatly valued’] and that the process of involvementwas acceptable [‘Really well organised, structured and pro-ductive’] and beneficial to the Cochrane review [‘OtherCochrane groups please copy’]. Furthermore, group dis-cussion clearly emphasised that the level of involvementwas perceived to be unique and was highly appreciated:

‘…I have taken part in quite a number of things ofthis nature over the past 20 years and this is the firsttime that I have really felt that it has been successfuland that I have been listened to..’‘To have been listened to as clinicians, um, with ouraspiration of trying to deliver high quality evidenced-based practice, where there are real challenges doingthat, but to really be listened to in that respect andthen to translate ……. is pretty unique’

DiscussionWe used clearly described structured methods to involvea stakeholder group in the update of a Cochrane system-atic review and collected data which objectively supportdecision-making within the review process. The involve-ment of stakeholders impacted substantially on the re-view, with changes to the inclusion of studies exploringnon-Western approaches to physiotherapy, classificationof treatments, comparisons and subgroup comparisons

explored within meta-analysis and dissemination of keymessages. The group members made important deci-sions around the classifications of interventions withinthe review, leading to the development of a new methodof intervention categorisation and enabling synthesis andanalysis of evidence which was perceived to be clinicallyrelevant. The stakeholder group members perceived thattheir involvement was valuable, and valued, and that thereview benefited from their involvement.We believe that this approach to user involvement has

implications for other systematic reviews, and this issupported by the members of the stakeholder group. Wehave demonstrated that, with appropriate planning andresources, user involvement can be integrated into theprocess of updating a complex review and timely com-pletion of the review achieved. The use of structuredmethods, based on the nominal group technique, en-abled consensus decisions to be reached in a transparentmanner with equal value placed on the opinion of eachmember of the stakeholder group.The review that we updated was, arguably, unique in

that the previous version had highlighted a number ofmajor limitations which required to be addressed priorto the subsequent update. This meant that there wereclear questions for which we were seeking consensus de-cisions. This, clearly, will not be the case for all reviewupdates. However, during the development of new pro-tocols for new systematic reviews, there are always anumber of methodological decisions which have to bemade. These decisions include, but are not limited to,the selection criteria for the review (participants, inter-ventions, comparisons and outcomes), planned reviewcomparisons, analyses and subgroup analyses. We be-lieve that user involvement - using structured methodssimilar to those which we describe - introduced at thestage of protocol development, has the potential to en-hance the clinical relevance, usefulness and usability ofsystematic reviews.Our review was focussed on highly complex interven-

tions, which comprised a range of treatment compo-nents, which could be delivered in different ways, bydifferent professionals in different settings to partici-pants with a wide range of varied impairments and dis-abilities. These levels of complexity clearly add to thechallenges of definition and focus within the review, ar-guably adding value to the role of stakeholders whobring a range of experiences and opinions. However,even the ‘simplest’ of interventions (for example, a singledose of a drug) will be delivered to participants who willvary, for example, in relation to their disease, medicalhistory, their past experiences and values placed on dif-ferent outcomes. Consequently, while not exploredwithin this study, we do believe that user involvementwithin a systematic review has the potential to enhance

Pollock et al. Systematic Reviews (2015) 4:55 Page 9 of 11

the clinical relevance, usefulness and usability even whenfocused on the most ‘simple’ of interventions.We designed our user involvement based on strategies

proposed by Boote [8] to facilitate effective involvementwithin systematic reviews. We believe that the pre-planning of the project, based on these strategies was akey contributor toward the success of our user involve-ment. Additional specific techniques that we believe con-tributed toward successful user involvement included:

� fixing all meeting dates prior to identification andrecruitment of stakeholder group members. Thisensured that potential recruits were all available forall meetings (with the exception of unforeseeableevents such as illness of oneself or a family member)

� agreeing on ‘meeting rules’ which included strategiesaimed at ensuring no one person dominated thediscussion and that all group members couldparticipate equally (described in the ‘Methods’section).

Although we pre-planned the user involvementwithin this project, we did not have any user involve-ment at the stage of project planning or involvementwith funding applications. We acknowledge that this isa limitation of our study and that user involvement atthe planning stage may have been beneficial (for ex-ample, in relation to decisions about who and how torecruit to the stakeholder group). However, there aremany additional challenges associated with user in-volvement at the project planning stage, often relatedto the absence of dedicated time and resources at thisearly stage of project development.We acknowledge that a key limitation of our user in-

volvement was the composition of our stakeholder group,which comprised four stroke survivors/carers in compari-son to nine physiotherapists. While we made attempts toengage greater numbers of stroke survivors/carers, cascad-ing information about the project through a variety ofroutes, time constraints of the project limited us to elec-tronic dissemination, which clearly restricted the popula-tion of people who had the opportunity to volunteer forinvolvement. We only received responses from four strokesurvivors/carers, all of whom joined the group. Ideally, wewould have liked to have at least as many, if not more,stroke survivors/carers as health professionals. Whilethere was an imbalance between stroke survivors/carersand health professionals, it is important to note that thephysiotherapists involved were ‘users’ of the evidence be-ing synthesised. Although we have no evidence of this, itis possible that our decision not to seek funds to enable usto offer financial support for involvement may have been abarrier to involvement, and we recognise that there is aneed for more detailed exploration of the potential

barriers and facilitators to involvement. Despite somegroup members highlighting a sense of dissatisfaction re-lating to the numbers of stroke survivors/carers, all groupmembers did report that they felt they were listened toand their opinions valued. For future projects in whichstakeholder groups are being set up, we recommend thatadditional time and resources are allocated to the set-upstage to enable greater and more equitable representationof patients and carers.No members of the stakeholder group became authors

on our review, although we did acknowledge the contri-butions of the individual members within the publishedreview. However, we acknowledge that it is likely to beappropriate within some other reviews to invite mem-bers of user groups to contribute as named authors to areview. This would require careful planning to ensurethat all review authors had appropriate training and re-sources to support meaningful contribution and that theroles and responsibilities of individual review authorswere clearly defined and agreed at the start of the reviewprocess.Throughout this process, we made it clear to the

stakeholder group members that the decisions takenwould be the decisions of the group members, and notof researchers or review authors. We emphasised thereal sense of control that this gave them over the outputof the final review. As researchers, we took direction dir-ectly from the decisions made by the group members,effectively handing over control to the group in relationto some highly important methodological decisions inrelation to the review. At all stages, we strove to betransparent in the reporting of decisions and have en-sured that we acknowledged and reported the role of thestakeholder group within the published Cochrane re-view. Our experience is that the act of effectively ‘hand-ing over control’ felt somewhat ‘alien’ to the researchers,who are generally in the position of having the final sayover methodological decisions relating to their research.However, the review authors also found that there wasconsiderable value associated with being able to report‘implications for practice’ which were reflecting the con-sensus viewpoint of the stakeholder group, rather thanthe views of individual researchers or review authorsonly. This was perceived to add validity to the key mes-sages within the review, consequently contributing tothe value placed on the review by patients, carers andhealth professionals.The evaluation and feedback from our stakeholder

group was highly positive and highlighted that the groupperceived their input to have been valued and have abeneficial impact on the final review. However, the par-ticipants’ viewpoints are at high risk of bias, and it is akey limitation of our study that we did not have any in-dependent evaluation of the impact of user involvement.

Pollock et al. Systematic Reviews (2015) 4:55 Page 10 of 11

ConclusionsWe have demonstrated that using clear pre-plannedstructured methods can facilitate effective user involve-ment within a major update of a complex systematic re-view. We found that the strategies proposed by Bootewere effective, and we identified a number of specifictechniques which we believe contributed to successfuluser involvement [8]. The user involvement led to keydecisions relating to the structure and methods of thereview, resulting in a review which was perceived to haveenhanced relevance, validity and accessibility. We arguethat the structured approach which we adopted has im-plications for other systematic reviews.

EndnotesaThere is no agreed definition of the term ‘consumer’

and lack of agreement that this is the best term to use[2,24]. For the purposes of this paper, we define ‘con-sumer’ as a person with a healthcare condition, or friend,carer (unpaid) or family member of that person. A dis-cussion of the varied terminology (for example, con-sumer, public, user) is beyond the scope of this article,and we do not distinguish between these terms. How-ever, within this paper, we have chosen to primarily usethe term ‘user involvement’, as we felt that this clearly in-cludes all ‘users’ of health information, including con-sumers, members of the public and health professionals.

bProfessionals who deliver physical rehabilitation, alsoknown as physical therapists or rehabilitation therapists.

Additional files

Additional file 1: Role description sent to physiotherapystakeholder group members. This is the information sheet which wascirculated in order to recruit physiotherapists onto the stakeholder group.

Additional file 2: Table summarising content and format ofstakeholder group meetings. This table provides details of each of thethree stakeholder group meetings, including the statements which werediscussed and voted on.

Additional file 3: Sample voting slip. This is a copy of one of the votingslips used during stakeholder group meeting 1. The slips were printed onA5 paper. Participants were instructed to circle the appropriate number toshow their agreement with the statement and write the reason for theirselection in the comments box.

Additional file 4: Overview of methods used to gain feedback andinvolvement. A number of different methods were used to communicatewith, and gain feedback and involvement of, the stakeholder groupmembers. This table describes the range of methods used, how these wereused and the resulting involvement of the stakeholder group members.

Additional file 5: Table of individual treatment components,definitions and categories, as defined by the stakeholder group.This table lists the categories and treatment components as defined bythe stakeholder group. These categories and treatment componentswere used to categorise the interventions of all the trials included withinthe Cochrane review.

Additional file 6: Results of written evaluation collected after thethird stakeholder group meeting. Written evaluation was collectedfrom nine participants after the third stakeholder group meeting.

Participants ranked their agreement with statements 1 to 8 on a five-pointscale from strongly agree to strongly disagree. The first table illustrates theagreement assigned by the nine participants. The feedback form alsocontained four open-questions, and participants provided written commentsin response to these. The second table lists all the written responses(unedited and anonymous) from the nine participants to these fouropen-questions.

Competing interestsThe authors declare that they have no competing interests.

Authors’ contributionsAP, JM and AF conceived the study and participated in its design andcoordination. PC contributed to the design, coordination and analysis andhelped draft the manuscript. GB and PLC contributed to the design. APdrafted the manuscript, and all authors read and approved the finalmanuscript.

AcknowledgementsWe thank all members of the stakeholder group, who played an essentialrole in ensuring the success of this project and whose enthusiasm andsupport throughout was invaluable. In particular, we thank Caroline Arshad,Audrey Campbell, Heather Goodare and Mark Smith, who described theirrole in the project at a workshop at the 2014 UK Cochrane Symposium andcontributed to discussions which impacted on this paper. This project wasfunded by the Scottish Government’s Health Directorate, Chief ScientistOffice (ref: CZG/2/544).

Author details1Nursing, Midwifery and Allied Health Professions Research Unit (NMAHP RU),Glasgow Caledonian University, Cowcaddens Road, Glasgow G4 0BA, UK.2Department of Physiotherapy, Queen Margaret University, Queen MargaretDrive, Edinburgh EH21 6UU, UK. 3School of Health and Life Sciences,Glasgow Caledonian University, Cowcaddens Roads, Glasgow G4 0BA, UK.4School of Nursing and Midwifery, University of Dundee, 11 Airlie Place,Dundee DD1 4HJ, UK. 5Academic Unit of Elderly Care and Rehabilitation,Bradford Institute for Health Research, Bradford Teaching Hospitals NHSFoundation Trust/University of Leeds, Duckworth Lane, Bradford BD9 6RJ, UK.

Received: 5 November 2014 Accepted: 3 March 2015

References1. Gartlehner G, Flamm M. Is the Cochrane Collaboration prepared for the era

of patient-centred outcomes research? Cochrane Database Syst Rev.2013;3:ED000054.

2. Bastian HT. The power of sharing knowledge: consumer participation in theCochrane Collaboration. 1994. In http://consumers.cochrane.org/sites/consumers.cochrane.org/files/BastianPowerofSharingKnowledge_1994.pdf;1994. Accessed 02/07/14.

3. Kreis J, Puhan MA, Schunemann HJ, Dickersin K. Consumer involvement insystematic reviews of comparative effectiveness research. Health Expect.2013;16:323–37.

4. INVOLVE. Public involvement in systematic reviews: supplement to thebriefing notes for researchers. Eastleigh: INVOLVE; 2012.

5. Wallace J, Nwosu B, Clarke M. Barriers to the uptake of evidence fromsystematic reviews and meta-analyses: a systematic review of decisionmakers’ perceptions. BMJ Open. 2012;2:e001220.

6. Serrano-Aguilar P, Trujillo-Martin MM, Ramos-Goni JM, Mahtani-Chugani V,Perestelo-Perez L, Posada-de La Paz M. Patient involvement in healthresearch: a contribution to a systematic review on the effectiveness oftreatments for degenerative ataxias. Soc Sci Med. 2009;69:920–5.

7. Horey D. Consumer involvement in the Cochrane Collaboration:background paper. In: Cochrane Collaboration steering group meeting.Auckland; 2010.

8. Boote J, Baird W, Sutton A. Public involvement in the systematic reviewprocess in health and social care: a narrative review of case examples.Health Policy. 2011;102:105–16.

Pollock et al. Systematic Reviews (2015) 4:55 Page 11 of 11

9. Ovbiagele B, Nguyen-Huynh MN. Stroke epidemiology: advancing ourunderstanding of disease mechanism and therapy. Neurotherapeutics.2011;8:319–29.

10. Fisher A, Martin J, Srikusalanukul W, Davis M. Trends in stroke survivalincidence rates in older Australians in the new millennium and forecastsinto the future. J Stroke Cerebrovasc Dis. 2014;23:759–70.

11. Mathers CD, Loncar D. Projections of global mortality and burden of diseasefrom 2002 to 2030. PLoS Med. 2006;3:e442.

12. Wade DT. Measurement in neurological rehabilitation. 1st ed. Oxford:Oxford University Press; 1992.

13. Langhorne P, Coupar F, Pollock A. Motor recovery after stroke: a systematicreview. Lancet Neurol. 2009;8:741–54.

14. Kollen BJ, Lennon S, Lyons B, Wheatley-Smith L, Scheper M, Buurke JH, et al.The effectiveness of the Bobath concept in stroke rehabilitation: what is theevidence? Stroke. 2009;40:e89–97.

15. Carlisle D. Bobath under fire. vol. 16 (1). http://www.csp.org.uk/frontline/article/bobath-under-fire 2010. Accessed 16/08/14.

16. Pollock A, Baer G, Pomeroy V, Langhorne P. Physiotherapy treatmentapproaches for the recovery of postural control and lower limb functionfollowing stroke. Cochrane Database Syst Rev. 2003;2:CD001920.

17. Pollock A, Baer G, Pomeroy V, Langhorne P. Physiotherapy treatmentapproaches for the recovery of postural control and lower limb functionfollowing stroke. Cochrane Database Syst Rev. 2007;1:CD001920.

18. Pollock A, Baer G, Campbell P, Choo PL, Forster A, Morris J, et al. Physicalrehabilitation approaches for the recovery of function and mobilityfollowing stroke. Cochrane Database Syst Rev. 2014;4:CD001920.

19. Smith E, Donovan S, Beresford P, Manthorpe J, Brearley S, Sitzia J, et al.Getting ready for user involvement in a systematic review. Health Expect.2009;12:197–208.

20. Stapleton T, Connelly D. Occupational therapy practice in predrivingassessment post stroke in the Irish context: findings from a nominal grouptechnique meeting. Top Stroke Rehabil. 2010;17:58–68.

21. Nair R, Aggarwal R, Khanna D. Methods of formal consensus inclassification/diagnostic criteria and guideline development. Semin ArthritisRheum. 2011;41:95–105.

22. Sinha I, Jones L, Smyth RL, Williamson PR. A systematic review of studiesthat aim to determine which outcomes to measure in clinical trials inchildren. PLoS Med. 2008;5:e96.

23. Pollock A, Campbell P, Arshad C, Campbell A, Goodare H, Smith M. MakingCochrane evidence useful: giving patients, carers and clinicians a real say ina Cochrane review (workshop). Cochrane UK and Ireland AnnualSymposium. Manchester, UK; 2014. http://manchester2014.cochrane.org/node/6840. Accessed 12/03/15.

24. Boote J, Telford R, Cooper C. Consumer involvement in health research: areview and research agenda. Health Policy. 2002;61:213–36.

Submit your next manuscript to BioMed Centraland take full advantage of:

• Convenient online submission

• Thorough peer review

• No space constraints or color figure charges

• Immediate publication on acceptance

• Inclusion in PubMed, CAS, Scopus and Google Scholar

• Research which is freely available for redistribution

Submit your manuscript at www.biomedcentral.com/submit