SAMPLE SIZE FOR PILOT STUDY

7/18/2019 SAMPLE SIZE FOR PILOT STUDY

http://slidepdf.com/reader/full/sample-size-for-pilot-study 1/4

Edwin R. van Teijlingen and Vanora Hundley

Dr Edwin van Teijlingen is amedical sociologist and Senior

Lecturer in Public Health at theDepartment of Public Health,University of Aberdeen. His re-search interests include the or-ganisation of maternity care athome and abroad, substancemisuse, and psychosocial as-pects of genetics.

Dr Vanora Hundley is a midwifeand Lecturer at the Centre forAdvanced Studies in Nursing,University of Aberdeen. Her re-search interests include mid-wifery care, consumer satisfac-tion and preferences, and re-search utilisation.

• The term ‘pilot studies’ refers to mini versions of a full-scale study (also called‘feasibility’ studies), as well as the spe-cific pre-testing of a particular researchinstrument such as a questionnaire or in-terview schedule.

University of Surrey Sociology at Surrey

socialresearchUPDATE

ISSUE

35

The importance of pilot studies

The term pilot study is used in two different ways in social science research. It can refer to so-called feasibility studies which are“small scale version[s], or trial run[s], donein preparation for the major study” (Polit et al., 2001: 467). However, a pilot study canalso be the pre-testing or ‘trying out’ of aparticular research instrument (Baker 1994:182-3). One of the advantages of conduct-ing a pilot study is that it might give advance warning about where the main researchproject could fail, where research protocolsmay not be followed, or whether proposedmethods or instruments are inappropriate or too complicated. In the words of De Vaus(1993: 54) “Do not take the risk. Pilot test

first.” These are important reasons for un-dertaking a pilot study, but there are ad-ditional reasons, for example convincingfunding bodies that your research proposalfor the main study is worth funding. Thuspilot studies are conducted for a range of different reasons (see Table 1).

Pilot studies can be based on quantitativeand/or qualitative methods and large-scalestudies might employ a number of pilot stud-ies before the main survey is conducted.Thus researchers may start with “qualitativedata collection and analysis on a relatively

unexplored topic, using the results to designa subsequent quantitative phase of the study”(Tashakkori & Teddlie 1998: 47). The firstphase of a pilot might involve using in-depthinterviews or focus groups to establish theissues to be addressed in a large-scale ques-

• Pilot studies are a crucial element of agood study design. Conducting a pilotstudy does not guarantee success in themain study, but it does increase the like-lihood.

• Pilot studies fulfil a range of importantfunctions and can provide valuableinsights for other researchers. There isa need for more discussion amongst re-searchers of both the process and out-comes of pilot studies.

tionnaire survey. Next the questionnaire, e.g.the wording and the order of the questions,or the range of answers on multiple-choicequestions, might be piloted. A final pilotcould be conducted to test the research proc-ess, e.g. the different ways of distributing andcollecting the questionnaires. For example,a recent study exploring nurses’ and midwives’ attitudes to research followed this pat-tern. In this study focus groups were usedto identify key issues from which a question-naire could be developed, and this was thenpiloted prior to the study proper (Hundley et al. 2000). On a much larger scale, thelargest (decennial) survey in the UK, theCensus (of 29th April 2001), tested methodo-

logical and other changes to the 1991 Cen-sus questionnaire on over 100,000 house-holds in 1997. This 1997 Census Test “pro- vided essential information on public reac-tion to new questions and form style as wellas assessing the success of collection andprocessing methods” (Office for NationalStatistics, General Register Office for Scot-land, Northern Ireland Statistical & Research Agency 1999: 15).

Pilot studies may also try to identify poten-tial practical problems in following the re-search procedure. For example, in a recent

Scottish study of maternity care the pilotphase demonstrated that the proposedmeans of distributing the questionnaires would not be adhered to (van Teijlingen et al. 2001). Without consulting the researchteam, the person responsible for distribut-



social research UPDATE

come obvious until the larger scale study isconducted.

A further concern is that of contamination.This may arise in two ways:

1. where data from the pilot study are in-cluded in the main results;

2. where pilot participants are included inthe main study, but new data are col-lected from these people.

Social scientists engaged in predominantly quantitative research are likely to argue that:“an essential feature of a pilot study is thatthe data are not used to test a hypothesis or included with data from the actual study when the results are reported” (Peat et al.2002: 57). The obvious concern is that if there were problems with the research tooland modifications had to be made in the lightof the findings from the pilot study, data

could be flawed or inaccurate. However, where an established and validated tool isbeing used and the pilot study is determin-ing other methodological aspects such asrecruitment rates, it could be argued thatsuch data may be of value.

A common problem isthe inclusion of pilotstudy participants in the site(s) of the mainstudy. Here the concern is that such partici-pants have already been exposed to an in-tervention and, therefore, may respond dif-ferently from those who have not previously experienced it. This may be positive, for ex-ample the participants may become moreadept at using a new tool or procedure. How-ever it may also be negative with participantsshowing a decline in following a protocolbecause it is no longer novel. Indeed bothchanges in behaviour have long been recog-nised and a ‘run in’ period, where an inter- vention is introduced prior to a study, is of-ten used for these reasons. The concernabout including participants from the pilotstudy in the main study arises because only those involved in the pilot, and not the wholegroup, will have had the experience. In somecases however it is simply not possible to

exclude these pilot-study participants be-cause to do so would result in too small asample in the main study. This problemarises in particular where the samples areclusters, for example schools, prisons or hospitals. In such cases one can conduct a

ing the questionnaires from the hospitalrecords department decided that it was bet-ter to distribute them through the commu-nity midwives. This was despite the fact thatthe hospital itself had suggested the recordsdepartment as a means of distribution. Other problems such as poor recording and re-sponse rates can also be identified and pre-cautionary procedures or safety nets can bedevised.

The steps used to pilot a questionnaire on asmall group of volunteers, who are as simi-lar as possible to the target population, arelisted in Table 2.

Pilot studies can also uncover local politicsor problems that may affect the researchprocess. In the study described above, themanagers of maternity services had differentperceptions of what the forthcomingchanges in the Data Protection Act (1998)

allowed them to do about the involvementof their clients in research. In the above men-tioned recent Scottish study of maternity careone head of midwifery voiced ethical con-cerns about the use of follow-up or reminder letters due to a previous local incident, whereparents of an ill baby had been sent a ques-tionnaire which was felt to be inappropriate,and as a result of changes to the UK DataProtection Act. Consequently reminders were sent out via the head of midwifery incase there were any problems with the new-born baby.

Problems of pilot studiesIt should be recognised pilot studies may alsohave a number of limitations. These includethe possibility of making inaccurate predic-tions or assumptions on the basis of pilotdata; problems arising from contamination;and problems related to funding. These is-sues are now discussed in turn.

Completing a pilot study successfully is nota guarantee of the success of the full-scalesurvey. Although pilot study findings may offer some indication of the likely size of theresponse rate in the main survey, they can-

not guarantee this because they do not havea statistical foundation and are nearly alwaysbased on small numbers. Furthermore,other problems or headaches may not be-

Table 1. Reasons for conducting pilotstudies

• Developing and testing adequacy of research instruments

• Assessing the feasibility of a (full-scale) study/survey

• Designing a research protocol

• Assessing whether the research pro-tocol is realistic and workable

• Establishing whether the samplingframe and technique are effective

• Assessing the likely success of pro-posed recruitment approaches

• Identifying logistical problems whichmight occur using proposed methods

• Estimating variability in outcomes tohelp determining sample size

• Collecting preliminary data

• Determining what resources (fi-nance, staff) are needed for a plannedstudy

• Assessing the proposed data analy-sis techniques to uncover potentialproblems

• Developing a research question and

research plan• Training a researcher in as many el-ements of the research process as pos-sible

• Convincing funding bodies that theresearch team is competent and knowl-edgeable

• Convincing funding bodies that themain study is feasible and worth fund-ing

• Convincing other stakeholders thatthe main study is worth supporting



sensitivity analysis (or sub-group analysis)to assess to what extent the process of piloting influences the size of the interven-tion effect.

Contamination is less of a concern in quali-tative research, where researchers often use

some or all of their pilot data as part of themain study. Qualitative data collection andanalysis is often progressive, in that a sec-ond or subsequent interview in a seriesshould be ‘better’ than the previous one asthe interviewer may have gained insightsfrom previous interviews which are used toimprove interview schedules and specificquestions. Some have therefore argued thatin qualitative approaches separate pilot stud-ies are not necessary (e.g. Holloway 1997:121). For example, a qualitative inter-viewer conducting 15 focus group inter-views will listen to the recordings or read

through the transcripts of the first three or four in order to improve the questions, theway of introducing the issues into the groupinterview or even to add new topics. Thus,although there is no specific pilot study,analysis of the earlier focus groups may helpimprove the later ones. However, Franklandand Bloor (1999: 154) argue that pilotingprovides the qualitative researcher with a“clear definition of the focus of the study”which in turn helps the researcher to con-centrate data collection on a narrow spec-trum of projected analytical topics. Piloting

of qualitative approaches can also be car-ried out if “the researcher lacks confidenceor is a novice, particularly when using theinterview technique” (Holloway 1997: 121).

Problems may also arise where a pilot study requires a significant investment of re-sources, making it difficult for the study team to call a halt to the research after anunsuccessful pilot study. Researchers mightbe tempted to make considerable changesin the main study, rather than deciding thatthe proposed study is not possible with theavailable resources, time, population, etc. Incontrast, funding bodies may be reluctant

to fund a further study if the pilot has beensubstantial as they may view the research asno longer original, especially if results fromthe pilot study are published.


social research UPDATE is

distributed without charge onrequest to social researchersin the United Kingdom by theDepartment of Sociology atthe University of Surrey as partof its commitment to support-ing social research trainingand development.Contributions to social re-search UPDATE that review current issues in social re-

search and methodology inabout 2,500 words are wel-come. All UPDATE articlesare peer-reviewed.

Why are pilot studies notreported?Publication bias may occur because of a ten-dency for journals to accept only papers thathave statistically significant results and notto report non-significant effects (Mahoney

1977; Chann 1982; Dickersin, 1990). A recentstudy exploring research on passive smok-ing found a difference of two years in themedian time to publication between findingsfrom significant and non-significant studies(Misakian & Bero 1998). It follows that pa-pers reporting methodological issues, suchas those identified during the pilot phase of a study, will also be less attractive to pub-lishers.

Selective publication of research results hasbeen recognised as a problem. It may leadto an overestimation of the effectiveness of

interventions, exposing patients to uselessor harmful treatments, while overestimationof adverse effects may mean that patients aredenied effective forms of care (Oxman et al.1994). In the past editors have recognisedthe dangers of publication bias with respectto clinical trials and have offered ‘an amnesty for unpublished trials’ in an attempt to over-come these problems (Smith & Roberts,1997). However, it is equally important toensure that lessons learned with respect tothe research method are shared, otherwisepatients may be subjected to poorly devel-oped tools or money may be wasted because

methods of recruitment failed. A consistentselection bias favouring reports of primary research over papers on research methods,theoretical thinking, or secondary analysis,can lead to many of researchers re-inventingthe wheel without having had the opportu-nity to learn from other people’s experience.

ConclusionsIt has been said that pilot studies are likely to be “underdiscussed, underused andunderreported” (Prescott and Soeken, 1989p60). Full reports of pilot studies are rare inthe research literature (Lindquist, 1991;Muoio et al, 1995, van Teijlingen et al. 2001). When reported, they often only justify theresearch methods or particular research toolused. Too often research papers only refer to one element of the pilot study, for exam-

Table 2. Pilot study procedures to im-prove the internal validity of a question-naire

• administer the questionnaire to pilotsubjects in exactly the same way as it willbe administered in the main study

• ask the subjects for feedback to iden-tify ambiguities and difficult questions

• record the time taken to completethe questionnaire and decide whether itis reasonable

• discard all unnecessary, difficult or ambiguous questions

• assess whether each question givesan adequate range of responses

• establish that replies can be inter-preted in terms of the information thatis required

• check that all questions are answered• re-word or re-scale any questions thatare not answered as expected

• shorten, revise and, if possible, pilotagain

(Source: Table 3.23 in Peat et al. 2002: 123)



social research UPDATE(ISSN: 1360-7898)

is published by

Department of Sociology

University of SurreyGuildford GU2 5XH

United Kingdom.

Tel: 01483 300800

Fax: 01483 689551Edited by Nigel Gilbert

(e-mail: [email protected])

Winter 2001 © University of Surrey


ple, to the ‘pre-testing’ or ‘pilot testing’ of aquestionnaire (De Vaus, 1993). Such paperssimply state: “the questionnaire was testedfor validity and reliability.” When pilot stud-ies are mentioned in more detail in academicpapers and reports, researchers regularly comment that they “had learned from thepilot study” and made the necessary changes, without offering the reader details about what exactly was learnt. Some of these proc-esses and outcomes from both successfuland failed pilot studies might be very usefulto others embarking on projects using simi-lar methods and instruments. This is par-ticularly important because pilot studies canbe “time-consuming, frustrating, and fraught with unanticipated problems, but it is better to … deal with them before investing a greatdeal of time, money, and effort in the fullstudy” (Mason and Zuercher, 1998). It has

also been argued that the current researchclimate demands accountability from re-searchers, which means that there is a needto ensure the best possible use of researchresults (Crosswaite and Curtice 1994). We would like to go one step further and arguethat researchers have an ethical obligationto make the best use of their research expe-rience by reporting issues arising from allparts of a study, including the pilot phase.

Well-designed and well-conducted pilot stud-ies can inform us about the best researchprocess and occasionally about likely out-

comes. Therefore investigators should beencouraged to report their pilot studies, andin particular to report in more detail the ac-tual improvements made to the study designand the research process.

ReferencesBaker, T.L. (1994), Doing Social research (2nd

Edn.), New York: McGraw-Hill Inc.

Bowling, A. (1997), Research Methods in Health: Investigating Health & Health Services, Buckingham: Open University Press. p 232.

Burns, N. and Grove, S.K. (1999), Understand-ing Nursing Research (2nd edn.), Philadel-

phia: W.B. Saunders Company. p 40.Crombie, I.K. and Davies, H.T.O. (1997),

Research in Health Care: Design, Conduct and Interpretations of Health Services Research, London: Wiley.

Crosswaite, C. and Curtice, L. (1994), Dissemi-nating research results-the challenge of bridging the gap between health researchand health action. Health Promotion

International 9: 289.

De Vaus, D.A. (1993), Surveys in Social Research (3rd edn.), London: UCL Press.

Frankland, J. and Bloor, M. (1999), Some issuesarising in the systematic analysis of focusgroup material, In: Barbour, R. andKitzinger, J. (eds) Developing Focus Group

Research: Politics, Theory & Practice,London: Sage

Holloway, I. (1997). Basic Concepts for Qualitative Research, Oxford: BlackwellScience.

Hundley, V, Milne, J, Leighton-Beck, L. et al.(2000). Raising research awareness amongmidwives and nurses: does it work? Journal

of Advanced Nursing 31 (1): 78-88.

Lindquist, R. (1991). Don’t forget the pilot work! Heart Lung 20: 91-92.

Mason, D.J. and Zuercher, S.L. (1995). Pilotstudies in clinical nursing research. Journal

of the New York State Nursing Association26: 11.

Muoio, R., Wolcott, L., and Seigel, H. (1995). A win-win situation: The pilot program. Journal of Continuing Education in Nursing. 26: 230-233.

Office for National Statistics, General Register Office for Scotland, Northern IrelandStatistical & Research Agency. (1999), 2001Census Information Paper , GovernmentStatistical Services: London.

Peat, J., Mellis, C., Williams, K. and Xuan W.

(2002), Health Science Research: A Handbook of Quantitative Methods,London: Sage.

Polit, D.F., Beck, C.T. and Hungler, B.P. (2001), Essentials of Nursing Research: Methods, Appraisal and Utilization. 5th Ed.,Philadelphia: Lippincott Williams & Wilkins

Prescott, P.A. and Soeken, K.L. (1989), Thepotential uses of pilot work. Nursing

Research 38: 60-62.

Rosenberg, K.M. and Daly, H.B. (1993), Foundations of Behavioural Research: A Basic Question Approach, Fort Worth/ London: Harcourt Brace College Publishers.p 182-184.

Tashakkori, A and Teddlie, C. (1998), Mixed Methodology: Combining Qualitative & Quantitative Approaches, Sage

Teijlingen van, E., Rennie, A.M., Hundley, V.,Graham, W. (2001), The importance of conducting and reporting pilot studies: theexample of the Scottish Births Survey,

Journal of Advanced Nursing 34: 289-295

SAMPLE SIZE FOR PILOT STUDY

Documents

Transcript of SAMPLE SIZE FOR PILOT STUDY