RESEARCH METHODSBy Gordon A. Wyner

Product Testing: Benefits and Risks

PRODUCT TESTING IS OFTEN CRiTictZHD FOR

doing little to reduce the new-product fail-ure rate. Management may hold it"responsible" for not identifying break-through consumer and business products:others view it as exceedingly narrow andconservative in its outlook. Many criticizeit for merely providing reasons not to pur-sue new products rather than generatingsuccesses. Another complaint is that prod-uct testing only deals with the necessarybut ultimately less important tacticaldetails of fine-tuning products. Is this afair assessment of product testing?

Some of the heat generated by thedebate over product testing is due toexcessive expectations. Some is undoubt-edly due to problems in applying thetnethods. Still other concerns are ground-ed in the inherent limitations of research.In order to better gauge its usefulness, it isimportant to assess what product testingean and cannot do. Following are some ofthe issues that this article will explore:

• What questions can product testinganswer?

• What are the risks associated withforgoitig product testing and withproduct testing itself?

Gordon A.Wyner is VicePresident ofMercerManagementConsulting inLexington. Mass.

• How do marketers protect against therisks?

• How is product testing being chal-lenged to provide more value?

QUESTIONS ADDRESSED

The key questions addressed by prod-uct testing fall into three categories thatreflect the underlying business issues andthe stage during which they occur in theproduct-development process: screening,optimization, and estimation. Thoughsome studies attempt to ask and answer ailthree questions at once, there is usually anemphasis on one particular type.

ScreeningIn the earliest stages of product devel-

opment., ideas for new products are beinggenerated and only prelitninary investiga-tions of concept viability are undertaken.At this point, product testing can be usedto help screen a large number of concepts,trimming them to a shoit list for deeperinvestigation.

The long list may have come frombrainstorming sessions, qualitativeresearch, competitive reviews, and judg-ment. Although it tnay he obvious tosome members of management whichideas are good or bad, objective researchwith input from the potential buyers helpsmake the cuts needed when developmen-tal resources are limited. The research isan aid to selection, not tbe determinant ofwhich product will go to market. Ifscreening research can eliminate the"dogs," it will have been usefui. If it canalso point to a small number of options topursue, it will have been successful.

OptimizationOptimization is the goal when testing a

particular product, implemented, for

example, as part of a detailed investiga-tion or planning for a new product. Bythis point, screening at various stages hasreduced the decision frotn what to pursueto how to make a particular product thebest it can be. In some instances, opti-mization can be achieved in the mathe-matical sense, perhaps determined by thehigh point of product appeal on a responsecurve or surface reflecting the best combi-nation of all its cotnponent ingredient lev-els. Typically, though, the term meansthat testing has identified the characteris-tics of the product that are critical to buy-ers' decisions to use it, and providesdirection in developing the most attractiveor appealing configuration of those char-acteristics.

In food testing, for example, attributesof the product relate to taste, such assweetness, saltiness, texture, and appear-ance. The combination of just the rightamount of sugar, salt, crunchiness, andcolor that will please consumers can beinferred from product tests.

In many product categories the price ofthe product is a critical characteristic orattribute. Product testing can identify theprice level at which revenue or profit isbest for the manufacturer. If the criterionfor the best product is the highest prof-itability, then information about productcosts must be brought into the analysis.

Some services bave several price vari-ables, including installation, monthly fees,and per-minute usage charges. On-linecomputer services could use product test-ing to determine how to best set prices onthese three dimensions. The criterion couldbe the price that maximizes the number ofsubscribers, the amount of revenue, or theamount of profit generated. In using prod-uct testing for optimization, the marketerlearns how to configure and price the prod-uct based on customer sensitivities towardproduct attributes and price.

46 MARKETING RESEARCH

EstimationWhen the product is a given, that is,

fixed or subject to only minor modifica-tions, testing is used for demand estima-tion. As part of the pre-launch-testing andvalidation process, the research tnustdetermine whether this particuUir producthas met or failed the action standard. Thiscan be a direct comparison with a controlproduct, a comparison against the normfor the company's products, or perhaps anestimate of actual business potential interms of unit volume or dollars, [f in-mar-ket performance is the criterion for esti-tnation. then other elements of the market-ing mix, sueh as promotion and distribu-tion, must be brought into the analysis.

Suppose a tnanufacturer has developedwhat it believes is an improved version ofan existing product. Testing will revealwhether customers agree with the manu-facturer's optimistic assessment. A naturalaction standard would be that the newproduct must achieve product-testingscores that are at least equal to thosereceived by the existing product. Ideally,to deliver on the "improved" claitn, theproduet should exceed the scores of theexisting fortnulation.

Determining how lo measure improve-ment is itself a major research designissue. Is an average improvement of 10%sufficient? Should the criterion be one ofpreference in head-to-head competitionamong customers? If the criteria are unitsales and revenue, then information aboutspending for advertising and promotionand plans for achieving distribution mustbe considered.

THE RISKS OF NOT TESTING

If no testing is done, product-develop-ment and tnarketing decisions are basedon an internal view, that is, what the com-pany and its management know about itscustotners. This approach is limited to thehistorical, basing predictions of futurecustomer behavior upon the past behaviorof its customers. Ail companies use priorinfonnation to advantage in constructingfuture marketing plans. The risk is thatexclusive reliance on history and avoid-

ance of testing may miss some critical andperhaps obvious variable that couldincrease the odds of success.

For example, marketers of food prod-ucts have their own hypotheses about theimponance of naturai, heaith-orientedproducts and claims for those products.Management may decide to make productshealthier and more natural on the basis ofoverall trends in consumer behavior andthe increased sales of such products. Yetthis conflicts with consistent product-test-ing results, which show tbat consumersgenerally like products with more sugar,more salt, and more fat when cotnparedwith their healthier counterparts.

To cotnpound confusion, consumerstnay respond positively to the claim of ahealthier product, overstating the sociallydesirable answer. They even may pur-chase thetn on occasion, seeking variety.But will they use these products with suf-ficient frequency to make them successfulfrotn a marketer's perspective?Furthermore, consider that consumersmay not be able to distinguish betweenproducts that have been altered to achievemore-favorable nutrition labels and theoriginai version, like a reduced-fat crackeror chip that drops down to 30% less fatfrom 20%, or a drink that adds 10% morepure ftuit juice.

How then does management sort outthese different variables without benefit ofproducl testing? Sotneone tnust make theJudgments that consumers will or won'tlike the new formulation and that theywill or will not respond to it as part of theproduct positioning.

in some situations and industries theneed for testing may be minimal if there isalready extensive knowledge about cus-tomer-behavior patterns from transactiondata. Even in these instances, however, ittnay not be possible to project futureresponse for new products that depart fromthe current product line. If future customersare likely to be different from current cus-tomers, this limits further the relevance ofprevious knowledge and experience.

How, for example, does a bank selectconcepts, optimize products, and estimatedemand for investment products that it has

never sold before, and for which it has nocustomer information? Without some exter-nal research, iticluding finding out aboutcompetitors' alternatives, there is a risk ofmisperceiving future customer demand.

The history of new-product failures isfull of weil-engineered products that didnot appeal to customers. In manyinstances these kinds of failures resultedfrom a lack of tnarket orientation and anexcessive internal focus.

Another cotntnon cause of new-productfailure is the introduction of products thatare simply copies of competitors' offerings.The lack of differentiating benefits per-ceived by customers often leads to the "sur-prising" outcome that custotners have noreason to switch brands, and don't. Producttesting, properly conceived and executed,cat! protect against the risk that customersdon't want what they ai'e offered.

THE RISKS OF TESTING

Testing carries all the well-known risksassociated with any kind of research. Theyinclude satnpling bias, insufficient samplesize, measurement error, and errors in con-structing and presenting the stimulus {theproduct) to the respondents. But even thestaunchest critic of product testing likelywould concede that many of these riskscan be controlled through proper applica-tion of design and analytical methods. Alarger issue arises when the process ofproduct testing is improperly aligned wilhthe business goals. For example, elaborateproduct-testing systems in which manystandardized tests are adtninistered overtime can take on a life of their own. Fittingtoday's research problem into the systemcan be of greater concern than finding theanswer to the business question.

Guidance testing systems that evaluateone product at a time may never get to theoptimal product if each test leads to newmodiftcations and the pattern of previoustest results is ignored. Suppose the first testsays to make the product more salty, thesecond says to make it more tangy, thethird says to make it less cninchy. Wouldtnore crunch work with less salt? A broad-er experimental design that systematically

MARKETING RESEARCH SCRING1997 4 7

tests all the relevant attributes, but requiresa greater planning effort, may get to thebest product faster, and may even cost less.

Problems in ApplicationMany of the concerns about product

testing relate to its implementation inpractice. They inelude:

• Testing the "wrong" product—testingthe physical product only, when thewhole value proposition, includingthe brand name, is at stake.

• Comparing with the "wrong" controlproduct—testing against a weak com-petitor or benchmark.

• Asking the "wrong" person for theanswer—incorrectly identifying thedecision maker for a product, e.g.,new prescription drugs: the patient,the doctor, and/or the third-partypayor?

• Testing the product in a marketingenvironment different from the realone—using a different level of mar-keting-resource investment from theactual rollout.

• Te.sting the "wrong" consumer .seg-ment—for example, the health-con-scious segment may be the relevanttarget for health-oriented products,but not for others.

• Testing the "wrong" dependent vari-able—ignoring the distinctionsbetween appeal, interest, purchase,and usage.

• Failing to consider the relevant com-petitive context—not anticipating thesignificance of available alternativesor a competitive reaction that maynullify the impact of an innovativenew product.

• Testing the "wrong" price—setting aprice too low or high during testing.(Demand will be lower at higherprices.)

Each of these product-testing problemssuggests a potential solution: in broad

terms, test a range of alternative products,segments, and environments. This increas-es the chances that the "right" test condi-tions will be included along witb the"wrong" ones.

By testing a variety of conditions, themarketer can determine how each inputinfluences the outeome and make the mostinformed decision. However, as designcomplexity has a cost, there is still a greatneed to reduce the number of test scenar-ios to the minimum necessary. Of course,there are no guarantees that design issuescan be eliminated entirely. At best, therisks can be managed, yielding a betteroutcome than would be achieved with noproduct-testing infonnation at all.

EVALUATING THE RISKS

A risk assessment is a way to deter-mine whether management believes it ispursuing acceptable odds. For example,what are the anticipated losses from beingwrong at each stage of product testing? Inthe early stages of concept screening, it isprobably better to err on the side of safe-ty, keeping an idea alive longer ratherthan eliminating it too soon. Further early-stage testing, with a modified concept,will provide another chance to kill theidea. The research cost per idea is rela-tively low at this point. Given the dearthof good new-product ideas, there is littlegain in premature rejection.

In the more advanced stages of testing,the risks of being wrong are higher.Considerable development cost has beenexpended by the demand-evaluation stage,for example. Tbe options for the producthave been narrowed, and the research out-comes should be tied directly to economicimpact, such as volume, revenue, andprofit. It is essential to measure demandaccurately, i.e., within +/- 10%; this typeof assessment should govern the amountinvested in research relative to judgmentalforecasts.

Given some understanding of the risks,decisions can be made as to how to solvethe problems outlined above. Researchplanning can identify the options forlocating the right product, decision maker,segment, marketing context, and indepen-dent and dependent variables. The rightanswers most likely are not the result of

technical expertise alone, but flow fromknowledge of the market. They combineJudgment with data from such sources asother research, transaction records, cus-tomer complaints, customer-satisfactionmeasurement, and channel perspectives.Product testing does not simply give theanswer— it refleets the input many peopleeontribute to arrive at conclusions that aregrounded in the business. It is one impor-tant element in a company's overall deci-sion-making process for product develop-ment.

MORE CHALLENGES TO PRODUCTTESTING

In addition to the specific concernsabout product testing, there are severalbroader challenges that require a response.One is that the new-product-developmentproeess is under great pressure to finishevery stage of investigation faster, reducecycle times, and get to market faster. As aconsequence, estimates of economicimpact need to be introduced earlier in theproeess. This leads to demand estimationat the optimization stage and even at thescreening stage. Though the interpretationof early-stage economic impact must beinterpreted carefully, this provides anadditional opportunity for research to con-tribute business value.

Another challenge is to better integratethe testing process into the organization toimprove business performance. Growththrough new-produet developmentrequires more than effective managementof the research itself. It should be linkedto the overall product-planning processthrough whieh the company sets priorities,develops technologies, and balances itsoverall produet-portfolio strategy. Thoseinvolved in product development and test-ing need to see themselves as part of theoverall management team focusing onbusiness development, not solely asproviders of research results.

Many companies have gone throughseveral cycles of reengineering and areseeking ways of achieving profitablegrowth. The relevance of new-productdevelopment to business growth undoubt-edly will increase. It presents an opportu-nity for eonsumer researcb to play a sig-nificant role in this emerging trend. •

4 8 SPRING 1997 MARKETING RESEARCH