40 IT Pro July ❘ August 2002 1520-9202/02/$17.00 © 2002 IEEE

Hit-and-Run TacticsEnable GuerrillaCapacity Planning

Neil J. Gunther

W e so-called performance expertstend to regurgitate certain per-formance clichés to one anotherand to anyone else who will listen.

Here are two such clichés:

• Acme Corp. lost a $40 million sale because itsnew application cannot meet service-level tar-gets under heavy load. How much money mustthe company lose before its managementembraces capacity planning?

• Company XYZ spent $1 million buying per-formance management tools,but it won’t spend$10,000 to train its staff to use the capacity plan-ning functionality. They just continue to pro-duce and consume endless strip charts withoutregard for what that data might imply abouttheir company’s future.

Several years ago, I stopped mindlessly reiteratingstories like these and took a hard look at the realsituation: People were not just failing to gravitate

toward capacity planning, theywere actually avoiding it at allcosts.I devised guerrilla capacityplanning—tactical methods forquickly assessing capacity andperformance—to solve the prob-lems managers had with tradi-tional capacity planning.

WHY MANAGERS RESISTCAPACITY PLANNING

In coming to grips with this

new awareness, I discovered some unusual rea-sons managers had for avoiding capacity plan-ning.Two of the most compelling have to do withmisperceptions about risk.

As long as it fails on timeSome executives and managers believe they

don’t need to do anything about capacity planning,because it will always be someone else that losesmillions of dollars due to poor performance. Butthis is a false perception about risk.Such faulty per-ceptions often subvert effective risk management.The “Risk Management versus Risk Perception’’sidebar explains why this view is so deep seated.

Companies primarily employ managers to con-trol schedules. To emphasize this fact to my stu-dents, I tell them that managers will even let aproject fail—as long as it fails on time! Many ofmy students are managers, and none has dis-agreed with me yet. In other words, managersoften resist capacity planning because they aresuspicious that it will interfere with project plan-ning. Under such scheduling pressures, the focusis on functionality first.

Unfortunately, senior managers and marketingoften over-prescribe new functionality because theysee it as a competitive differentiator.All the devel-opment time, therefore, tends to go toward imple-menting the new functionality. In the clamor to getthe new functionality to market as fast as possible,the overall application often fails to meet perform-ance expectations. So, the application arrives onschedule, but it fails to meet all the requirements(especially performance requirements); it fails on

A leaner version of traditionalcapacity planning can help you hone system performancewithout inflating schedules ordelaying time to market.

Risk Management Versus RiskPerception

Guerrilla ToolsGuerrilla Resources

Inside

July ❘ August 2002 IT Pro 41

time! As this article tries to explain,applicationdevelopment doesn’t have to be like this.

Performance homunculusA homunculus is the sensate representation

of the human body. Reflecting this sensoryweighting, the homunculus’ hands and mouthare huge, while its torso and head appear rel-atively small, as Figure 1 shows. This isbecause humans receive vastly more sen-sory information via fingers and tonguesthan via the skin on our torsos, forexample.

Just as the homunculus emphasizes thedisproportionate weighting of our sensesto our bodies, so too performance manage-ment (which draws on many skills and disci-plines) should receive a disproportionatelylarge part of system management investment.This is not, however, conventional wisdom.

Performance management, most often, isseen as just another system managementactivity. In other words, the resources requiredto carry out performance management areroughly the same as those required for man-aging the distribution of software,backup andrecovery, charge back, and security. But thisis another misconception like the one involv-ing risk.Almost without exception, it’s possi-ble to accommodate most of the latteractivities by purchasing appropriate COTS(commercial off-the-shelf) packages, whichrequire little support in terms of human infra-structure.

Because performance management in gen-eral (and capacity planning in particular) islike a homunculus, it requires disproportion-ately more resources,more human infrastruc-ture, and more training than almost any othersystem management activity.Nonetheless, theplethora of COTS performance tools on themarket continues to provide false assurancethat performance management is just anotherlimb on the body of system management,rather than the huge helping hands of thehomunculus that it could be.

WHY THE CAPACITY PLANNINGPROCESS MUST CHANGE

Given these observations, I began to thinkabout explaining why managers were avoiding capacity plan-ning in today’s computer environments (Neil Gunther,“Shooting the RAPPIDs:Swift Performance Techniques forTurbulent Times,’’ CMG 97 Proc.,Computer MeasurementGroup, Turnersville, N.J., 1997, pp. 602-613). I realized that

it’s the capacity planning process itself that must change.

Software is (often) the performance bottleneck

Capacity planning has long been accepted as a necessary

Consider the poor fellow driving to the airport with white knuck-les because he just heard a radio news report about a plane crash,andnow he’s fretting about the safety of his upcoming flight. What’s

wrong with this picture? Statistics tell us that he has agreater risk—by a factor of 30 or more—of being killed

on the freeways than the airways.Our traveler has also heard these same statis-tics. So, why doesn’t he remind himself of thisimportant fact and look forward to his flight,

in spite of the air disaster that day? Try doingthis sometime—it doesn’t work. It’s a psycho-logical issue, not one of rational thought.On the freeway, our intrepid driver feels he is

in control because his hands are firmly on the steer-ing wheel. But on the aircraft, he is just another fearful passengerstrapped into his seat. This fear registers at a deep, personal levelof (false) insecurity. On the freeway, he remains oblivious to thepossibility of complete obliteration by another careless driver.

This illustrates the essential difference between risk perceptionand risk management. Companies pay managers to be in control.Therefore, a manager’s perception is that bad things will not hap-pen to his project because doing so would be tantamount to admit-ting that he is not really in control, a catch-22 situation.

Risk Management Versus Risk Perception

Figure 1. Performance homunculus.

Capacity planning is more like the hands of the homunculusthan simply part of the torso (systems management).

42 IT Pro July ❘ August 2002

C A P A C I T Y P L A N N I N G

evil for mainframe and data network procurement (FrankHuebner, “Performance and Capacity Evaluations of IPNetworks and Systems,’’ IT Professional, Nov.-Dec. 2001,pp. 38-43). The motivation is simple: Hardware compo-nents have always been expensive, and budgets havealways been limited. But today, hardware—even main-frame hardware—has become relatively cheap.

In launching a new application and system, however, ITprofessionals must consider a less-obvious caution:Bottlenecks are more likely to arisein the application software than inthe hardware. So throwing morehardware at a performance prob-lem might not necessarily help.

In this sense, capacity planninghas not gone away. Time is money,even if you have all the hardware inthe world. The new emphasis is onsoftware scalability, and thatemphasis affects the way you do capacity planning.Today’sdemands on time and scheduling constraints no longeraccommodate the traditional detailed approach to capac-ity planning based on procurement for the monolithicmainframe. Moreover, today’s computer systems employdistributed computing architectures. Distributed comput-ing comprises many software pieces running in manyhardware places.

Complex distributed-computing environment

Several factors make the more complex environment ofdistributed computing less amenable to traditional main-frame capacity planning methods:

• Third-party or in-house applications have little or noinstrumentation.

• There’s no such thing as Unix; there’s AIX, HPUX,Solaris, BSDI, FreeBSD, RH Linux, Debian Linux, theMacOS, and so on.

• Scripts built on one form of Unix are almost guaranteednot to work on another variant.

• There’s not even one type of Windows operating systemanymore.

• Multiple COTS applications run on multiple vendorplatforms.

• Today’s systems lack a common performance databaselike RMF (Resource Measurement Facility) or SMF(System Management Facility), which are available forMVS (multiple virtual storage) mainframes.

• Most commercial tools have mainframe roots and arethus server-centric in their data collection capabilities.Monitoring network and application data does requireadditional tools.

• There’s no convenient way to comprehend resource con-sumption across multiple tiers.

Little standardization or instrumentationThe Universal Measurement Architecture (UMA) stan-

dard from the Open Group (http://www.opengroup.org/pubs/catalog/c427.htm) could have helped surmount someof these difficulties. This standard offers a means to nor-malize collection and management tools for performancedata (especially Unix data).

But vendors and other competing parties had no strongincentive to adopt UMA.To provide performance data in

UMA format, platform vendorswould incur a cost burden withoutan obvious return on investment.Tool vendors thought a standardformat made it easier for new com-petitors to enter the market, threat-ening their revenue streams.Similarly, ARM (ApplicationResource Measurement),a standardprimarily promoted by Hewlett-

Packard and Tivoli, has not really caught on either.So, today’s IT professionals are building more complex

architectures with less instrumentation available to man-age them. I’m glad Boeing doesn’t build aircraft this way!

GETTING STARTED WITH CAPACITY PLANNINGBy now it should be clear that capacity planning is as

important as ever, but capacity planners today must sur-mount two new hurdles that traditional planners did not:

• We’re trying to do more with less. Distributed systemsare ubiquitous and more complex than monolithic main-frames, but we have fewer standardized tools to manageperformance and plan for growth.

• We have to do more in less time.Whatever performancemanagement and capacity planning we do, it must bedone in less time so as not to inflate product/projectschedules.

Taken together, these constraints seem impossible tomeet. Is it any wonder that managers give up and ignorecapacity planning? But, as you will see, there is a way out.

First, it’s important to realize that performance man-agement has three major levels: monitoring, analysis, andplanning.

Managers usually pay the most attention to performancemonitoring, because it is generally the easiest area toaddress. If you want to manage performance and capacity,you must measure it. Naturally, this is the activity that themajority of commercial tool vendors target. (As a man-ager, if you spend $250,000 on tools,you inevitably feel likeyou must have accomplished something.)

If you can’t afford these prices,you can turn to your Unixor Windows NT system administrator. They are usuallyvery good at writing scripts to collect all sorts of perform-ance data and presenting it to you in your Web browser.

Think of traditionalcapacity planning as

the 800-pound gorilla.That gorilla needs to go on a diet.

July ❘ August 2002 IT Pro 43

But data collection just generates data.Thenext level is analysis: uncovering the infor-mation hidden in the data.These days, unfor-tunately, the usual motivation forperformance analysis is to firefight an unfore-seen performance problem that is delaying arelease schedule or reducing deployed func-tionality.This is performance analysis after thehorse has bolted.

With a little more investment in infrastruc-ture, managers can plan ahead, minimizingthe outbreak of those unforeseen fires. But,as was explained earlier, managers usuallyskip this third level of planning for fear ofinflating project schedules. How can man-agers cut this Gordian knot?

GUERRILLA CAPACITY PLANNINGIn my view, managers need a more oppor-

tunistic approach to capacity planning; enterguerrilla capacity planning.

The notion of planning tactically mightseem contradictory. At the risk of mixingmetaphors, think of traditional capacity plan-ning as the 800-pound gorilla. That gorillaneeds to go on a diet to produce a leanerapproach to capacity planning that is adaptedto today’s business environment. By lean, Idon’t mean skinny. “Skinny” means remain-ing stuck in the rut of simply monitoringeverything that moves with the false hope thatcapacity issues will never arise, and thusavoids the planning level altogether.Monitoring requires that someone watch the“meter needles” wiggle. Inherent in thisapproach is the notion that action isn’t nec-essary unless the meter redlines. But per-formance meters only convey the system’scurrent state.Such a purely reactive approachdoes not provide any means for forecastingwhat lies ahead. You can’t forecast the weather by listen-ing to leaves rustle.

The irony is that a lot of predictive information is likelycontained in the collected monitoring data. But, like pan-ning for gold, it takes additional processing to reveal thehidden gems about the future.

It’s not a model railwayThe goal of capacity planning is prediction and that

requires a consistent framework in which to couch anyassumptions. Capacity planners call that framework amodel. The word “model,’’ however, is one of the mostoverloaded terms in English; it can mean everything froma fashion model to a model railway set.

For example, the best model train set is usually the one

that includes not just a scale model of the locomotive, notjust a model of an engineer driving the scaled locomotive,but the one that includes the pupil painted on the eyeballof the engineer driving the scaled locomotive. In otherwords, the more detail the better.

This is precisely what a capacity planning model is not.For capacity planning, the goal is to discard as much detailas possible while still retaining the essence of the system’sperformance characteristics.This goal tends to argue againstthe construction and use of detailed simulation models, infavor of spreadsheets or even automated forecasting. Theskill lies in finding the correct balance.Linear trend modelscan be too simple in many cases, while event-based simula-tion models can be overkill.To paraphrase Einstein: Keepthe model as simple as possible, but no simpler.

Books➤ Computer Performance Analysis

with Mathematica, A. Allen, Aca-demic Press, San Diego, 1994: Con-tains a very readable overview ofcapacity planning issues and tech-niques written by a master of thesubject. Also, it is the only book Iknow of that shows you how to apply Mathematica to solve per-formance problems.

➤ Guerrilla Capacity Planning: Hit and Run Tactics for SizingUNIX, Windows and Web Applications, N.J. Gunther: Thisbook is in preparation; see http://www.perfdynamics.com forinformation.

➤ Sun Blueprints: Capacity Planning for Internet Services, A.Cockcroft and W. Walker, Prentice Hall, Upper Saddle River,N.J., 2000: One of these authors has adopted and extendedmany of the concepts I present in this article. The book alsooutlines guerrilla-style management processes and scenarioplanning; some chapters are Sun/Solaris specific.

➤ The Web Testing Handbook, Steven Splaine and Stefan P.Jaskiel, STQE Publishing, Orange Park, Fla., 2001: This bookcovers the interface between functional testing and perform-ance measurement. Chapter 8 covers application scalabilityand, in particular, the “retrograde” throughput effect discussedin example 1 of this article.

Classes➤ Guerrilla Planning (http://www.perfdynamics.com/PitV/

guerilla.html): I teach a five-day class on these techniques.

➤ Guerrilla Tools (http://www.perfdynamics.com/PitV/gtk.html):This five-day class is a lab in which students participate in theconstruction of performance and capacity models.

Guerrilla Resources

44 IT Pro July ❘ August 2002

No compass requiredTraditional capacity planning has required relatively

high precision because each significant digit of the calcu-lation had many thousands of dollars attached to it. Intoday’s lower-cost climate, however, managers often justwant a sense of direction rather than the actual compassbearing. In this sense, the precision of traditional capacitypredictions has become less important than their accuracy.Managers often see little virtue in spending two monthsdebugging and verifying a full-blown simulation if theaccuracy of a simpler spreadsheet model would suffice.

At a level of performance data collection, there is littlesupport for high-precision measurements in open systems.Take Unix, for example: It’s basically an experiment thatescaped from the lab around 1975 and has been mutatingever since.What little performance instrumentation existedback then in the Unix kernel was for the benefit of earlydevelopers, not the grand purpose of capacity planning intoday’s distributed systems. Nonetheless, every capacityplanning tool in existence relies primarily on those samekernel counters with little modification.And since the PCrevolution of the 1980s fueled the move to distributed sys-tems,performance management has become ad hoc,at best.

Guerrilla capacity planning, on the other hand, tries tofacilitate rapid forecasting of capacity requirements basedon available performance data without inflating schedules.

Consistent investment in human infrastructure is alsoimportant. People must have the training and know-howto use the methods and tools at hand. Another key pointis to keep ahead in the procurement cycle.This was nevertruer than it is for exponentially growing Web sites. If youdon’t accurately forecast capacity, user traffic can disap-pear to competitor sites. Or latent and unanticipateddemand can consume newly procured capacity the instantyou install new servers. Let’s look at two brief examplesof guerrilla capacity planning in action.

Example 1: Sizing by numbersAs a first example of how to use guerrilla

capacity planning, here is a tactical methodfor quantitatively determining applicationscalability. It’s noteworthy that scalability—particularly application scalability—is aperennial hot button that involves notions ofperformance and planning,yet few people canquantify the concept.

Scalability has to do with the laws of dimin-ishing returns so a single number cannot rep-resent it. Scalability is a function. Figure 2shows an example of actual load test through-put plotted as number of scripts S executedper hour on the y-axis as a function of num-ber of virtual users N on the x-axis.

Superimposed on this data is the corre-sponding scaling function predicted by a sim-ple capacity model that does not involve

queueing theory or simulations. This means that sizingserver capacity can be relatively quick using a spreadsheet.

The formula for the simple capacity model is

S(α, β, N) = N/{1 + α [(N − 1) + βN (N − 1)]}

It involves just two parameters (N.J.Gunther,The PracticalPerformance Analyst, iUniverse Inc., Lincoln, Neb., 2000).You identify α with contention delays, for example, timespent waiting on a database lock. β is associated with addi-tional delays due to pair-wise coherency mismatches, suchas time to fetch a cache miss.These delays can be in hard-ware, software, or (most likely) a combination of both.

You can easily enter this function in a Microsoft Excelspreadsheet. If the number of virtual users resides in col-umn N, and regression parameters α and β reside in cellsA1 and B1, the equation becomes the Excel cell formula

=Nn/(1+A1*((Nn-1)+B1*Nn*(Nn-1)))

where Nn is the value in the cell at column N and row n.You can now determine scalability parameters α and βusing the linear-regression tools built into Excel (see, forexample,D.Levine,M.Berenson,and D.Stephan,Statisticsfor Managers Using Microsoft EXCEL, Prentice-Hall,Upper Saddle River, N.J., 1999). In summary, the basicsteps of this guerrilla technique are as follows:

• Measure throughput as a function of load N using toolslike LoadRunner.

• Collect a sparse data sample; having at least four loadpoints is usually sufficient.

• Calculate α and β by performing a regression fit of Excelagainst transformed versions of N and S.

• Use those values to predict the complete application scal-ability function using the Excel formula discussed earlier.

C A P A C I T Y P L A N N I N GN

o. o

f sc

rip

ts p

er h

ou

r

180 36 54 72 90 108 126 144 162 180 198 216 234No. of virtual users

200

400

600

800

1,000

1,200

1,400

1,600

1,800

2,000

ModeledMeasured

Figure 2. Load test measurements and capacity model.

July ❘ August 2002 IT Pro 45

You can find a more detailed account of thisprocedure at http://www.teamquest.com/html/gunther/fitting.shtml.

An essential feature of this simple model isthat it can predict retrograde throughputs(where the amount of completed computa-tion decreases as the system load increases)like that in Figure 2. This effect is not easilymodeled using conventional performancemodeling tools based on queueing theorywithout specialized load-dependent servers.

What are the benefits of this guerrilla sizingmethodology? Primarily, it avoids the needfor more complicated queueing theory or sim-ulation models. However, the most significantbenefit is not the model’s technical merits butthe fact that it creates a framework againstwhich to check the consistency of load meas-urements. If the data does not fit the model inthe first equation, there is very likely a prob-lem with the measurement process that maybe worth more detailed investigation.Moreover, because each of the model’s termshas a real physical interpretation, engineersfrom disparate groups quickly recognizewhich parts of the application or platformneed further tuning to improve scalability. Inthis way, the spreadsheet model is a capacityplanning tool that forecasts scalability with-out inflating the release schedule.

There are many other tools that can help insuch a “guerrilla” analysis; the “Guerrilla Tools” sidebarlists a few.

Example 2: Estimating capacity doublingtime

You can also apply the spreadsheet model to Web sitetraffic analysis, where the rapid increase in traffic growthdemands a more tactical approach to capacity planning.The growth of sites like those of eBay,AOL, and Amazonmight not be as extreme as it was last year (A.C. Lear,“Managing e-Commerce Reliability, eBay Style,’’ ITProfessional, Mar.-Apr. 2000, pp. 80, 77-79), but serverdemand will still grow.As already noted,managers of theseWeb sites know they need capacity; it’s the planning partthat is culturally unfamiliar.

I have devised a useful metric for high-growth Web sites,the capacity doubling period (N.J. Gunther,“Performanceand Scalability Models for a Hypergrowth e-CommerceWeb Site,’’ Performance Engineering: State of the Art andCurrent Trends, Lecture Notes in Computer Science 2047,Reiner Dumke and Claus Rautenstrauch, eds., Springer-Verlag, Heidelberg, Germany, 2001).This period is simplythe time until the amount of consumed processing capac-ity is twice that now being consumed. In some cases, this

period can be as short as six months.That’s about 10 timesfaster than typical data processing centers and four timesfaster than the original version of Moore’s law.

Such exponential demand for server capacity can leadto a new definition of bankruptcy—if you have to purchasea lot of cheap servers, pretty soon you’re talking realmoney. Such costs force the need to plan for capacity wellin advance of the procurement cycle.

Once again, you can determine the capacity doublingperiod by using elementary tools, like spreadsheets. If, forexample, you measure processor utilization U at regularlyscheduled intervals, you can estimate long-term con-sumption by assuming an exponential trend

Ufuture = Unow eλW

where growth rate λ is determined by using the AddTrendline facility in Excel, and W is the number of weeksover which to fit the data. Given this relationship, the dou-bling time is

Tdouble = (ln 2)/ λ

I chose an exponential-growth model because it is the

➤ CSIM (Mesquite Software, http://www.mesquite.com): Moderately priced simu-lation package.

➤ Excel (Microsoft,http://www.microsoft. com):Spreadsheet in MicrosoftOffice suite.

➤ MathCad (MathSoft Engineering& Education Inc., http://www. math-soft.com): Commercial symbolic-computation program.

➤ Mathematica (Wolfram Research, http://www.wolfram.com):Commercial, general-purpose program for symbolic computa-tion.

➤ Minitab statistical package (http://www.minitab.com): Nextlevel above Excel.

➤ PDQ (Performance Dynamics Consulting, http://www.perfdynamics.com/Tools/PDQ.html): An open-source per-formance analyzer.

➤ R (http://www.r-project.org): Public-domain version of S+ statistical-analysis product.

➤ SPE *ED (Performance Engineering Services, http://www.perfeng.com/sped.htm): Tool for software performance engi-neering (SPE).

➤ TeamQuest performance software (http://www.teamquest.com): Commercial capacity-modeling tools.

Guerrilla Tools

46 IT Pro July ❘ August 2002

simplest function that captures the notion of compoundedgrowth. It also reflects superlinear revenue growth models.If you decide to use statistical models, you might want toconsider using more robust tools like Minitab or R, men-tioned in the “Guerrilla Tools” sidebar.

The next task is to translate these trends into procurementrequirements. Because trend lines pertain only to measure-ments from the current system configuration,you need a wayto extrapolate to other possible configurations.For this pur-pose, I used the scalability function discussed in Example 1.

A s I hope these examples demonstrate,guerrilla capac-ity planning provides an approach to assessing appli-cation scalability that matches management’s

requirement to keep a tight rein on project schedules.In many situations where managers tend to avoid tradi-

tional capacity planning, the guerrilla approach can pro-vide a simple framework to bring disparate groupstogether and reveal unanticipated performance issues.Once revealed, these issues are addressable within the con-text of existing schedules. In this way, guerrilla capacityplanning helps keep projects on schedule and minimizesrevisions. Think of it as a way of managing hidden time

sinks. It’s also a way of replacing risk perceptions with riskmanagement. Sometimes, the biggest hurdle preventingthe introduction of guerrilla capacity planning is simplygetting started.

Another way of defining guerrilla capacity planning isto see it from the viewpoint of those of us who do it:

• Management plans; we review.• Management decides; we measure.• Management dithers; we propose.• Management revises; we plan.

Aspects of capacity planning that I have not discussedhere include floor space, power, cooling, disk storage, tapestorage, and so on. You can address many of these issueswith spreadsheet models similar to those presented here.I encourage you to consider adopting guerrilla capacityplanning in your organization. �

Neil J. Gunther is chief scientist at Performance DynamicsConsulting in Castro Valley,Calif.;http://www.perfdynamics.com. Contact him at njgunther@perfdynamics.com.

C A P A C I T Y P L A N N I N G

IT FOR MEDICINE &PUBLIC HEALTH

IT FOR MEDICINE &PUBLIC HEALTH

Call for Articles

The increasing use of computer- andtelecommunications-based technolo-gies in the fields of healthcare,telemedicine, biomedical research,and pharmaceutical research hasgenerated a great deal of cross-fertilization among these previouslyseparate disciplines, resulting innew IT-based approaches. IT Proseeks articles highlighting theseinnovative applications.

Submissions deadline: 13 September 2002

Contact lead editor Thomas Jepsen (tjepsen@mindspring.com) or see ourWeb site for more details.Jan.-Feb.

2003 http://computer.org/itpro/