The ITEA Journal of Test and Evaluation

September/October 2003Volume 24, Number 3

continued…

TECHNICAL PAPER ABSTRACTS:

The Test and Evaluation Master Plan:An Operational View

Stephen C.Daly, Larry W. Miller, Dr. Ernest A. Seglie and George WauerOffice of the Dire c t o r, O p e ra t i onal Test and Evaluation ,Office of the Se c re t a ry of Defe n s e, Wa s h i n g t on , D. C .

New regulations go verning defense acquisition give less guidance than ever bef o re on whatshould be te s te d, how to plan a test and evaluation pro gram and how to document that planningin a Test and Evaluation Master Plan (TE M P ) . The authors of this pap er, e ach of whom hasre s p o n s i b i l i ty to recommend to the Dire c t o r, O p erational Test and Eva l u a t i o n ,w h e t h er plannedtesting is ad e q u a te, p resent a view of what is an ad e q u a te TEMP in the new acquisition envi-ronment of s p i ral devel o p m e n t , evo l u t i o n a ry ac q u i s i t i o n , and more integra ted and joint te s t i n g.

Biotechnology: Implications and O p p o rtunities for Test and Evaluation

J o rdan C. Fe i dler and Olivia J. Ta t eThe MITRE Corporation, McLean, Virginia

The dramatic growth of the life sciences over the past two decades has profound implications forthe Department of D efense (DoD). A d vances being made in molecular, c ell and sy s tems biology,and bioinformatics will produce a new set of d efense threats and ch a l l e n ges here t o f o re unimag-i n a bl e, as well as powerful new enabling te ch n o l o g i e s . The formulation of M a x wel l ’s equationsin the late 19th centu ry led to a $300 billion el e c tronics industry by 1990 with an incre d i bl yd iverse set of ap p l i ca t i o n s , many of w h i ch were inconceiva ble by Maxwel l ’s conte m p o ra ri e s .Watson and Cri c k’s elucidation of the three-dimensional stru c tu re of d e ox yribose nucleic ac i d(DNA) is ca talyzing a similar pattern of growth in the biote ch n o l o gy industry and will likel yl e ad to just as ri ch a set of eve n tual ap p l i ca t i o n s . A substantial amount of p riva te and public sec-tor money is being inve s ted in developing novel biote ch ap p l i ca t i o n s . A key issue for DoD is thedual-use potential for these new te ch n o l o g i e s , and this pap er presents a bri ef o verv i ew of how thedual-use natu re of b i o te ch re s e a rch will impact the test and evaluation community.

FE ATU RE D FAC I L ITY

Federal Aviation Administration’sWilliam J. Hughes Technical Center,

Atlantic City, New Jersey

The ITEA Journal of Test and Evaluation, September/October 2003, Volume 24, Number 3, continued

T&E of Chemical and Biological Weapon Defense Systems for Homeland Defense: New Challenges Requiring Novel Approaches

Tim MoshierH omeland Se c u ri ty Division ,

S PA RTA , I n c o rp o ra t e d , A l e x a n d ri a , Vi r g i n i a

The Department of D efense (DoD) is emba rking on an ambitious pro gram to develop and fiel dsy s tems for homeland defense against ch e m i cal and biological (CB) attac k s , but these new sy s te m swill be re q u i red to work in ways that are fundamentally different from the more familiar tac t i ca lsy s te m s . The fixe d - s i te homeland defense enviro n m e n t , w o rk forc e, t h reat and operational pro f i l ere q u i re the test and evaluation (T&E) community to look at the pro blem from a new per s p e c t ive.Fo rtu n a tel y, t h o u g h , lessons learned from recent CB sy s tem developments and deploy m e n t s ,a l o n gwith matu ring te ch n i cal cap a b i l i t i e s ,p rovide evaluators with cri t i cal insights and cap a b i l i t i e s .T h i sp ap er bri efly examines the homeland CB defense operating enviro n m e n t , rev i ews some lessonsl e a rned from the post-Septe m b er 11, 2001 (“9-11”) deployment of the Joint Biological Po i n tD e tection System (JBPDS) and recommends ap p ro aches for T&E of this new class of sy s te m s .

continued…

Using Vi rtual Environments to Support Testing of Chemical and Biological Sensors for Homeland Security

Dr. Jeffery D. PetersonBattelle Memorial Institute,

Dugway Proving Ground (DPG), Dugway, UtahHal Meyer

West Desert Test Center, DPG, Dugway, UtahDennis L.Jones and Michael J. O’Connor

ITT Industries, Alexandria, Virginia

With the escalating threat of chemical,biological, radiological and nuclear (CBRN) terrorismfacing the world today, the United States has placed increased emphasis on defending againstthis threat. To protect both military personnel and civilians alike, the U.S. Department ofDefense is actively developing and testing equipment that will detect the presence of CBRNmaterials. The West Desert Test Center, located at Dugway Proving Ground, Utah, isAmerica’s testing center for chemical and biological (CB) sensors.This paper describes past andcurrent activities where the use of CB sensors, unified in a virtual environment, can be usedto support homeland security now and in the future.

The ITEA Journal of Test and Evaluation, September/October 2003, Volume 24, Number 3, continued

Defining & Testing the Right—Rather Than the Obvious—Parameters in X-Ray-Based Inspection Systems

William J. BaukusAm e ri can Science and Engineering (AS&E), I n c . ,

B i ll e ri ca , M a s s a ch u s e t t s

X - ra y - based inspection sy s tems are re c e iving increasing attention in securi ty - o ri e n ted ap p l i ca t i o n s ,p a rt i c u l a rly in view of the tragic events of S e p te m b er 11, 2 0 0 1 . In ad d i t i o n , recent alerts by U. S .go vernment agencies rega rding the potential for both terro rists themsel ves and/or we apons of m a s sd e s truction to potentially enter the United Sta tes via shipping conta i n ers has increased aw a reness ofthe need for discriminating inspection. Based on the premise that the purpose of an inspection sy s te mis to detect contra band and dangerous material in complex ca r go e s , this pap er discusses the va ri o u smethods used to ch a rac terize X-ra y - based inspection sy s te m s , as well as the cri teria by which the te s t-ed para m e ters are interp re ted in predicting how well an X-ray sy s tem perf o rms its function of d e te c t-ing suspect ite m s . “S ta n d a rd” X - ray ch a rac teri za t i o n s ,s u ch as penetration and ability to image va r-ious wire gau ge s , a re not of t h e m s el ves sufficient to permit ad e q u a te measures of an X-ray sy s te m’sef f e c t iveness in finding contra band materi a l s . “D e te c ta b i l i ty” as a defining metric is discussed, w i t hboth transmission and Compton bac k s ca t ter X-ray images used to illustra te points.

Intelligence Analysis Problems with Open-Ended Solution Spaces: Testing and Evaluation of Systems

Dr. Gary L. KleinThe MIT RE Corp o ra t i on’s Wa s h i n g t on C3 Center, M c Le a n , Vi r g i n i a

Dr. Inderjeet ManiThe MIT RE Corp o ra t i on , R e s t on , Vi r g i n i a

The dire importance of i m p roving the ability to make decisions and predictions in open-ended circ u m-s tances was brought home by the disaster Septe m b er 11,2 0 0 1 .L o oking bac k ,n u m erous factors have sincebeen identified that appear to be indica t ive of the events that were to come. It is possible that these clueso riginally were missed due to fau l ty processes or constraints on re s o u rc e s .L au d a ble ef f o rts have been mad eto fix the processes and to ac q u i re new sy s tems to better support such open-ended decision-making.Ho wever, testing and evaluating these new sy s tems is difficult because of two ch a rac teristics that def i n eopen-ended decision situ a t i o n s : (1) non-uniqueness of s i tuation models that theref o re yields non-uniquep o s s i ble predictions from indica t o r s ; and (2) subjectiv i ty in evaluating indica t o r s . This pap er ex a m i n e sthese pro blems and provides suggestions on how previously used pro c e d u res applied to evaluation of o t h ersy s tems with open-ended ch a rac teristics could be ad ap ted to intel l i gence analysis sy s te m s .

continued…

New Range Capabilities for Chem-Bio Test & Evaluation:The Contamination Avoidance Detector Suite (CADTS)

Lorraine C. CastilloU . S . A rmy Pro g ram Exe c u t i ve Office—Si mu l a t i on , Training and Instru m e n t a t i on

( PEO STRI ) , O rl a n d o, Fl o ri d aEddie M. Meadows

AEgis Te chnologies Gro u p, I n c o rp o ra t e d , O rl a n d o, Fl o ri d aDr. Arthur R. Maret

A S T, I n c o rp o ra t e d , O rl a n d o, Fl o ri d a

S everal ad vanced ch e m i cal and biological (CB) detector sy s tems are being developed for the U. S .Armed Serv i c e s . C u rrent ra n ge test equipment and methodologies are inad e q u a te for the com-p l e te evaluation of these emerging detector sy s te m s ;t h eref o re, i m p ro vements are needed ac ross thee n t i re test spectru m , ranging from agent-simulation correlation studies and equipment upgrad e s ,to field testing te ch n i q u e s . The Contamination Avoidance Detector Test Suite (CA DTS) pro j e c tis funded by the Central Test and Evaluation Investment Pro gram (CTEIP) under the au s p i c e so f the Dire c t o r, O p erational Test and Evaluation (DOT & E ) . This pap er discusses the te ch n i ca lissues invo l ved in testing CB detectors and provides an overv i ew of the CA DTS pro j e c t .

U.S. Army Operational Testing S u p p o rts Homeland Security

D r. M i ch ael T. R o b i n s o nU.S. Army Operational Test Command, Fort Hood, Texas

The creation of the Department of Ho m eland Securi ty (DHS) is the most significant tra n s f o r-mation of the U. S .G o vernment since 1947, when President Ha rry S Truman mer ged the va r-ious bra n ches of the U. S . Armed Fo rces into the Department of D efense to better coord i n a te then a t i o n’s defense against milita ry thre a t s . The U. S . Army stands poised to provide ac t ive supportto the DHS in numerous ways, e n s u ring the utmost safety and rel i a b i l i ty of equipment ands ervices needed to continue the crucial task of p ro tecting the United Sta tes from thre a t s .

The ITEA Journal of Test and Evaluation, September/October 2003, Volume 24, Number 3, continued

6 ITEA Journal • September/October 2003

Guest Editorial

La b o ra t o ry (TSL) was at the center of U.S. e f f o rt sto develop and deploy technologies crucial todetecting and preventing acts oft e r ro ri s m .

The Tra n s p o rt a t i on Se c u ri tyAd m i n i s t ra t i on’s labora t o ry ismeeting that ch a l lenge withn ew or improved equipm e n tthat in a few short years willd ra m a t i ca lly change howt h reats are detected.

I n t e g rating systems will bethe key. As scientists and otherre s e a rchers at TSL pursue specif-ic pro j e c t s , t h ey are eve r - c on-scious of the need to fuse tech-n o l o g i e s , s u ch as tying togetherpassenger and baggage inform a-t i on , for example. As tech n o l o gym a t u re s , a bro a dly integra t e ds ystem will exist that cuts acro s sa ll modes of tra n s p o rt a t i on ,i n cluding sea, rail and surf a c e, a sw e ll as air.

A huge job is ahead, one re q u i ring complex part-nerships throughout private industry and gove rn-ment at eve ry leve l . Producing cutting-edge re s e a rchand deve l o pment must become a con s t a n t .Som e t i m e s - s k e t chy intelligence must be analyze dw e ll and reacted to swift ly. TSL must be eve r - c o g-n i zant of the human factor, designing tech n o l o g i e sthat fit people and reduce the chance of erro r.

Fi n a lly, TSL must be sure that what works wellwithin the 27,000-square-foot labora t o ry near

Atlantic City also works—or can be made to work —e q u a lly well in the field. That is why it is so impor-tant to com mu n i cate before, d u ring and aft e rre s e a rch and deve l o pm e n t .

Last ye a r, s e c u ri ty was substantially strengthened atm o re than 400 airp o rt s , i n cluding the screening of

eve ry passenger and bag, p l a c i n gthousands of fe d e ral air marshalson flights and hardening cock p i td o o r s . Some 5,300 explosive tra c ed e t e c t i on machines are now beingu s e d , as will some 1,060 explosived e t e c t i on systems (EDS) in thenear future .

That alone was a tre m e n d o u sa ch i eve m e n t , d one under tightc on g re s s i onal deadl i n e s . TSL wasin the fore f ront of testing,i m p roving and cert i f ying equip-m e n t , as well as in guiding T S Afield staff, a i rp o rts and air ca r ri e r sas equipment was install e d .

But that ch a llenge pales inc om p a ri s on to the litera lly thou-sands of threat scenarios that ca nbe con j u red up when all forms oft ra n s p o rt a t i on are con s i d e re d .

Ac ross all modes of tra n s-p o rt a t i on , TSA is developing consistent securi ty stan-d a rd s , i n cluding risk assessments, to identify vulnera-bilities and to aid in making the right changes thefirst time aro u n d .

The role of TSL includes fielding a bigger tool boxof securi ty equipment that will all ow varied entities—public or pri v a t e, large or small , f rom ports to ship-ping companies to bus lines and freight forw a rd e r s —to implement the equipment or processes that bestp rotect their employe e s , c u s t omers and pro p e rty.

The Tr a n s p o rtation Security Administration:A d d ressing U.S. Homeland Security

ADM James M. Loy, USCG (Ret.)Tra n s p o rt a t i on Se c u ri ty Ad m i n i s t ra t i on ,U . S . D e p a rtment of Homeland Se c u ri ty,

Wa s h i n g t on , D. C .

ra n s p o rt a t i on securi ty was tra n s f o rmed bySeptember 11, 2001 (“9-11”), f rom a re a c-t i ve appro a ch to a highly pro - a c t i ve on e .Su d d e n ly, the Tra n s p o rt a t i on Se c u ri tyT

ADM James M. Loy, USCG (Ret.)

Guest Editorial

The labora t o ry is continuing to build on itse x p e rtise in explosives detection by deve l o p i n gfaster machines that can find smaller amounts ofe x p l o s i ves and better identify new thre a t s . Wi t han urgent need to probe sealed containers fort h reat materi a l s , the labora t o ry is exploring newa p p l i ca t i ons of X-ray, e l e c t ro-magnetic andn u clear tech n o l o g i e s . And substantial pro g ress hasbeen made on two trace detection portals thata n a lyze air for explosives when a passenger passest h rough them.

To protect against the kind of explosion thatd e s t royed Pan Am Flight 103 in 1988 over Lo ck e rb i e,S c o t l a n d , the labora t o ry developed hardened luggagec on t a i n e r s , ca lled unit load dev i c e s , that typ i ca lly holdup to 50 pieces of luggage and are loaded onto wide-body airc ra ft . The labora t o ry is looking at ways toreduce weight and cost.

The T h reat Image Pro j e c t i on Libra ry all ows fic-titious images of weapons or bombs to be loaded onan EDS or portal X-ray machine and mixed with theimages of real bags moving thro u g h . This is a gre a tt raining tool for scre e n e r s .

C l e a rly, those pro j e c t s , just to name a few, s h owthat the labora t o ry is staying focused on yi e l d i n gp roducts that answer specific needs. That is impera-t i ve .T h e re are simply not enough re s o u rces availableto do otherw i s e .

At the same time, TSL must cl o s e ly coordinate itsp rojects with those of labora t o ries at the Depart m e n tof Defense and within the Department of Hom e l a n dSe c u ri ty. And it must smart ly balance its re s e a rche f f o rt s , not on ly meeting securi ty needs that areo bvious today, but also having the foresight to add amix of projects that will come to timely fru i t i on inf i ve to 10 ye a r s .

Time is our enemy if we all ow ourselves to becom ec om p l a c e n t . Although labora t o ries are places forthought and cre a t i v i ty, to stay ahead of the terro ri s t s ,we must have a sense of urgency and be quick andf l e x i b l e . A fter the Shoe Bomber incident, TSL wasq u i ck ly able to duplicate the threat and provide a re c-ommended course of action .

We also must be flexible in cra fting part n e r s h i p s ,w h i ch will become even more important as TSA andthe Department of Homeland Se c u ri ty incre a s i n g lyd evote re s o u rces across all forms of tra n s p o rt a t i onand re a ch out to foreign gove rn m e n t s .

TSL already has intern a t i onal partnerships withm o re than a dozen foreign gove rn m e n t s . That makes

the labora t o ry’s work more com p l e x , b e cause eachc o u n t ry’s unique agenda helps shape its re s e a rch pri-o ri t i e s . But building partnerships also creates oppor-tunities for learning from each other, for shari n gi n f o rm a t i on and for coopera t i ve ly fielding new tech-n o l o g i e s .

T S L’s intern a t i onal re p u t a t i on was establishedt h rough the deve l o pment of standards and testmethodologies for ED S s . Ob j e c t i v i ty and a lack of biasb e came the bywords of T S L’s cert i f i ca t i on pro c e s s . J u s tc onsider that, a fter 9-11, q u a l i f i ca t i on testing was don eon nine walk-through metal detectors seeking an“e n h a n c e d” s t a t u s . O n ly three eve n t u a lly passed, b u tt h e re were no howls of pro t e s t , b e cause the standard sw e re unquestion a b ly fair.

Now the “human factor” is being injected into thec e rt i f i ca t i on process on some equipm e n t , i n cl u d i n gEDS mach i n e s . The basic issue is how well an opera-tor can perf o rm on a particular mach i n e . H e n c e, ye tanother ch a llenge emerges: h ow to test the mach i n eso it cannot be charged that it failed cert i f i ca t i onb e cause of the opera t o r. Should the vendor be all ow e dto pick from a pool of expert screeners? Or should them a n u f a c t u rer provide the operator? If it is the latter,is there a problem in the ve n d o r’s operator seeing thebaggage set and being able to game the system onf u t u re tests? These are difficult, p ra c t i cal issues withi m p o rtant ra m i f i ca t i ons for securi ty and for the eco-n omic health of vendors who TSL would like to sees u c c e e d .

T h e re is also what is ca lled “a ll o ca t i on of func-t i on , ” a term that has spawned a debate between thosewho believe machine operators are the weakest link inthe securi ty process and want highly autom a t e dm a ch i n e s , and those who warn against being ove rlyreliant on tech n o l o gy.

That debate will con t ribute to finding the ri g h ta ll o ca t i on . M e a nw h i l e, t h e re should be no questionabout making machines more opera t o r - f ri e n dly.O p e ra t o r s — s c reeners in the case of T S A — a re avital part of how well the organiza t i on delivers secu-ri ty. T h ey will do a better job if machines aredesigned to reduce re p e t i t i ve lifting and are notp rone to bag jamming or other problems that ca ncause injuri e s .

TSL is interested not on ly in a mach i n e’st h roughput and fre q u e n cy of false positive s , but alsoin how well it does when operated by a scre e n e r.Vendors who tack on ball o ons and whistles that donot con t ribute to enhanced securi ty are just adding

ITEA Journal • September/October 2003 7

8 ITEA Journal • September/October 2003

Guest Editorial

l evels of com p l e x i ty that make scre e n e r s ’ jobs moredifficult and tra velers less safe .

D i f fe rent makes of EDS machines need to havem o re com m on a l i ty so that, when a screener switch e sf rom one to another, the possibility of error isre d u c e d . For a stra i g h t f o rw a rd example, think for aminute about being a screener on a machine that hasi nverse imaging and then going to a machine withouti t . Not adjusting quick ly could result in a tragic mis-t a k e .

It is appro p riate to end on a human note beca u s e,when all is said and don e, that is what this war is alla b o u t . H ow secure Am e ri cans actually are willdepend on many factors, not the least of which ish ow securi ty - c onscious each of us becom e s . But fewof us, as individuals, a re able to make securi ty inno-v a t i on s . And that is what makes the people at T S Lso important to the safe ty of us all . ❏

On March 1, 2 0 0 3 , ADM James M. L oy, U S CG( R e t . ) , b e came the first ad m i n i s trator for theTra n s p o rtation Securi ty Administration (T S A ) , the newf e d eral age n cy ch a r ged with providing securi ty for theU. S . tra n s p o rtation sy s te m . On that day, TSA move d

f rom the Department of Tra n s p o rtation to the newly cre-a ted Department of Ho m eland Securi ty. On No ve m b er2 2 , 2 0 0 2 , C o n gress confirmed ADM Loy as UnderS e c re ta ry of Tra n s p o rtation for Securi ty, within theD e p a rtment of Tra n s p o rta t i o n .

ADM Loy re t i red from the Coast Gu a rd as comman-dant in May 2002. S erving as commandant from 1998to 2002, he focused his lead ership on re s t o ring re ad i n e s sand shaping the serv i c e ’s futu re. Prev i o u s l y, he served asthe Coast Gu a rd ch i ef o f s ta f f ; c o m m a n d er of the CoastGu a rd’s Atlantic Are a ; ch i ef, Office of Per s o n n el andTra i n i n g ; and commander, Eighth Coast Gu a rd Distri c t .A ca re er seagoing officer, ADM Loy served tours aboardsix Coast Gu a rd cutter s , including command of a patro lboat in combat during the Vietnam Wa r, and command ofmajor cutters in both the Atlantic and Pacific Oceans. Hegrad u a ted from the U. S . Coast Gu a rd Academy in 1964and holds master’s degrees from both We s l eyan Univer s i tyand the Univer s i ty of Rhode Island. In 2003, he re c e ive dthe Ho n o ra ry Degree in Science from the Webb Institu te.He also attended the Industrial College of the Arm e dFo rces and interned at the John F. Kennedy School ofG o vernment at Ha rva rd Univer s i ty. He holds numero u scommendations and other unit and campaign aw a rd s .