GAO Technology Division - Government Accountability Office

United States General Accounting Office

GAO Information Management andTechnology Division

December 1992

InformationTechnology: An AuditGuide for AssessingAcquisition Risks

GAO/IMTEC-8.1.4

Preface

The federal government depends heavily on a varietyof information technology products and services toserve the public. Each year the government spendsbillions of dollars on computer equipment, services,software, and telecommunications. The success orfailure of information system acquisitions affectsexecutive agencies’ credibility with the Congress andthe public as well as their abilities to carry out theirmissions effectively and efficiently.

The General Accounting Office (GAO) and offices ofinspectors general have consistently identifiedproblems with information technology acquisitions.Problems identified in numerous evaluations includeinformation systems that do not meet users’ needs,exceed cost estimates, or take significantly longerthan expected to complete.

This guide provides a logical framework forevaluating information technology acquisitions. Itincorporates a risk assessment methodology intendedto reduce audit planning time and ensure thatsignificant issues are included. It is based on a modelof the acquisition process developed by GAO incooperation with a wide range of federal and privatesector officials. 1 The model outlines the process usedto acquire information technologies and identifieselements of the process that are essential for planningand carrying out acquisitions.

This guide is intended for use in planning andconducting risk assessments of computer hardwareand software, telecommunications, and systemdevelopment acquisitions. A risk assessment is theprocess of identifying potential risks in a systemunder development and then determining thesignificance of each risk in terms of its likelihood andimpact on the acquisition’s cost, schedule, and ability

1Information Technology: A Model to Help Managers DecreaseAcquisition Risks (GAO/IMTEC-8.1.6, August 1990).

GAO/IMTEC-8.1.4 Assessing Acquisition RisksPage 1

Preface

to meet the agency’s needs. Such assessments mayhave their greatest impact if carried out early, whenan agency can more easily alter its acquisition plansand strategy to manage and control the identifiedrisks.

The audit guide consists of 10 chapters, withappendixes. Chapter 1 introduces the purpose of theguide, explains its essential concepts and techniques,and provides direction in tailoring the guide for useon specific assignments. Chapters 2 through 10address specific activities in the acquisition process.The nine chapters present audit guidance on:

• management and user support,• project staffing,• needs/requirements/ specifications,• alternatives,• acquisition planning,• solicitation document,• source selection,• contract management, and• test and acceptance.

Each chapter lists audit objectives, commonlyexpected documentation, detailed audit questions,and references to federal regulations and guidance.The chapters are intended to assist in theidentification of specific risk areas and to contributeto an overall assessment of how well an agency ismeeting its acquisition objectives.

This audit guide is also available in a software format,accompanied by reference materials such as the GAOmodel of the acquisition process, relevant federalacquisition regulations, General ServicesAdministration (GSA) guidance, and Office ofManagement and Budget (OMB) circulars. Thesoftware version utilizes a hypertext softwarepackage to help auditors quickly and flexibly review


Preface

all the included documents. This software may berequested from GAO by returning the card includedwith this guide.

This guide supplements and does not replace otherGAO policies or procedures. It was prepared underthe direction of Jack L. Brock, Jr., Director,Government Information and Financial Management,who can be reached at (202) 512-6406. Other majorcontributors to the guide are listed in appendix IV.

Ralph V. CarloneAssistant Comptroller GeneralInformation Management and Technology Division

Werner GrosshansAssistant Comptroller General for Policy


Contents

Preface 1

Chapter 1 Introduction

8Objectives 8Approach 9Assignment Planning 9Organization and Use of the Audit Guide 10Application of Audit Guide to Prototyping

Methodology13

Presenting Conclusions andRecommendations

14

Chapter 2 Management andUser Support

16Audit Objectives 17Documentation Required 17Audit Steps: Top Management Support 17Audit Steps: User Involvement 19References 21

Chapter 3 Project Staffing

22Audit Objective 22Documentation Required 22Audit Steps: Project Management 22Audit Steps: Project Staff 23References 24

Chapter 4 Needs / Requirements / Specifications

25Audit Objectives 26Documentation Required 26Audit Steps: Needs Determination 26Audit Steps: Requirements Analysis 27Audit Steps: Specifications 30Audit Steps: Test Plans 32References 32


Contents

Chapter 5 Alternatives

34Audit Objectives 34Documentation Required 34Audit Steps 35References 39

Chapter 6 AcquisitionPlanning

40Audit Objective 40Documentation Required 40Audit Steps 41References 43

Chapter 7 SolicitationDocument


Chapter 8 Source Selection

50Audit Objective 50Documentation Required 51Audit Steps 51References 54

Chapter 9 ContractManagement


Chapter 10 Test andAcceptance


Appendixes Appendix I: Acquisition Profile 64


Contents

Appendix II: Management Metrics 68Appendix III: Reference Materials 80Appendix IV: Major Contributors to This

Audit Guide86

Glossary 87

Table Table I.1: Map to the Audit Guide 11

Figures Figure II.1: Problem Reports in SoftwareProject

73

Figure II.2: Problems By Priority Levels andNumber of Months Unresolved

74

Figure II.3: Software Requirements Changes 75Figure II.4: Development Progress 76Figure II.5: Software Size Estimates 78


Contents

Abbreviations

ANSI American National Standards InstituteAPR agency procurement requestCO Contracting OfficerCOCOMO Constructive Cost ModelCOTR Contracting Officer Technical

RepresentativeDPA delegation of procurement authorityFAR Federal Acquisition RegulationFIPS Federal Information Processing

StandardsFIRMR Federal Information Resource

Management RegulationGAO General Accounting OfficeGSA General Services AdministrationIEEE Institute for Electrical and Electronic

EngineersIFB invitation for bidIMTEC Information Management and

Technology DivisionNIST National Institute of Standards and

TechnologyOMB Office of Management and BudgetPM program managerRFP request for proposalsSSA source selection authoritySSAC Source Selection Advisory CommitteeSSEB Source Selection Evaluation Board


Chapter 1

Introduction

This audit guide is based on and incorporates GAO’sinformation technology acquisition model. The modeldescribes three phases in the acquisition process:presolicitation, solicitation and award, andpostaward. It sets out essential activities in eachphase along with critical factors related to thoseactivities. The model is intended to give managers anoverview of the acquisition process and to help themdecrease acquisition risks.

The model’s critical factors were drawn from thejudgment and expertise of a wide range ofknowledgeable individuals from government andprivate industry. Compliance with these criticalfactors can increase the likelihood that an acquisitionwill meet an agency’s needs at a reasonable cost andin a timely manner.

Objectives This guide is intended to help auditors conduct morefocused reviews of information technologyacquisitions by enabling them to quickly identifysignificant areas of risk. Using this guide will helpauditors identify critical factors not addressed bymanagement, make a general assessment of anyprocurement risks, and provide rapid feedback toagency officials so they can take corrective action in atimely and efficient manner. Use of the guide shouldbe selectively tailored to the requirements ofparticular reviews and adapted to the status of theacquisition.

Auditors will need to exercise professional judgmentin assessing the significance of audit results orfindings. For example, the guide assists auditors indetermining how an agency identified and defined itsrequirements. Professional judgment is necessary toevaluate this information and determine if the agencyconducted an adequate requirements analysis.


Chapter 1

Introduction

Some areas of assessment will require the expertiseof auditors with specific technical skills andexperience. These specific areas include knowledgeof solicitation procedures, benchmarking and otherperformance or capability validation techniques, andknowledge of technical areas such as databasemanagement and telecommunications networks.These areas are identified within the guide. The auditteam should include experienced members withenough knowledge of information technologies tosatisfy the Government Auditing Standardsrequirement that auditors have appropriate skills andknowledge.

Approach The audit approach described in this guide is intendedto result in a risk assessment of an acquisition projectat any point in its development. The auditor will betrying to determine whether a project will result in aspecified product or level of performance and will bedelivered at a specified time and cost. An auditorshould use this guide to identify areas that are mostlikely to result in technical failures, unmet user needs,cost overruns, or schedule delays. Those risks shouldthen be brought to the attention of appropriateagency officials. The audit steps and questionsprovided are directed toward assessing whether anagency has sufficiently addressed critical factors,including support from managers and users, adequateproject staff, and controls over the acquisition’s scopebefore and after a contract is awarded.

AssignmentPlanning

When planning a risk assessment, the auditor shouldfirst review the agency’s acquisition policies anddirectives to identify the organizations and individualsresponsible for approving procurements. Approvalthresholds, for example, should show which officialshave the authority to review and approve acquisitions.


Chapter 1

Introduction

The agency’s directives should also show the specificmilestones and documentation required for aprocurement.

The auditor should also review previous studies oraudits of the acquisition project and of the agency’sinformation resources management functions.Reports by GAO or other auditing institutions canprovide valuable background information. Theauditor should also determine whether previousrecommendations have been carried out.

Organization andUse of the AuditGuide

The chapters of this audit guide focus on logicallydistinct steps of the acquisition process, as describedon page 10 of GAO’s acquisition model. 1 Thefollowing table identifies the appropriate chapters forreviewing the various steps of an acquisition. Ingeneral, the auditor will want to concentrate on thesteps that are relevant to the phase of the acquisitionbeing reviewed. However, regardless of how far theacquisition has advanced, at a minimum the auditorshould always ascertain whether senior managers andusers were involved in the project’s initiation. Theauditor should also verify that the agency has definedits needs and requirements to support its mission, andthat those requirements continue to be valid as theacquisition progresses through contract award andcontract management.

1GAO/IMTEC-8.1.6, August 1990.


Chapter 1

Introduction

Table I.1: Map to theAudit Guide Acquisition Phase Steps in Each Phase Chapter

I: Presolicitation Initiate Project 2, 3

AnalyzeRequirements 4

Identify Alternatives 5

Prepare AcquisitionPlan 6

PrepareSpecifications 4

II. Solicitation andAward

Maintain ProjectStructure 2, 3

Prepare Solicitation 7

Release Solicitation 8

Evaluate Proposals 8

Negotiate withVendors 8

Select Contractor 8

III. Postaward Establish ContractManagement 9

Monitor ContractPerformance 9

Test and AcceptSystem 10

Each of the following chapters provides references toregulations and other guidance relevant to material inthe chapter. In addition, each chapter identifiesspecific audit objectives and documentation expectedfor the major activities at that point in the acquisitionprocess. The documents listed may exist withdifferent names than those used here. The auditorshould refer to agency-specific requirements for moreinformation. Finally, each chapter sets out audit stepsto help plan and conduct the assessment of anacquisition. The questions may need to be tailored to


Chapter 1

Introduction

the particular circumstances of an audit, using thesubject agency’s requirements.

The appendixes provide further information for use inplanning and conducting an acquisition audit.Appendix I contains a worksheet, called anacquisition profile, to summarize essentialinformation about the acquisition project. Theworksheet should contain such information as thenames and locations of units responsible for theacquisition, the project purpose, and the expectedcost and time frames. The completed profile shouldbe kept available for later reference. If a profile existsfrom an earlier assignment, the auditor should reviewand update it.

Appendix II of this guide describes techniques to usein identifying software development risks of delaysand cost overruns, known collectively as managementmetrics or indicators. These techniques requireinformation about the size of the system beingdeveloped and about progress after development hasbegun. Thus, the techniques are only appropriate foruse after the agency has completed its design andprogressed into developing the system. An auditorusing this guide to review an acquisition that includessignificant software development and has a contractin place should review appendix II for techniques tohelp identify variances from the agency’s cost andschedule estimates. In some cases, the project teammay be unable to complete a project on time due toan unrealistic schedule. The cost models described inappendix II can be used to make a rough estimate ofhow long a system development may require. Such anestimate can be compared to agency plans to see ifthe agency has committed itself to unrealisticallyshort time frames. In other cases, delays incompleting scheduled activities, such as systemdesign, coding, and testing, can lead to project


Chapter 1

Introduction

slippage later on. Auditors will need to tailor the useof the tools described in appendix II to thecircumstances of the audit.

Appendix III provides a comprehensive list ofreferences cited throughout the guide, as well aspublications that may be useful for technicalinformation. Finally, a glossary is attached thatdefines a wide range of technical and procurementterminology.

Application ofAudit Guide toPrototypingMethodology

This acquisition guide is intended for use in reviewingany information technology acquisition, regardless ofthe system development methodology being used. Theguide is structured around the federal acquisitionprocess, and is independent of development methodsfor information systems. An auditor should recognizethat there are different system development modelsthat may be used when a system development effort isacquired. Such models may include the “waterfall”model, rapid prototyping, or evolutionarydevelopment. Some of the documents required underdifferent development approaches may differ.Prototyping, for example, may act as part of therequirements definition process, helping the agencyidentify and control areas of high uncertainty andtechnical risk. In this situation the auditor should (1) focus on determining how one or more prototypesor incremental versions function to define theagency’s requirements and (2) determine how thesystem development methodology used by the agencycontrols the prototyping process.

One approach to using prototyping as part of thesystem development process has been described as a“spiral” model of system development. 2 The spiral

2For a description of prototyping and the spiral model, see Roger S.Pressman, Software Engineering: A Practitioner’s Approach, 3rd ed.(New York: McGraw-Hill, Inc., 1992), pp. 26-34.


Chapter 1

Introduction

model portrays a process in which an agencyiteratively (1) determines its objectives, alternatives,and constraints; (2) evaluates its alternatives andidentifies and resolves risk issues; (3) develops andverifies its next-level products; and (4) plans its nextphases. Prototypes are developed or modified as partof the second phase. As part of this model, aprototype is developed or revised whenever a riskanalysis shows that significant areas of uncertaintyremain that pose substantial risks to project success.When the system has been defined well enough tomanage risks effectively, the agency develops andtests a full-scale system.

In order to review an acquisition that is usingprototypes, the auditor should determine whatregulations or guidance the agency has to define aprototyping methodology. This methodology willestablish the documentation and approval points thatagency officials should meet. The auditor can thenmeasure the progress of the prototypes against theagency’s criteria.

PresentingConclusions andRecommendations

The auditor should conclude the review by identifyingoutstanding areas of risk and the agency’s actions toaddress those risks. Note should be made of how wellthe agency has addressed the critical factors in GAO’s1990 model of information technology acquisitions aswell as agency compliance with acquisitionregulations, standards, and other federal guidance. 3

Results of the review should be communicated toagency officials for their comment, in accordancewith government auditing standards.

The auditor should also recommend changes toensure that the agency is properly addressing thecritical factors in the GAO model. For example, the

3GAO/IMTEC-8.1.6, August 1990.


Chapter 1

Introduction

auditor may recommend a greater role for users ifthey are not involved in approving alternatives orvalidating system requirements. Similarly, the auditorshould recommend that the agency take appropriateactions where senior management involvement seemslacking, or where the project organization is unstableand subject to high turnover. The recommendationsshould be reviewed with agency officials and beappropriate to the probability and significance of therisks identified.


Chapter 2

Management and User Support

This chapter focuses on levels of commitment andsupport for a planned acquisition by senior managersand users, two key stakeholders in an acquisition.Senior managers include those who have overallagency responsibility for strategic, includinginformation-related, objectives. Users are those whooperate or rely on agency information resources andinclude the managers and staff responsible for agencypolicies and programs supported by the acquisition.

The involvement and support of senior managementthroughout an acquisition is essential for success.Senior management should envision the agency’sacquisition goals, define strategic objectives, andoversee the projects that implement the overallvision. One or more senior managers should act assystem sponsors, with sufficient authority to ensurethat applicable resources are available for the project.

Users should also be involved and provide supportthroughout the acquisition to ensure that theirrequirements are understood and that the resultingsystem is both accepted and used. User involvementshould be sustained from the needs determinationphase through final acceptance and implementation.User involvement in the acquisition process will helpavoid the development of products that ultimately donot meet agency requirements.

The audit steps in this section should be used toassess the potential risks posed by the lack ofmanagement or user support. An acquisition thatlacks either or both of these elements is at risk ofincurring unnecessary cost overruns, not meeting itsplanned delivery schedule, and not satisfying agencyneeds.


Chapter 2


Audit Objectives 1. To ensure that senior managers support and areactively involved throughout the development andimplementation of an acquisition.

2. To ensure that users support the acquisition byactively participating in defining procurementrequirements, developing the solicitation document,and verifying that the equipment and/or servicescontracted for meet the agency’s needs.

DocumentationRequired

• Decision papers, memoranda, or other records ofsenior management oversight and approval ofacquisition objectives and plans.

• Program management directives or other writtendirectives from senior managers stating goals andobjectives of the acquisition and delegating authorityto carry out the acquisition.

• Budget exhibits and plans showing that sufficientfunding is committed to the acquisition.

• Project plans showing the role of users in planningand overseeing the acquisition. Any documentation ofthe users’ role in validating acquisition requirements,alternatives, and the solicitation document. Users’roles and responsibilities may be detailed in amemorandum of understanding between userorganizations and the program office.

• Agency policies or guidelines on the structure ofsteering committees or other oversight bodies, withresponsibilities of project members and seniormanagers delineated.

Audit Steps: TopManagementSupport

1. Identify senior management officials responsiblefor the acquisition. Include senior program officialsheading the user organizations, information resourcemanagement officials, and members of senioroversight or steering committees. Determine the roles


Chapter 2


and responsibilities of any groups or committees ofsenior managers, and the relationships between them.

2. Review the documentation related to theacquisition and determine if the senior managersidentified in the acquisition profile:

a. Approve the goals and objectives of the acquisition.

b. Designate a program sponsor who is responsibleand accountable for the acquisition.

c. Establish a formal process to keep concernedparties appropriately informed.

d. Participate in the specified reviews and decisions.

• Determine how promptly management reviews areconducted and approvals given and if the approvalsare given at the appropriate levels.

• Review documentation of such reviews, such asdecision memoranda and review-committee minutes.

• Determine the frequency of reviews and the quality ofdirection senior managers give to project personnel.

e. Provide initial funding for the project and establishnear- and long-term funding commitments,periodically informing Congress of acquisitionobjectives and status.

f. Secure any necessary support from key externalorganizations, such as OMB, GSA, and relevantcongressional committees.

g. Assign independent officials to ensure that securityand internal controls needs are met.

3. On the basis of the above audit steps and oncontacts with other project officials, determine


Chapter 2


whether senior managers fostered good workingrelations among the sponsor, acquisition manager(program manager), other top managers, the technicaloffices, and the contracting community. Specifically,determine whether the managers:

a. Coordinate agreement for developing theevaluation and source selection plans and gainacceptance of the evaluation process and criteria.

b. Coordinate an agreement between the programstaff and technical and contracting offices formanaging the contract.

c. Obtain the support of important acquisitionofficials in the department or agency, such as theofficial responsible for source selection.

4. Obtain users’ and project staff’s evaluations ofmanagement’s support in the above steps. Find outhow long it took to get approvals from managementand the direction management gave to projectpersonnel.

Audit Steps: UserInvolvement

1. Identify the population of users from projectdocuments and the agency’s organization charts.Review agency criteria (regulations and procedures)to determine the roles the agency assigns to users.Determine from the program manager and selectedusers which user organizations are actively involvedin the acquisition. Identify significant user groupswho are not involved in the acquisition.

2. Determine if users:

a. Are involved in periodic reviews, and if so, howfrequently they are involved.


Chapter 2


b. Sign off on needs and requirements statements orotherwise validate the requirements andcorresponding solutions when developed. (Formore information see ch. 4, step 3 under NeedsDetermination, step 4 under RequirementsAnalysis, and step 1 under Specifications.)

c. Validate alternatives against original requirements.(See step 1 of ch. 5 for related audit steps.)

d. Approve the alternative selected (such as thechoice between off-the-shelf technologies orcustom development, centralized or distributedprocessing, etc). (See step 1 of ch. 5 for relatedaudit steps.)

e. Validate the specifications against therequirements. (For more information see step 4aunder Specifications, in ch. 4.)

f. Provide acceptance criteria.

g. Assist in preparing the solicitation document andawarding the contract (for example, userrepresentatives may assist in developing evaluationcriteria and participate in the source selection teamevaluating alternative proposals). (See ch. 7, step 4,and ch. 8, step 1d, for related information.)

h. Participate in postaward activities that may includeinstallation, test, and acceptance of equipment.

i. Participate in reviews of contractor-prepareddeliverable documents such as designspecifications, system analyses, and user andtraining manuals.

j. Participate in government/contractor workinggroups.


Chapter 2


k. Participate in the postaward audit for assessing thedegree of success of the acquisition.

3. Determine if users allocate staff time and otherresources to the project.

4. Determine if the users participating in the programhave a high, moderate, or low turnover rate.

5. Assess the adequacy of users’ funding support tothe project.

a. Obtain funding commitment from users.

b. Identify appropriations to be used and theiravailability to support contract award.

References • Federal Information Resource ManagementRegulation (FIRMR), Part 201-2: Designated SeniorOfficials.

• GAO, Information Technology: A Model to HelpManagers Decrease Acquisition Risks(GAO/IMTEC-8.1.6, August 1990), Phase I, steps 2 and3; Phase II, step 1; Phase III, step 1.


Chapter 3

Project Staffing

Project management for an acquisition isaccomplished primarily by a program manager andstaff responsible for carrying out project activities.The program manager should have sufficientauthority and an appropriate mix of skills andexperience to successfully manage the project.

The acquisition project staff should be assigned clearroles and responsibilities. The team should includemembers who are skilled in the informationtechnology procurement process, understand thetechnology, and have experience in managingcontracts. The team should also have membersknowledgeable about the programs that theacquisition is to support.

Audit Objective To determine if the acquisition team has thenecessary skills and authority to effectively plan andexecute the acquisition.


• A list of key project team members showing theirresponsibilities, job titles, and experience. Theauditor may have to generate this list on the basis ofinterviews if one is not available.

• The agency procurement request for a delegation ofprocurement authority from GSA, showing names andexperience of senior project officials (as required byGSA guidance detailed in its FIRMR Bulletin C-5).

Audit Steps:ProjectManagement

1. Review the agency’s criteria for acquisition todetermine the responsibility and accountability of theprogram manager and his/her required qualifications.

2. Review the documentation related to theacquisition to determine how clearly the program


Chapter 3

Project Staffing

manager’s responsibility and accountability aredefined. Determine if the program manager has:

a. A charter to establish authority, responsibility, and accountability.

b. A clearly defined relationship with the program, technical, and procurement offices.

c. A clearly defined relationship with the sponsor and users.

d. Access to senior agency officials.

e. The authority to manage acquisition funds.

f. Input to the budgetary process.

3. Review the qualifications of the program managerto determine if the program manager has theappropriate mix of skills and experience. Has theprogram manager directed other projects of similarsize and complexity? Is he or she trained to managecomplex acquisitions (GSA’s Trail Boss program maybe an example of such training).

4. Review management continuity on the acquisitionas shown by the turnover rate of program managers.

Audit Steps:Project Staff

1. Review the agency’s acquisition criteria todetermine both the responsibility and accountabilityof project personnel as well as their requiredqualifications.

2. Review the make-up of the project team to ensurethat a mix of appropriate acquisition skills arerepresented. (See ch. 9, steps 2 and 3, for related auditsteps.)


Chapter 3

Project Staffing

a. Determine the authority and experience level of theContracting Officer’s Technical Representative.

b. Identify key project staff and review theirexperience and qualifications. Determine whetherthe Contracting Officer is trained and experiencedin information technology acquisitions.

c. Determine if the project staff is experienced inmanaging contractors.

d. Determine the extent of turnover within the projectstaff.

3. Determine if the agency trains project staff tomaintain their skills and qualifications.

4. Review the reasonableness of project milestonesand schedule with project team members.

References • OMB Circular A-109: Major System Acquisitions.• GAO, Information Technology: A Model to Help

Managers Decrease Acquisition Risks(GAO/IMTEC-8.1.6, August 1990), Phase I, step 5...i

• GSA, Overview Guide, pp. 2-3 to 2-7.• GSA, Guide for Contracting Officers’ Technical

Representatives, Chapter 2.• FIRMR Bulletin C-5: Delegation of Procurement

Authority for a Specific Acquisition.


Chapter 4

Needs / Requirements / Specifications

The purpose of this chapter is to guide the auditor indetermining whether the agency has developed anaccurate description of its information technologyneeds. The acquisition should be clearly linked toprogram needs, to the agency’s overall strategies, andto governmentwide policies and standards. Theagency should expand on its basic description ofneeds to define specific requirements so providers ofinformation technology can respond with meaningfulsolutions. In some cases, an agency may useprototypes to help define or validate its requirements.The requirements then form the basis for even moredetailed specifications.

In identifying its requirements, an agency should planfor testing the information resources it needs. Theseplans should cover acceptance, security, andcertification requirements. The test plans developedat this point form the basis for later evaluations ofcontract performance.

Failure to clearly and accurately define informationtechnology requirements poses high risks for anyagency. For instance, improperly definedrequirements may preclude alternatives, restrictcompetition, increase the risk of cost and scheduleoverruns, and lead to systems that are inconsistentwith an agency’s overall architecture andincompatible with other agency systems. Thehardware or software purchased may also beinconsistent with government standards. Designingand implementing a system is also more difficult ifinput, output, and processing specifications areincomplete or inaccurate. In addition, if security andinternal control requirements are not well defined,control over sensitive information or other assets maybe lost.


Chapter 4

Needs /

Requirements /

Specifications

Audit Objectives 1. To ensure that the acquisition is based on clearlyunderstood needs or opportunities and that it isconsistent with the overall strategy and architecturesused by the agency.

2. To ensure that the agency defines its requirements,based on the needs identified earlier and validated byfunctional users, well enough to support theacquisition of hardware, software,telecommunications, and system developmentservices. These requirements should primarily beexpressed in functional terms in accordance withFIRMR policy.

3. To ensure that system specifications clearly andaccurately summarize the agency’s requirements.


• Needs statement.• Requirements analysis or functional requirements

document.• System specifications, if prepared. Also, draft

specifications with industry comments if draftspecifications were released.

• Test plan and requirements prepared before contractaward. Test requirements may be summarized in atest and evaluation master plan.

Audit Steps:NeedsDetermination

1. Review the agency’s stated needs, which may bedocumented in a Mission Element Needs Statement,Statement of Operational Need, or SystemOperational Concept. Determine whether the needsstatement clearly and accurately reflects the users’needs as indicated in the mission statement andstrategic objectives of the users’ organization, thestrategic information plan, or the computer securityplan.


Chapter 4

Needs /

Requirements /

Specifications

2. Check the needs statement for:

a. Existing system architecture and functions to besupported (i.e., description, cost, volume of work,projected growth).

b. Justification for changes, such as correctingdeficiencies in existing capabilities, complying withnew or changed program requirements, or takingadvantage of opportunities for increased economyand efficiency.

3. Contact several users as well as project staff whoare not users to determine if they generally agree thatthe needs analysis presented in the needs statementadequately addresses actual problems. (See ch. 2 forrelated questions.)

Audit Steps:RequirementsAnalysis

1. Review the requirements analysis to determine if itdescribes the current system. This description shouldinclude all the functions of the existing system thatany new system will have to perform. The users,functions, work load, operating costs, andcomponents of the current system should also beidentified.

2. Confirm that the agency has defined its informationrequirements for the new information resources.These requirements include:

a. Information now being received or information thatis needed but that is not being received.

b. Information to be provided to or obtained fromother agencies or the public.

c. Sources available from which to obtain the neededinformation.


Chapter 4

Needs /

Requirements /

Specifications

d. Information relationships.

e. The degree of information validation, integrity, accuracy, completeness, and reliability.

f. The quantity of information to be processed and types of output expected.

g. The timeliness and format of the information.

h. The security, accessibility, and privacy requirements.

3. Determine if the agency has defined its functionaland support requirements, including:

a. Present and projected work loads and capacity analysis, including peak load requirements and requirements for future capacity management.

b. Privacy and security requirements.

c. Contingency requirements for resources whose losswould either prevent or significantly impair theagency from performing its mission or would havean adverse impact on the nation.

d. Records management factors relating to integrationof electronic records with other agency records,records retention and disposition, and safeguardsagainst unauthorized use or destruction of records.

e. Space and environment factors, such as floorloading, heat dissipation, and power supply.

f. Federal standards with which the new technologymust comply.

g. Organizational training needs.


Chapter 4

Needs /

Requirements /

Specifications

h. Interfaces with other systems.

i. User interface requirements.

j. Compatibility limitation requirements.

k. Capability or performance validation requirements.

4. Determine whether the requirements analysisprovides for:

a. A methodology for having users validate therequirements analysis for both capability andperformance. (See ch. 2 for related questions).

b. Measurable requirements that may be used later toverify system effectiveness.

5. Determine if the requirements are presented infunctional or performance terms in consideration offull and open competition. Functional requirementspromote full and open competition, whileperformance requirements may not.

6. With regard to restrictive requirements (which donot lead to full and open competition), determine:

a. If brand-name-or-equal or specific make and modelrestrictions are appropriately justified.

b. Whether all required justifications are completed and approved for other compatibility-limited requirements.

(See ch. 6, step 3, for more information onprocurements that do not promote full and opencompetition).


Chapter 4

Needs /

Requirements /

Specifications

7. With regard to the process of revising requirementsduring the acquisition, determine:

a. Whether a core of basic requirements has beenidentified in order to maintain project scope.

b. If a formal change control process has beenestablished to manage changes when necessitatedby a changing environment.

c. Who is responsible for reviewing and approvingchanges to requirements.

d. How often requirements have been changed.

e. Whether proposed new requirements are validatedagainst mission needs.

f. What process is used to analyze the impacts ofchanges on the other elements of the requirements.

Audit Steps:Specifications

1. Determine whether functional users and/or theprogram manager confirmed that the specificationsaccurately reflect the requirements and conform tothe approved acquisition strategy discussed in theacquisition strategy module. Did users or the programmanager sign off on the system specifications? (Seech. 2 for related questions.)

2. Examine the specifications document for:

a. A summary of the functional requirements to besatisfied by the technology.

b. Performance evaluation requirements.

c. Performance requirements that addressinformation accuracy; data integrity requirements;


Chapter 4

Needs /

Requirements /

Specifications

timing for response, update processing, informationtransfer, transmission, and throughput; and flexibilityto changes in the requirements.

d. An identification of new types of equipmentrequired (e.g., processors, input/output devices, orinformation transmission devices).

e. An identification of support and test software.

f. A description of interfaces.

g. A description of overall security and privacyrequirements.

h. A description of the operational controls needed.

i. A description of the operating characteristics of theuser and computer centers where the software will beused.

j. A description of the logic flow of the entire system.

k. A specification of the functions to be satisfied bythe software.

3. Determine how restrictions to full and opencompetition in the specifications (e.g., equipmentcharacteristics and performance elements) arehandled. Ensure that all required justifications arecompleted and properly approved for suchrestrictions. (See ch. 6, step 3, for more informationon procurements that do not promote full and opencompetition.)

4. Determine how changes to the originalspecifications are handled.


Chapter 4

Needs /

Requirements /

Specifications

a. Establish responsibility for reviewing and approving changes to specifications. Interview users to determine whether or not they actually reviewed and approved changes to specifications. (See ch. 2, step 2e, under User Involvement.)

b. Review documentation on change requests.Determine how often specifications are changed,whether new specifications are validated againstrequirements, and what process is used to identifyimpacts of changes on other elements ofspecifications.

5. Determine if feedback (comments and questions)from users and industry is accounted for, considered,and incorporated as appropriate on a continual basis.(See ch. 7, step 6, for related information.)

Audit Steps: TestPlans

1. Determine if the agency has developed test plansbased on acceptance criteria furnished or validatedby the users.

2. Verify that test plans incorporate security andcertification requirements.

3. Determine if test plans adequately measure systemperformance requirements to be specified in therequest for proposals (RFP).

(Note: Refer to ch. 10 for more information on testplans.)

References • FIRMR 201-20.1: Requirements Analysis.• FIRMR 201-20.303: Standards.• Federal Acquisition Regulation (FAR) Part 6:

Competition Requirements.


Chapter 4

Needs /

Requirements /

Specifications

• FAR Part 10: Specifications, Standards, and OtherPurchase Descriptions.

• GSA, Guide for Requirements Analysis and Analysisof Alternatives, Chapter 2: Requirements Analysis.

• American National Standards Institute/Institute forElectrical and Electronic Engineers (ANSI/IEEE)Standard 830: IEEE Guide to Software RequirementsSpecifications.

• GAO, Information Technology: A Model to HelpManagers Decrease Acquisition Risks(GAO/IMTEC-8.1.6, August 1990), Phase I, steps 1, 6,11, 12, 13, 14.

• Federal Information Processing Standards (FIPS)Publication 64: Guidelines for Documentation ofComputer Programs and Automated Data Systems forthe Initiation Phase.

• FIPS Publication 101: Guideline for LifecycleValidation, Verification, and Testing of ComputerSoftware.


Chapter 5

Alternatives

After identifying its requirements, the agency shouldassess alternatives for cost-effectively meeting thoserequirements. The approach selected should reflectan understanding of what is available in thecommercial market as well as what is available withinthe government. Approaching the acquisition this waywill lessen, but not eliminate, the risk that an agencymay select an alternative that does not fully meet userrequirements or that is unnecessarily complex andexpensive.

Audit Objectives 1. To determine if the agency has considered allreasonable alternatives for meeting its needs.

2. To determine if the agency identified the risks,costs, and benefits of each alternative.

3. To verify that the agency selected an alternativebalancing expected benefits against costs, time, andrisks of failure.


• Record of alternatives analysis, such as a systemdecision paper. Economic and risk analyses shouldaccompany or be a part of the decision paper.

• Market survey research conducted to identifyalternatives for meeting user needs and to supportcost estimates.

• Findings and approvals statements to supportrestrictions on specifications, such ascompatibility-limited requirements.

• Cost/benefit analysis to justify the selection of thealternative selected over other alternatives, in dollarterms or in terms of some other criteria, such aseffectiveness.


Chapter 5

Alternatives

Audit Steps 1. Assess the involvement of responsible parties andverify whether:

a. Users agreed with the range of alternativesconsidered and were involved in validating thosealternatives against the original requirements. (Forrelated information on this and the next point seech. 2, steps 2c and 2d, under User Involvement.)

b. Users agreed with the alternative finally selected.

c. Appropriate senior management approved thealternative selected. (See ch. 2, step 2, underTop-management Support, for more information.)

d. The Contracting Officer or other contracting personnel participated in the alternatives analysis to ensure that a feasible acquisition approach was selected.

e. Project staff conducted market surveys to determine how industry can best meet the agency’s requirements.

2. Ensure that the agency considered, as appropriate,the alternatives included in GAO’s acquisition modeland FIRMR 201-20.203-1.

3. Assess how the agency evaluated alternatives bydetermining whether:

a. The agency consistently analyzed alternatives usingthe same criteria for each alternative.

b. The alternatives are described in sufficient detail to support time and cost estimates and cost/benefit analyses.


Chapter 5

Alternatives

c. The alternatives considered fit within the agency’s information architecture.

d. The range of alternatives considered was restricted by resource assumptions (staff or funding limitations).

4. Determine whether the agency consistentlyanalyzed the costs and benefits of each alternative.Ensure that the economic analysis includes presentvalues for costs and benefits and is updatedperiodically. Identify the system life used as a basisfor evaluating alternatives and determine whether itappears realistic in light of user needs, expectedchanges in the technology, expected availability ofmaintenance and other support, and the time neededto prepare subsequent acquisitions. Verify that theeconomic analysis includes a sensitivity analysis toidentify factors that affect the choice of onealternative over another. The costs and benefitsconsidered for each alternative should include:

a. Conversion costs.

b. Personnel costs.

c. Operation and maintenance costs.

d. Nonrecurring but quantifiable benefits in terms of information processing, administration, and support (these may include cost reductions resulting from improved system operations or value enhancement through improved use of resources).

e. Recurring and quantifiable benefits on a monthly and/or quarterly basis over the system life from reductions in such items as salaries, fringe benefits, supplies, utilities, and space occupancy.


Chapter 5

Alternatives

f. Any nonquantifiable benefits, such as improved service and enhanced organizational image.

5. Determine whether the agency analyzed thefollowing noncost factors for each feasiblealternative:

a. Obsolescence: strategies for avoiding outdatedresources over the system life. A technologyupgrade clause is one way to avoid obsolescence byallowing an agency to buy advanced versions ofequipment or software when they become available.

b. Availability: to what extent the system will beavailable to users.

c. Reliability: how frequently the system requirescorrective maintenance.

d. Maintainability: the ease with which failed systemcomponents can be repaired, taking into accountthe level of service, personnel support, andsupplies needed.

e. Expandability: the ease with which the system canbe enhanced to meet anticipated growth.

f. Flexibility: the extent to which the alternative can accommodate changes in the nature of the work load.

g. Security: the ability to prevent unauthorized access and tampering and consideration of national security and emergency preparations.

h. Privacy: the extent to which the privacy ofpersonnel-related data can be maintained.


Chapter 5

Alternatives

i. Affect on personnel: the impact on the level of support personnel needed, including the skills required.

j. User acceptance: the overall impact on the user community, including the amount of change to user procedures.

k. Accountability: the ability of the alternative toallow system activity to be tracked and measured.

6. Review the risk analysis to see if it identifiessensitive data and vulnerabilities. Verify that themagnitude of each vulnerability has been stated.Determine whether or not the risk analysis conformswith the standards identified in FIPS Publications 65and 73 and OMB Circular A-130.

7. Verify that each alternative is evaluated forfinancial, technical, and schedule risks. Financial riskrefers to the extent to which each alternative issubject to unexpected additional costs. Technical riskindicates the probability that each alternative’stechnical objectives will prove difficult to achieve inwhole or in part. Schedule risk is the extent to whicheach alternative is subject to unexpected scheduledelays and slippage in meeting the system’s technicalobjectives, regardless of cost.

8. Ensure that the agency has selected the mostadvantageous and realistic alternative with respect tobenefits, costs, and risks (based on steps 4 and 7).

9. Verify that users and senior managers approve anychanges to the planned scope of the project.


Chapter 5

Alternatives

References • FIRMR Part 201-20.2: Analysis of Alternatives.• GAO, Information Technology: A Model to Help

Managers Decrease Acquisition Risks(GAO/IMTEC-8.1.6, August 1990), Phase I, steps 7, 8,9.

• GSA, Guide for Requirements Analysis and Analysisof Alternatives, Chapter 3: Analysis of Alternatives.

• OMB Circular A-130: Management of FederalInformation Resources.

• OMB Circular A-109: Major System Acquisitions.• OMB Circular A-76: Performance of Commercial

Activities.• FIPS Publication 64: Guidelines for Documentation of

Computer Programs and Automated Data Systems forthe Initiation Phase.

• FIPS Publication 65: Guidelines for Automatic DataProcessing Risk Analysis.

• FIPS Publication 73: Guidelines for Security ofComputer Applications.


Chapter 6

Acquisition Planning

Acquisition planning is the process of coordinatingand integrating the efforts of personnel responsiblefor acquisitions. A major objective of acquisitionplanning is to promote and provide for full and opencompetition. To ensure that the planning isaccomplished in an effective, economical, and timelymanner, the agency should prepare an acquisitionplan containing an overall strategy for managing thepreaward, acquisition, and postaward phases.

An effective acquisition plan is critical to projectsuccess. The plan sets out what the agency will do tocomplete a procurement and how it will do it. Theplan also specifies the type of contract that will beawarded, how the agency will select a contractor,cost and schedule goals, milestones, significant riskareas, and contract management controls.

Auditors should assess the extent to which theagency’s acquisition planning is realistic andcomprehensive. One part of this assessment shouldbe the review of the Agency Procurement Request(APR) to ascertain whether it is complete andaccurately reflects the objectives and scope of theproject.

Audit Objective To verify that the agency has defined an effectivestrategy and plan for selecting a contractor andmanaging contract performance.


• Acquisition plan and related documents asappropriate, such as a plan of action and milestones.

• Agency procurement request and othercorrespondence with GSA.


Chapter 6


Audit Steps 1. Determine whether the acquisition plan wasreviewed and approved in a timely manner by theofficials designated in the agency’s acquisitionregulations. Ensure that the program managerperiodically reviews the acquisition plan and updatesit when necessary.

2. Review the acquisition plan and determine if itcontains the elements required by the FAR Section 7.1and GAO’s Information Technology: A Model to HelpManagers Decrease Acquisition Risks. These elementsinclude acquisition objectives; cost goals; responsibledecision-makers; capability or performancecharacteristics; risks associated with technicalmatters, scheduling, and costs; plan of action;competition; source selection procedures; contracttype and special contract provisions; contractmanagement procedures or organization; budget andfunding; information needed to monitor contractorperformance; test and evaluation; security andprivacy; and acquisition milestones. The plan shouldalso identify the acquisition method, key “go/no-go”points, a formal training plan, and a contingency planto minimize losses.

3. Determine if the acquisition plan calls for full andopen competition. If the plan calls for limiting theacquisition to resources compatible with existingequipment, verify that conversion cost studies havebeen completed to justify compatibility restrictions. Ifthe plan calls for other than full and opencompetition, verify that restrictions on competition,such as make and model restrictions or sole sourcerequirements, have been justified and approved by thedesignated authority. (See ch. 4 step 6, underRequirements and step 3, under Specifications.)

4. Determine whether the agency has planned a“grand design” project or organized the acquisition


Chapter 6


into modules. Incremental purchasing may limit risksby identifying problems earlier, which allows foreasier change or correction.

5. Evaluate the project management tools andtechniques to satisfy management informationrequirements for monitoring contractor performance,tracking progress against the acquisition plan, andtaking action on cost or schedule slippage.

6. Review the agency’s schedule for developing thesolicitation document and for source selectionactivities. Assess the reasonableness of the schedulethrough discussions with procurement officials andreviews of project progress.

7. Identify the dollar limit of the agency’s generaldelegation of procurement authority from GSA.Confirm that the agency receives a specific delegationof authority from GSA if the value of the acquisitionexceeds the agency’s authority level. Confirm that thenew delegation of procurement authority wasreceived before a solicitation document is issued or acontract awarded.

8. Review the APR if the acquisition exceeds theagency’s delegated procurement authority level.Determine if the APR identifies the officialsresponsible for managing the effort, in accordancewith the GSA guidance provided in FIRMR BulletinC-5. The APR should include:


Chapter 6


a. Names and titles of senior project officials, with adescription of their roles in the organization. If theacquisition exceeds $25 million, the APR shouldalso describe the project manager’s experience inprevious acquisitions, responsibilities, and scope ofauthority, and the reporting structure for eachofficial as well as whether each official is assignedfull- or part-time to the acquisition.

b. The project title and a brief description of the acquisition.

c. Information resources currently in use.

d. Resources to be acquired.

e. The contracting approach. The approach shouldinclude any limitations on competition, the planneddates for release of the solicitation and for contractaward, and a strategy for a follow-onimplementation if a prototype is to be used.

f. Estimated contract life and contract cost.

g. Completion dates for key project documents.

References • FAR Part 7: Acquisition Planning.• FAR Part 34: Major System Acquisition.• GSA, Overview Guide, p. 4-3.• OMB Circular A-109: Major System Acquisitions.• GAO, Information Technology: A Model to Help

Managers Decrease Acquisition Risks(GAO/IMTEC-8.1.6, August 1990), Phase I, step 10.

• FIRMR Bulletin C-5: Delegation of ProcurementAuthority for a Specific Acquisition.


Chapter 7

Solicitation Document

A solicitation document provides informationnecessary for vendors to propose equipment,software, and services to meet the agency’srequirements. In most cases, information resourceswill be purchased by issuing an RFP, which forms thebasis for the resulting contract. Less commonly, anagency may acquire information resources using aninvitation for bids. An RFP may be preceded by arequest for information or request for quotation.

An RFP should be clear and comprehensive andinclude the elements described in GSA’s guidance onstandard solicitation documents. 1 The areas of mostinterest to auditors include section C:Description/Specifications/Work Statement, sectionE: Inspection and Acceptance, and section M:Evaluation Factors for Award. Section C describesthe tasks to be performed by the contractor and theproducts to be delivered. Section E sets outgovernment and contractor responsibilities inensuring that contract deliverables are acceptable tothe agency. Section M explains how the agencyintends to select a winning contractor by describingthe importance of all factors to be considered inevaluating proposals.

In developing the RFP an agency may holdpresolicitation or preproposal conferences in order toseek industry views on the planned acquisition and toencourage companies to offer proposals. Once theRFP is developed, it may be released in draft form inorder to obtain industry questions and reactions.Auditors should determine what steps the agency hastaken to get feedback on its requirements, how theagency has handled comments or questions on aproposed RFP, and whether the agency has acted toensure that contractor proposals are competitive.

1See, for example, U.S. General Services Administration,Information Resources Management Service, Overview Guide:Acquisition of Information Resources, (Jan. 1990).


Chapter 7


A proper RFP is a critical element of a successfulacquisition because it becomes part of the bindingcontract once a proposal is made and accepted. If theRFP does not accurately and clearly describe theagency’s requirements, or if the evaluation factors donot accurately reflect the agency’s priorities, then theresulting acquisition may not meet user needs. If theRFP is unjustifiably restrictive, favoring onecontractor over others, the agency may be unable tobenefit from full and open competition.

Auditors should become familiar with the standardformat for an RFP. The audit team should includepersons sufficiently knowledgeable about theinformation technology being purchased to judge howwell the agency has defined its requirements in theRFP. Team members should include or have access topeople who can identify areas where the RFP doesnot define the agency’s needs well enough to protectthe government’s interests. These persons should alsoknow enough about performance or capabilityvalidation techniques to determine whether or not theagency’s requirements are reasonable and effective.

Audit Objectives To determine whether or not the solicitationdocument is complete, clear, and consistent, verifythat requirements continue to reflect user needs, anddetermine if the proposed evaluation process willresult in an effective and economical acquisition.


• Record of presolicitation or preproposal conference.• Solicitation document: RFP or invitation for bid. Draft

RFPs and Requests for Information, if any wereissued.

• Report of a solicitation review panel or committee ifappropriate.

• Source selection plan.


Chapter 7


• Benchmark materials or other capability andperformance validation requirements.

• Vendor comments or questions on solicitationdocument. If a vendor protests the solicitationdetermine the basis of the protest and how it wasresolved.

• Proposal evaluation guide.

Audit Steps 1. Determine whether the solicitation documentcontains:

a. A statement of work or specifications statementthat clearly and accurately describes thegovernment’s requirements, including a cleardefinition of all deliverables and the conditions oftheir acceptability.

b. A clear definition of government and contractor responsibilities.

c. The relative importance of evaluation factors.

d. A proposal format requiring that cost and technicalelements be separated.

e. Reasonable provisions that protect the agency(such as liquidated damages provisions) or giveincentives to the contractor (such as bonuses forgood performance). Identify option clauses thatcreate uncertainty in work-load projections.

2. Examine the evaluation criteria to ensure:

a. They are consistent with the requirements analysis,specifications, and proposal preparationinstructions.


Chapter 7


b. They provide all the factors and significantsubfactors to be considered in evaluating offersand the relative importance of different technicalor cost factors, in accordance with FAR 15.605.

3. Evaluate the performance evaluation package theagency will use. Determine whether the agencyreimburses contractors for participating inbenchmarking or other testing, and whether the costof such performance evaluation efforts constitutes abarrier to competition.

a. If there is a benchmark, has the benchmark been independently examined by some outside organization? Review the benchmark plan to confirm that demonstration criteria are clearly stated. Determine how the agency selected a representative mix of programs for the benchmark. Determine if the complexity of programs in the benchmark is representative of the projected work load. Determine how the agency validated the benchmark as representative of the agency’s work load.

b. If there is simulation or modeling, determine how the agency selected parameters for the model. Review any concerns or complaints raised by vendors.

c. If a compute-off (demonstration of prototypes) is to be used, confirm that the agency has established plans for prototype and follow-on contracts.

4. Interview users and managers, if necessary, todetermine whether they concur with the solicitation.Determine whether users or managers identify new orchanged requirements not included in the RFP. Findout if there are any factors considered important forselecting an offer that are not included in the


Chapter 7


evaluation criteria or if other included factors unfairlyrestrict the competition. (For related information seech. 2, step 2g, under User Involvement.)

5. Review the agency’s source selection plan to ensurethat it clearly describes the source selectionorganization and activities. Ensure that the sourceselection plan has been approved by the sourceselection authority before presolicitation conferencesare held or the solicitation document is issued, inaccordance with FAR 15.612.

6. Review the industry feedback process anddetermine if:

a. Comments received on the draft solicitationidentify any need for clarification, restrictivespecifications, or alternative ways of satisfying userneeds. (See ch. 4, step 5 under Specifications, for arelated question.)

b. The agency responded promptly and thoroughly tothe comments received. (Judgmental samplingtechniques may be required in this section if thenumber of vendors and vendor comments aresubstantial.)

c. The feedback provided adequate comments onproduct availability in the market.

d. The use of an ombudsman facilitated the process of addressing vendor concerns, disputes, and grievances.

7. Determine who has performed legal reviews of thesolicitation document.

8. Determine if any vendor submitted a protest, towhom (the agency, GAO, or GSA’s Board of Contract


Chapter 7


Appeals), and how it was resolved. Determine thebasis of the protest and its resolution.

References • GSA, Overview Guide, Chapter 6.• FAR, Part 15: Contracting by Negotiation, Subpart

15.4: Solicitation and Receipt of Proposals andQuotations; Subpart 15.612: Formal Source Selection;Subpart 15.605: Evaluation Factors.

• FAR Part 14: Sealed Bidding.• FAR Part 5: Publicizing Contract Actions.• FAR Part 34: Major System Acquisition.• GAO, Information Technology: A Model to Help

Managers Decrease Acquisition Risks(GAO/IMTEC-8.1.6, August 1990), Phase II, steps 2through 7.

• FIPS Publications 75 and 42-1.


Chapter 8

Source Selection

The source selection process is critical to securingthe best value for the government. All proposals mustbe evaluated in accordance with the criteriapublished in the RFP. If the evaluation process doesnot conform to the RFP the agency runs a greater riskof a successful bid protest by losing vendors. Theagency should receive proposals, evaluate thetechnical and cost merits of different proposals,negotiate with contractors, and award a contract inaccordance with a source selection plan developedbefore release of the RFP.

The auditor should be aware of the agency’sorganization and procedures for making a contractaward. Agencies may use some or all of the followingpositions:

• Contracting Officer (CO). The Contracting Officerpublicizes the procurement, amends the RFP ifnecessary, and conducts all negotiations withofferors.

• Source Selection Authority (SSA). The SSA makes thefinal decision on contract award. The ContractingOfficer may be the SSA for some procurements, whilein other cases a more senior manager may serve asSSA.

• Source Selection Advisory Council (SSAC). The SSACreviews the evaluations of different proposals andmakes a recommendation to the SSA on contractaward.

• Source Selection Evaluation Board (SSEB). The SSEBconducts technical and cost evaluations of vendorproposals.

Audit Objective To ensure that the source selection process isplanned and carried out in order to successfully reacha contract that gives the best value to the government.


Chapter 8

Source Selection


• Source selection plan, including source selectionorganization.

• Lessons learned report or other report by theContracting Officer describing negotiations andselection activities.

• Contracting Officer’s contract file.• Records of debriefings, if any.• Results of benchmarks or other performance and

capability validation techniques used.• Correspondence between offerors and the agency

regarding questions or clarifications and anyamendments to the RFP.

• Proposal evaluation guide.• Preaward survey reports.

Audit Steps 1. Examine the evaluation process by reviewingrecords of the source selection procedures anddetermine:

a. If evaluation personnel strictly adhered to andapplied the publicized evaluation process andcriteria in the solicitation.

b. If evaluation factors were applied that were notlisted in the solicitation.

c. Whether cost and technical evaluations were done separately.

d. The role users played in the evaluation process. (See ch. 2, step 2g, for related question.)

e. If the process resulted in (1) the establishment of a competitive range and (2) removal of offerors from further consideration, in accordance with FAR 15.609.


Chapter 8

Source Selection

2. Determine how many vendors received thesolicitation and how many submitted proposals.

3. Determine if any vendor submitted a protest and, ifso, whether the protest was handled by the agency,GAO, or GSA’s Board of Contract Appeals. What wasthe basis of the protest? How was the protestresolved?

4. Determine whether the Contracting Officerestablished prenegotiation objectives for cost, profitand fee, and other issues in accordance with FAR15.807. These objectives should help determine theoverall reasonableness of proposed prices, and maybe based on an independent government costestimate and other information, such as a field pricingreport on each contractor’s proposal.

5. Determine if the agency obtained field pricingsupport in accordance with FAR 15.805-5. Was apreaward audit of the cost proposal obtained andused during negotiations? Were the offeror’sproposed rates compared with the direct, indirect,overhead, and general and administrative ratesrecommended by the appropriate contract auditactivity?

6. Examine the negotiation process.

a. Confirm that discussions with all vendors in the competitive range were held and the proceedings documented.

b. Determine from a review of documentation the controls that existed to protect the security of sensitive information.

c. Contact responsible officials for their evaluationsof security.


Chapter 8

Source Selection

d. Determine if technical leveling or technicaltransfusions occurred that would change vendors’proposals. FAR 15.610 describes these and otherprohibited actions, such as indicating a cost orprice that offerors must meet to be considered forcontract award or informing a vendor of its pricestanding relative to others seeking the contract.

e. Determine if estimated life cycle costs werereconciled with each vendor’s proposal to ensurethat cost estimates appear realistic.

7. Examine how the agency handled best and finaloffers.

a. Determine if the agency made multiple calls forbest and final offers, justified in accordance withFAR 15.611.

b. Determine if the best and final offers were solicitedand evaluated in accordance with the sourceselection plan.

8. Determine if all debriefings:

a. Were scheduled as soon as possible whenrequested by the vendor.

b. Were based on a debriefing plan that addressed andresolved issues likely to cause concern andcomplaints among the losing vendors.

c. Were documented.


Chapter 8

Source Selection

d. Adequately explained why the losing vendor(s) lostthe contract. (Note: In the explanation, the agencycannot make point-by-point comparisons withother proposals, but can point out thegovernment’s evaluation of significantly weak ordeficient elements in the proposal of the vendorbeing debriefed. Refer to FAR 15.1003 for moreinformation.)

9. Examine the effort made to identify lessonslearned.

a. Determine what policies or procedures the agencyusers have to evaluate acquisition results andcommunicate “lessons learned” to staff conductingfuture assignments and to other agencies. Do theseinclude a comparison of the preaward activities tothe acquisition plan?

b. Review the lessons learned report, if one wascompleted, to determine how agency officialsassessed the contracting process.

References • FAR Part 15: Contracting by Negotiation:Subpart 15.4: Proposals and Quotations.Subpart 15.6: Source Selection.Subpart 15.8: Price Negotiations.Subpart 15.10: Notifications, Protests, and Mistakes.

• FIRMR Part 201-39: Acquisition of FederalInformation Processing Resources by Contracting.

• GSA, Overview Guide, Chapter 7: Source Selection.• GSA, Guide for Acquiring Commercial Software.• GAO, Information Technology: A Model to Help

Managers Decrease Acquisition Risks(GAO/IMTEC-8.1.6, August 1990), Phase II, steps 8through 13.


Chapter 9

Contract Management

Contract management includes the steps required toensure that the agency receives products and serviceswithin established costs and time frames. An agencyis required to monitor contractor performance,ensuring that work done conforms to the agency’srequirements. The agency must also control changesto the contract and accept or reject contractdeliverables. Finally, an agency should conductpostimplementation reviews to determine how wellacquisition goals were met and whether theinformation resources acquired should be added to orreplaced.

The agency’s Contracting Officer and ContractingOfficer’s Representative/Contracting Officer’sTechnical Representative hold primary responsibilityfor administering the contract. The ContractingOfficer monitors costs as required by the contracttype (fixed-price or cost-reimbursable) and makescontract modifications as needed. The programmanager helps monitor contractor performance toensure that user requirements are met by theproducts or services delivered and that seniorofficials provide support and oversight.

A contract consists of the agency’s RFP, as amended,and the successful vendor’s proposal. The contractshould specify all deliverables required from thevendor. The Contracting Officer and ContractingOfficer’s Representative/Contracting Officer’sTechnical Representative should ensure thatdeliverables are received as required. Any status andcost reports required from the contractor should bereviewed and action taken to correct problems, ifnecessary. Training, documentation, and maintenancerequirements should be fulfilled.


Chapter 9

Contract Management

Audit Objectives To ensure that the agency:

1. Oversees contractor performance.

2. Ensures that contract requirements continue toaccurately reflect user needs.

3. Verifies that products and services delivered meetuser needs.

4. Implements configuration management.

5. Modifies the contract only as needed.

6. Enforces contract provisions intended to protectthe agency, such as warranties or liquidated damagesclauses.


• Agency regulations or directives specifyingrequirements for periodic reviews, managementoversight, and configuration management.

• The contract as awarded and with modifications.• The agency’s contract management organization and

structure.• Current status reports and cost or schedule

projections.• Current budget reports.• The configuration management plan for the project.

Audit Steps 1. Review agency directives to identify the agency’srequirements for contract oversight. Department ofDefense Standard 2167A, for example, requiresperiodic reviews of contract deliverables, with costand status reports from the contractor. Defense alsohas directives governing configuration managementactivities to ensure that the contractor delivers theequipment or services called for and that no changes


Chapter 9

Contract Management

to the contract are made without consideration oftheir overall impact.

2. Identify the roles of users and senior managers inmonitoring the contract, verifying that both users andsenior managers are involved in managing thecontract and approving any changes.

3. Determine the authority and experience level of theContracting Officer’s Representative or ContractingOfficer’s Technical Representative. (See ch. 3, step 2,for a related step.)

4. Evaluate the project staff.

a. Identify key project staff and review theirexperience and qualifications. Is the ContractingOfficer trained and experienced in informationtechnology procurement? Does the project includestaff experienced in managing contractors?

b. Determine how much turnover there has beenwithin the project staff, including the projectmanager. (See ch. 3, step 2, for a related step.)

5. Assess changes to the agency requirements toensure that the contract continues to reflect validuser needs. Review configuration managementactivities to verify that changes to requirements arerecorded and controlled and that impacts of changesto contract requirements are identified.

6. Assess changes to the agency’s cost and scheduleestimates. Are variances in cost and scheduleprojections tracked by the project manager orContracting Officer’s Technical Representative? Arecost and schedule estimates changed appropriately?


Chapter 9

Contract Management

7. Determine if agency monitoring of the contractor’sperformance includes:

a. Periodically reviewing both scheduled and completed deliverables and effectively reacting to any delays. Determine how the agency compares contractor progress with the contract work schedule.

b. Periodically reviewing contract reports. Review a sample of status and cost reports to verify that they are regularly submitted by the contractor as required by the contract. Discuss their usefulness with the project staff.

c. Assessing the adequacy of the contractor’s qualityassurance process.

8. Assess the effectiveness of the agency’s workingrelationship with the contractor.

a. Verify that the agency has controlled changes to thecontract and integrated the change process into theacquisition management structure. Determine theimpact of changes on contract cost and schedule.

b. Determine if corrections are made, awards are implemented, and damages assessed, as appropriate.

9. Determine if the agency controls contractmodifications by:

a. Requiring the contracting office to approve allcontract modifications.

b. Establishing a review process to ensure thatproposed engineering changes are within the scopeof the contract.


Chapter 9

Contract Management

c. Regularly comparing contract expenditures withthe delegation of procurement authority to ensurethat the agency does not exceed its authorized levelof total expenditures.

References • GSA, Overview Guide, Chapter 8: ContractAdministration.

• GSA, Guide for Contracting Officers’ TechnicalRepresentatives, Chapter 2.

• GAO, Information Technology: A Model to HelpManagers Decrease Acquisition Risks(GAO/IMTEC-8.1.6, August 1990), Phase III.


Chapter 10

Test and Acceptance

Testing provides the basis for making decisions onwhether to accept contract deliverables. For testingto be effective, it must be addressed relatively early inthe acquisition so it can be properly included inplanning. Test plans provide testing procedures andthe evaluation criteria to assess results of the testing.

An agency should establish its initial test plans in thepresolicitation phase. These plans should show howthe agency will verify that the acquired equipment,software, or services meet user needs and satisfysecurity requirements. After a contract is awarded theagency will need to carry out test and acceptanceprocedures. The auditor should ensure that testplanning is conducted early enough so that testrequirements are included in the contract.

In assessing the postaward phase, the auditor shouldensure that the agency has not accepted equipment orsoftware that does not meet its requirements. Thecontract should specify conditions for acceptableperformance. For example, the contract may requirethat a computer operate successfully for 30consecutive days out of a 90-day test period. Agencypersonnel should ensure that the contractor fullymeets the conditions for acceptable performance.Internal auditors may be required to verify that theequipment or software pass the specified tests.

Assessing the test and acceptance phase may requirea high level of technical skill on the part of auditors,such as when an agency has contracted for softwaredevelopment services and must test the quality ofdelivered software. The auditor should be able tounderstand the system requirements, developmentmethodologies, and test tools being used.


Chapter 10

Test and Acceptance

Audit Objectives To confirm that the agency has:

1. Fully defined its requirements for testing thetechnology to be purchased.

2. Effectively carried out test and acceptanceprocedures to verify that the resources purchasedmeet the agency’s needs.


• Agency directives giving requirements for cost andstatus reporting, configuration management, andmanagement oversight.

• Records of configuration reviews or other progressreports.

• Trouble reports or other records of deficiencies foundby agency personnel.

• Records of product acceptances.• Test plans for inspection and acceptance.

Audit Steps 1. Determine if test plans were developed todetermine whether the mandatory:

a. Functional requirements were satisfied.

b. Security requirements arising from governmentalpolicy, agency mission needs, and specific userneeds were satisfied.

2. Determine if the test plans include:

a. Types of testing.

• Unit testing—e.g., in software, individual codemodules are tested by the programmer who wrotethem.


Chapter 10

Test and Acceptance

• Integrated testing—e.g., in software, aggregatefunctions formed by groups of modules andintermodule communication links are tested.

• System testing—examines the operation of thesystem as an entity in an actual or simulatedoperating environment.

b. The locations for testing.

c. A realistic testing schedule.

d. The resource requirements.

• Test equipment needed, including the specific periodof use, types, and quantities needed.

• Software needed to support the testing.• Personnel from both user and acquisition groups with

their needed numbers and skills specified.

e. Testing materials to be used.

• Documentation needed, such as source code andmanuals.

• Software to be tested and its medium.• Test inputs and sample outputs.• Test control software and worksheets.

f. Training in testing to be given, personnel to betrained, and the training staff.

3. Determine whether criteria have been establishedfor certifying that security requirements are met.

4. Determine whether the appropriate userrepresentative has formally acknowledged thecompletion of testing and acceptance of the system. Ifnot, determine the reasons and the potential impact.


Chapter 10

Test and Acceptance

a. Determine if deficiencies discovered in contractdeliverables are expeditiously resolved.

b. Determine if any requirements that were not metby the delivered hardware, software, andtelecommunications are still pending and why.

5. Interview system operators and users to determineif the system has been successfully integrated into theexisting environment.

References • GSA, Overview Guide, chapter 9: Installation andOperation.

• GAO, Information Technology: A Model to HelpManagers Decrease Acquisition Risks(GAO/IMTEC-8.1.6, August 1990), Phase I, step 14;Phase III, steps 4 to 6.

• FIPS Publication 101: Guideline for LifecycleValidation, Verification, and Testing of ComputerSoftware.


Appendix I

Acquisition Profile

The acquisition profile, which is a mechanism fordocumenting key information about an acquisitionunder review, is used to help auditors plan andconduct assessments of an acquisition. The profile,which is kept available for future reference, includesthe

• overall characteristics of the acquisition including itsobjectives shown within the context of the mission(s)or function(s) to be supported,

• management organization and staffing of theacquisition project, and

• acquisition schedule and cost estimates.

OverallCharacteristics

1. What is the project’s name?

2. What is the purpose of the information technologyacquisition? What missions or functions is theacquisition to support?

3. What type of acquisition is it?

• system integration,• commercial off-the-shelf applications,• software conversion,• software development,• hardware, or• other (specify).

4. Are there any related in-house efforts?

5. With what systems will this acquisition interface?

6. Is the acquisition based on

• full and open competition,• competition restricted by compatibility limitations,


Appendix I

Acquisition Profile

• limited competition (such as make or model limited),or

• noncompetitive, sole source.

7. Is the contract based on

• fixed prices (specify type of fixed-price contract), or• cost reimbursement (specify type of

cost-reimbursement contract).

8. What project management tools or techniques arein use to oversee the project (such as Gantt charts orcritical path method/Program Evaluation and ReviewTechnique).

9. What, if any, development tools and techniques(such as Computer-Aided SoftwareEngineering-CASE) are in use?

10. If a primary contractor has been selected for theacquisition, list the contractor name, address,telephone number, and point of contact.

11. Identify key subcontractors, with company names,addresses, telephone numbers, and points of contact.

ManagementOrganizationand Staffing

12. How is the management of the acquisition projectorganized and structured?

13. What are the title, name, and phone numbers ofthe

• acquisition sponsor,• acquisition manager,• contracting officer,• user representatives, and• senior officials who approve acquisitions.


Appendix I

Acquisition Profile

14. What are the responsibilities and duties of theacquisition manager? What is his/her authority (forexample, planning, budget, staffing, or progressreporting)?

15. Has an acquisition steering committee beenestablished? What are the committee’s responsibilitiesand duties?

16. Who staffs the acquisition team and what are theteam members’ qualifications?

AcquisitionSchedule andCost Estimates

17. What was the most recent milestone or keydecision point approved?

18. What are the actual or estimated dates for thefollowing:

• original start/completion dates,• current start/completion dates,• requirements analysis completed and approved,• solicitation document completed for release,• contract awarded,• initial operation,• test and acceptance, and• full operation.

19. Is the project on schedule or has there been aslippage?

20. Has the agency completed an estimate of life cyclecost? If so, what are the original and currentestimates?

21. Identify the budget authority and outlays by yearfor the project.


Appendix I

Acquisition Profile

22. Identify the net benefits or cost savings projectedby the cost/benefit analysis used to justify a chosenalternative, if the acquisition has progressed to thispoint.

23. Has the project received scheduled funds andresources or have there been shortfalls? (Explain anyvariances and their effects.)


Appendix II

Management Metrics

Purpose This appendix describes tools and techniques formeasuring the status of acquisitions involvingsignificant software development. It describestechniques, known as software metrics, forquantitatively measuring how closely a projectconforms to development plans and assessingwhether an acquisition is at risk of delay or costincreases. These techniques require considerableinformation about the system being developed andcan only be used after system design. The metricsdescribed here will generally be used after contractaward, in order to assess how well the agency ismanaging the system development process.

Software metrics, which use mathematical models tomeasure elements of the development process, areintended to help organizations better understand andmanage the relationships between resource decisions,development schedules, and the cost of softwareprojects. By using software metric tools an auditorcan independently evaluate software developmentprojects by analyzing project budgets, requirements,schedules, and resources, and then confirm orquestion cost, schedule, and resource estimates.

Different metrics may be useful for an audit,depending on the objectives and status of anacquisition. Using cost models to estimate the costand length of time necessary to develop a newsoftware system, for example, is appropriate onlyafter requirements have been defined, a system designhas been developed, and the size of the new systemhas been estimated. The models described hererequire estimates of either the lines of code ornumber of function points that the new system willinclude. Cost models project the time and cost todevelop a system on the basis of estimates of thesystem’s size and other pertinent factors. They may beused before a solicitation for software development is


Appendix II

Management Metrics

issued in order to assess the reasonableness of theagency’s schedule (if a system design has beenprepared), or after a contract has been awarded andsoftware development has begun. The other metricsdescribed in this appendix require that systemdevelopment work be underway. A comparison of theactual number of functions successfully tested to thenumber planned in the acquisition schedule, forexample, requires that system testing be underway.

Cost Models Cost models are tools that estimate the effort neededto develop software, based on assumed relationshipsbetween the size of a system and the effort needed todesign, code, and test the software. These models canhelp the auditor assess whether the acquisition’sestimated cost and schedule are reasonable. Eachmodel uses cost drivers, which are parameters suchas the level of experience of the programmers, thereliability requirements of the programs, and thecomplexity of the project, along with the estimatedproject size, to derive overall cost and scheduleestimates for the acquisition. When a system involvingsoftware is being developed, one or more cost modelsmay be useful. A model will be more reliable if ittakes into account the agency’s historical experiencein developing systems.

Cost models are available both commercially andfrom a few agencies within the Department ofDefense. Most of the tools were developed initially foruse with Defense department projects, but can also beused with non-Defense systems. Many are based onindustry-recognized models such as the ConstructiveCost Model (COCOMO), PRICE, Putnam, and Jensen.The commercial tools range in cost from about $500to well over $20,000. Government-developed orgovernment-modified tools are available free of


Appendix II

Management Metrics

charge with a nominal charge for upgraded copies ofthe software.

Because many different packages are available, andbecause more than one can be used, auditors shoulddetermine what models, if any, are used in theiragencies. Typically cost models are derived using datagathered from years of experience with a wide rangeof software projects. Therefore, accuracy ofpredictions may improve as further experience isaccumulated in an agency.

Auditors should use cost models to look fordiscrepancies with the cost and schedule estimatesestablished for an acquisition. By varying theestimated values for cost drivers, the auditor may alsobe able to perform a sensitivity analysis illustratingwhere project estimates are most susceptible tochange. However, cost models have significantlimitations in accuracy unless their underlyingassumptions of system size and cost drivers arecarefully chosen and reflect the agency’s previousexperience in system development. It is a matter ofauditor judgment to decide how discrepant projectestimates and estimates provided by cost modelsshould be to raise concerns about risks of cost andschedule overruns. In making this judgment, theauditor should take into account the uncertainty ofestimates and assumptions made in using a costmodel.

Auditors should use a cost model to provide generalestimates and not precise figures. Therefore, inapplying software metrics to audit work, care must betaken to follow generally accepted governmentauditing standards when drawing conclusions basedon the results of software metrics. Specifically, allfindings should be qualified by the recognition thatthese tools are limited by the accuracy of the


Appendix II

Management Metrics

estimated system size and other project data providedfor the model, the historical data from which themodel was developed, and the fact that all estimatesare projections of an inherently uncertain future.Procedures for minimizing the effect of theselimitations require, for example, proper technicaladvice, qualifying language in audit reports, and areview of data and assumptions by agency officials.

Auditors should also ensure that the model used isconsistent with the software developmentmethodology of the project under consideration. Asystem developed through rapid prototyping, forexample, should be evaluated with a model that takesprototyping into account.

Other Indicators The following indicators are intended to help auditorsassess how effectively an agency is managing asystem development contract. These indicatorsmeasure differences between what an agency plannedfor its contractor to accomplish by certain points inthe systems development life cycle and the actualresults. In most cases this comparison is most easilydone graphically. Using more than one indicator cangive a broader picture of a project’s status. Auditorsshould use indicators appropriate to the project underreview and for which data are available.

Problem Reports This indicator involves tracking the number of openand closed problems reported by a contractor as asystem is developed. A problem could be any anomalydiscovered during design, coding, testing, orimplementation. Problems are distinguished fromfailures of code, which represent defects discoveredduring operation. Contracts should specify howproblems are to be identified and reported. Byreviewing these reports and noting how quickly


Appendix II

Management Metrics

problems are resolved, auditors can obtain anunderstanding of how well the contractor isperforming.

The project manager or Contracting Officer’sTechnical Representative may be the best source forproblem reports. Because an agency may choose toprioritize problems in order of their impact, thereports should distinguish between priority levels ofproblems.

Figure II.1 shows one way to present this kind ofanalysis. It shows the number of new problemsreported and the number of problems closed in orderto show a trend over time and to show any backlogthat may be developing. The auditor may also chooseto report total problems reported and resolved orbreak them out by level of priority. Figure II.2 showsthe length of time that problems have remained openin order to demonstrate how quickly softwareproblems are resolved once found. In this example,three levels of priority are distinguished for reportedproblems.


Appendix II

Management Metrics

Figure II.1: Problem Reports in Software Project


Appendix II

Management Metrics

Figure II.2: Problems byPriority Levels andNumber of MonthsUnresolved

Software Volatility Software volatility is also measured graphically.Figure II.3 shows changes in the total number ofapproved system requirements. Cumulative changesare also tracked, including additions, deletions, andmodifications to requirements. Steady increases in thenumber of requirements and changes to requirementsmay indicate that the project is at risk for delays andcost overruns.


Appendix II

Management Metrics

Figure II.3: Software Requirements Changes

DevelopmentProgress

This indicator involves the comparison of actual andexpected progress in the design, coding, andintegrating of system units. Units can be measured interms of computer software units or computersoftware configuration items. If the project team doesnot complete its design or programming and testingactivities as planned, this indicator can showschedule delays before major milestones are reached.The progress indicator can show all elements ofdesign, coding, testing, and integrating, or it may treatthem as separate indicators. Figure II.4 shows how


Appendix II

Management Metrics

planned and actual progress in the design and codingof software units can be displayed.

Figure II.4: Development Progress

Software Size This indicator records changes in the expectedmagnitude of the software development effort. Thesize of the system, measured in source lines of codeas established by the system design, may change asthe system is coded. Changes in size can be expressed


Appendix II

Management Metrics

as total lines of code or can be broken out into new,modified, and reused lines of code. Changes in theestimated system size may indicate that the projectwas overestimated or underestimated in size andcomplexity. These changes may also indicate thatrequirements are changing and may be related to thesoftware volatility issue described earlier. Increasingestimates of software size should alert the auditorthat the project’s schedule and expected cost may beunderestimated. Changes in the expected system sizemay necessitate reestimates of the development cost,using the cost models described earlier in thisappendix. Figure II.5 shows how the estimates ofsoftware size can be tracked over time.


Appendix II

Management Metrics

Figure II.5: Software Size Estimates

Personnel Stability Tracking the total number of people assigned to asystem development effort compared to plannedstaffing levels provides another indicator of potentialproblems. The auditor can examine projected versusactual levels of total personnel or of key, experiencedpersonnel. Understaffing may result in scheduleslippage. Adding personnel late in a project canactually cause further slippage.


Appendix II

Management Metrics

Other metrics may be chosen as needed. Auditorscould, for example, compare expected to actualusages of hardware resources in order to identifyemerging computer capacity problems. Anotherapproach would involve tracking changes to theestimated completion date for a system. The metricsdescribed here offer suggestions to be tailored for useas appropriate to particular projects.


Appendix III

Reference Materials

Each chapter of this guide lists applicable referencematerials including federal regulations and guidancepublished by GSA, OMB, and other agencies. Whenusing these references, summarized below, auditorsshould ensure that the most current version availableis used.

FederalRegulations

Applicable regulations include both general rules forprocurement, the FAR, and the FIRMR regulationswritten by GSA specifically for acquiring federalinformation processing resources. GSA issuesregulations under its Brooks Act authority. In additionto the FIRMR, GSA issues bulletins that provideguidance on a wide range of federal informationprocessing acquisition issues:

GSA • Federal Acquisition Regulation (FAR)• Federal Information Resource Management

Regulation (FIRMR)

OMB Circulars • A-109: Major System Acquisitions• A-130: Management of Federal Information

Resources—under proposed revision• A-76: Performance of Commercial Activities• A-11: Preparation and Submission of Budget

Estimates

GSA AcquisitionGuide Series

• Overview Guide• Guide for Requirements Analysis and Analysis of

Alternatives• Guide for Acquiring Maintenance Services• Guide for Acquiring Commercial Software• Guide for Contracting Officer’s Technical

Representatives• Guide for Acquiring Systems Integration Services


Appendix III

Reference Materials

National Institute ofScience andTechnology(NIST)—formerlythe National Bureauof Standards

• Federal Information Processing Standard (FIPS)Publications. FIPS Publications describe federalstandards for hardware, software, and themanagement of information technologies. FIPSPublications relevant to the acquisition of informationtechnology include the following:

FIPS PUB 11-3—Guideline: American NationalDictionary for Information Processing Systems, Feb. 1, 1991.

FIPS PUB 38—Guidelines for Documentation ofComputer Programs and Automated Data Systems,Feb. 15, 1976.

FIPS PUB 42-1—Guidelines for Benchmarking ADPSystems in the Competitive ProcurementEnvironment, May 15, 1977.

FIPS PUB 46-1—Data Encryption Standard, Jan. 22, 1988.

FIPS PUB 64—Guidelines for Documentation ofComputer Programs and Automated Data Systems forthe Initiation Phase, Aug. 1, 1979.

FIPS PUB 65—Guideline for Automatic DataProcessing Risk Analysis, Aug. 1, 1979.

FIPS PUB 73—Guidelines for Security of ComputerApplications, June 30, 1980.

FIPS PUB 75—Guideline on ConstructingBenchmarks for ADP System Acquisitions, Sept. 18, 1980.

FIPS PUB 76—Guideline for Planning and Using aData Dictionary System, Aug. 20, 1980.


Appendix III

Reference Materials

FIPS PUB 87—Guidelines for ADP ContingencyPlanning, Mar. 27, 1981.

FIPS PUB 88—Guideline on Integrity Assurance andControl in Database Administration, Aug. 14, 1981.

FIPS PUB 96—Guideline for Developing andImplementing A Charging System for Data ProcessingServices, Dec. 6, 1982.

FIPS PUB 99—Guideline: A Framework for theEvaluation and Comparison of Software DevelopmentTools, Mar. 31, 1983.

FIPS PUB 101—Guideline for Lifecycle Validation,Verification, and Testing of Computer Software, June 6, 1983.

FIPS PUB 102—Guidelines for Computer SecurityCertification and Accreditation, Sept. 27, 1983.

FIPS PUB 106—Guideline on Software Maintenance,June 15, 1984.

FIPS PUB 124—Guideline on FunctionalSpecifications for Database Management Systems,Sept. 30, 1986.

FIPS PUB 127-1—Database Language SQL, Feb. 2, 1990.

FIPS PUB 132—Guideline for Software Verificationand Validation Plans, Nov. 19, 1987.

FIPS PUB 146-1—Government Open SystemsInterconnection Profile, Apr. 3, 1991.

FIPS PUB 151-1—POSIX: Portable Operating SystemInterface for Computer Environments, Mar. 28, 1990.


Appendix III

Reference Materials

FIPS PUB 158—The User Interface Component of theApplications Portability Profile, May 29, 1990.

• Special publications. Publications relevant toinformation technology acquisition include thefollowing:

500-087—Management Guide for SoftwareDocumentation, January 1982.

500-090—Guide to Contracting for SoftwareConversion Services, May 1982.

500-105—Guide to Software Conversion Management.

500-106—Guide on Software Maintenance.

500-109—Overview of Computer SecurityCertification and Accreditation.

500-120—Security of Personal Computer Systems: AManagement Guide.

500-133—Technology Assessment: Methods forMeasuring the Level of Computer Security.

500-134—Guide on Selecting ADP Backup ProcessingAlternatives.

500-147—Guidance on Requirements Analysis forOffice Automation Systems (Update).

500-148—Application Software Prototyping andFourth Generation Languages.

500-153—Guide to Auditing for Controls and Security:A System Development Life Cycle Approach.

500-154—Guide to Distributed Database Management.


Appendix III

Reference Materials

500-155—Management Guide to Software Reuse.

500-161—Software Configuration Management: AnOverview.

500-165—Software Verification and Validation: ItsRole in Computer Assurance and Its Relationshipwith Software Product Management Standards.

500-172—Computer Security Training Guidelines.

500-173—Guide to Data Administration.

500-174—Guide for Selecting Automated RiskAnalysis Tools.

500-175—Management of Networks Based on OpenSystems Interconnection (OSI) Standards: FunctionalRequirements and Analysis.

500-180—Guide to Software Acceptance.

500-183—Stable Implementation Agreements forOpen System Interconnection Protocols.

500-184—Functional Benchmarks for FourthGeneration Languages.

500-187—Application Portability Profile, The U.S.Government’s Open System Environment ProfileOSE/1 Version 1.0.

500-192—Government Open Systems InterconnectionProfile Users’ Guide, Version 2.

500-193—Software Reengineering: A Case Study andLessons Learned.


Appendix III

Reference Materials

800-4—Computer Security Considerations in FederalProcurements: A Guide for Procurement Initiators,

Contracting Officers, and Computer Security Officials,March 1992.

GAO Audit Guidance • Information Technology: A Model to Help ManagersDecrease Acquisition Risks (GAO/IMTEC-8.1.6,August 1990).

• Government Auditing Standards, 1988 Revision.


Appendix IV

Major Contributors to This Audit Guide

InformationManagement andTechnologyDivision,Washington, D.C.

Mark E. Heatwole, Assistant DirectorDavid R. Turner, Evaluator-in-ChargeBernard R. Anderson, Senior EvaluatorPeter C. Wade, Senior EvaluatorTrinh N. Hoang, Computer ProgrammerDavid Chao, Technical ReviewerShane D. Hartzler, Editor


Glossary

Acquisition The obtaining, by contract with appropriated funds,of supplies or services (including construction) byand for the use of the federal government throughpurchase or lease, whether the supplies or servicesare already in existence or must be created,developed, demonstrated, and evaluated. Acquisitionbegins at the point when agency needs are establishedand includes the description of requirements tosatisfy agency needs, solicitation and selection ofsources, award of contracts, contract financing,contract performance, contract administration, andthose technical and management functions directlyrelated to the process of fulfilling agency needs bycontract.

AcquisitionPlanning

The process by which the efforts of all personnelresponsible for an acquisition are coordinated andintegrated through a comprehensive plan for fulfillingthe agency need in a timely manner and at areasonable cost. It includes developing the overallstrategy for managing the acquisition.

AgencyProcurementRequest (APR)

A request by a federal agency for GSA to acquireinformation processing resources or for GSA todelegate the authority to acquire these resources.

Architecture The overall structure of a computer system includinghardware and software.

Availability The degree to which a system or component isoperational and accessible when required for use,often expressed as a probability.


Glossary

Baseline (1) A specification or product that has been formallyreviewed and agreed upon that thereafter serves as

the basis for further development and that can bechanged only through formal change controlprocedures. (2) A document or set of such documentsformally designated and fixed at a specific timeduring the life cycle of a configuration item. Note:Baselines, plus approved changes from thosebaselines, constitute the current configurationidentification. (3) Any agreement or result designatedand fixed at a given time from which changes requirejustification and approval.

Benchmark Test A test that uses a representative set of programs anddata designed to evaluate the performance ofcomputer hardware and software in a givenconfiguration.

Best and FinalOffer

A final opportunity for offerors in the competitiverange to revise proposals.

CapabilityValidation

The technical verification of the ability of a proposedsystem configuration, replacement component, or thefeatures or functions of its software, to satisfyfunctional requirements. The intent is to ensure thatthe proposed resources can provide the requiredfunctions. Performance requirements are not impliedor measured in the validation. Examples of capabilityvalidation include:

a. operational capability demonstrations of thefunctions of the hardware, operating system, orsupport software;

b. verification of conformance with informationprocessing standards;


Glossary

c. expert examination of the technical literaturesupplied with the offer;

d. contacts with other users of the proposedinformation processing resource; and

e. vendor certification of conformance with thefunctional requirements.

Certification (1) A written guarantee that a system or componentcomplies with its specified requirements and isacceptable for operational use. For example, a writtenauthorization that a computer system is secure and ispermitted to operate in a defined environment. (2) Aformal demonstration that a system or componentcomplies with its specified requirements and isacceptable for operational use. (3) The process ofconfirming that a system or component complies withits specified requirements and is acceptable foroperational use.

Change Control See configuration control.

CommerceBusiness Daily

A daily publication that lists the government’sprocurement invitations, contract awards,subcontracting leads, sales, surplus property, andforeign business opportunities. (GSA/IRMS, A Guidefor Acquiring Commercial Software, Jan. 1991, p. A-1.)

Compatibility (1) The ability of two or more systems or componentsto perform their required functions while sharing thesame hardware or software environment. (2) Theability of two or more systems or components toexchange information.


Glossary

Compatibility-LimitedRequirement

A statement of requirements expressed in terms thatrequire items to be compatible with existinginformation processing resources.

CompetitiveRange

The group of offerors selected, after technical andcost evaluation, to whom award of a contract is areasonable possibility.

Configuration (1) The arrangement of a computer system ornetwork as defined by the nature, number, and chiefcharacteristics of its functional units. (2) The physicaland logical elements of an information processingsystem, the manner in which they are organized andconnected, or both. The term may refer to a hardwareconfiguration or a software configuration.

ConfigurationControl

An element of configuration management, consistingof the evaluation, coordination, approval ordisapproval, and implementation of changes toconfiguration items after formal establishment oftheir configuration identification.

ConfigurationItem

An aggregation of hardware and/or software that isdesignated for configuration management and treatedas a single entity in the configuration managementprocess.

ConfigurationManagement

The continuous control of changes made to a system’shardware, software, and documentation throughoutthe development and operational life of the system.


Glossary

ContractingOfficer (CO)

A person with the authority to enter into, administer,and/or terminate contracts and make relateddeterminations and findings.

ContractingOfficer’sTechnicalRepresentative(COTR)

An individual to whom the CO delegates certaincontract responsibilities, usually related to technicalacceptance issues.

Conversion Modification of existing software to enable it tooperate with similar functional capability in adifferent environment; for example, converting aprogram from FORTRAN to Ada or converting aprogram that runs on one computer to run onanother.

Cost-ReimbursementContract

A contract in which the government reimburses thecontractor for expenses so long as the contractorprovides its “best effort” to complete the work calledfor.

Delegation ofProcurementAuthority (DPA)

Authority to acquire information processing resourcesup to a specified limit, issued by GSA in response toan agency procurement request.

FederalAcquisitionRegulation (FAR)

The regulation that codifies uniform acquisitionpolicies and procedures for executive agenciesgovernmentwide.


Glossary

FederalInformationResourcesManagementRegulation(FIRMR)

The regulation that sets forth uniform policies andprocedures for acquiring information processingresources; used in conjunction with the FAR.

Fixed-PriceContract

A contract that provides for a firm price or inappropriate cases, a firm price with fees or otheradjustments.

FunctionalRequirement

A requirement that specifies a function that a systemor system component must be able to perform.

IndependentVerification andValidation (IV&V)

Verification and validation performed by anorganization that is technically, managerially, andfinancially independent of the developmentorganization. (See verification and validation, definedseparately as follows.)

Invitation for Bid(IFB)

The solicitation document used when contracting bysealed bidding.

Interoperability The ability of information technology resources toprovide services to and accept services from otherresources and to use the services so exchanged toenable them to operate effectively together.

LiquidatedDamages

Compensation to the government for a contractor’sfailure to perform in a timely manner.


Glossary

Maintainability The ease with which maintenance of a functional unitcan be performed in accordance with prescribedrequirements.

Market Survey Attempts to ascertain whether other qualified sourcescapable of satisfying the government’s requirementexist. This testing of the marketplace may range fromwritten or telephone contacts with knowledgeablefederal and non-federal experts regarding similar orduplicate requirements and the results of any markettest recently undertaken, to the more formalsources-sought announcements in pertinentpublications (e.g., technical/scientific journals, theCommerce Business Daily), or solicitations forinformation or planning purposes.

PerformanceValidation

The technical verification of the ability of a proposedsystem configuration or replacement component tohandle agency-specific work-load volumes (presentand expected) within agency-determinedperformance time constraints.

Program Manager The key management official who represents theprogram office in formulating resource requirementsand managing presolicitation activities. In someorganizations the program manager or anothermanagement official is designated as the acquisitionmanager for a specific acquisition.

Protest A written objection by an interested party to (1) asolicitation for a proposed contract, (2) a proposedaward, or (3) the award of a contract.


Glossary

Prototype A preliminary type, form, or instance of a system thatserves as a model for later stages or for the final,complete version of the system.

Prototyping A hardware and software development technique inwhich a preliminary version of part or all of thehardware or software is developed to permit userfeedback, determine feasibility, or investigate timingor other issues in support of the developmentprocess.

RapidPrototyping

A type of prototyping in which emphasis is placed ondeveloping prototypes early in the developmentprocess to permit early feedback and analysis insupport of the development process.

Reliability The ability of a system or component to perform itsrequired functions under stated conditions for astated period of time.

Request forComment

An announcement in the Commerce Business Daily orother publication requesting industry comment ondraft specifications for resources.

Request forInformation

An announcement in the Commerce Business Daily orother publication requesting information fromindustry about a planned acquisition, and, in somecases, corporate capability information.

Request forProposals (RFP)

The solicitation document used in negotiatedprocurements to communicate governmentrequirements and to solicit proposals.


Glossary

Solicitation An official government request for bids/proposalsgenerally publicized in the Commerce Business Dailyin accordance with federal regulations.

Source SelectionAuthority (SSA)

The government official in charge of selecting thesource for an acquisition. Most often the title is usedwhen the selection process is formal and the officialis other than the Contracting Officer.

Source SelectionEvaluation Board(SSEB)

A board composed of technical, contract, informationresources managers, and other government personnelwhose primary function is to evaluate proposalsreceived in response to an RFP.

Source SelectionPlan

A document that describes the entire process forawarding a contract—proposal evaluation criteria,evaluation methodology, evaluator’s responsibilities,and final selection procedures.

Specific Makeand ModelSpecification

A description of the government’s requirement forresources which is so restrictive that only a particularmanufacturer’s products will satisfy the government’sneeds.

Specification A written description of the technical requirementsfor resources stated in an IFB or RFP.

Statement ofWork

A technical description of resources, prepared forinclusion in a solicitation document.

System Life A projection of the time period that begins with theinstallation of the resource and ends when theagency’s need for that resource has terminated.


Glossary

TechnicalLeveling

Helping an offeror to bring its proposal up to the levelof other proposals through successive rounds ofdiscussion, such as by pointing out weaknessesresulting from the offeror’s lack of diligence,competence, or inventiveness in preparing theproposal.

TechnicalTransfusions

Government disclosure of technical informationpertaining to a proposal that results in improvementof a competing proposal.

Test Plan A plan prepared by the government that details thespecific tests and procedures to be followed.

Uniform ContractFormat

The format required by the FAR for preparation of asolicitation.

User (1) Any person, organization, or functional unit thatuses the services of an information processingsystem. (2) Any person or any thing that may issue orreceive commands and messages to or from theinformation system.

Validation The process of evaluating a system or componentduring or at the end of the development process todetermine whether it satisfies specified requirements.

Verification (1) The process of evaluating a system or componentto determine whether the products of a givendevelopment phase satisfy the conditions imposed atthe start of that phase. (2) Formal proof of programcorrectness.


Glossary

Work Load The mix of tasks typically run on a given computersystem. Major characteristics include input/outputrequirements, amount and kinds of computation, andcomputer resources required.

(510757) GAO/IMTEC-8.1.4 Assessing Acquisition RisksPage 97

Ordering Information

The first copy of each GAO report and testimony

is free. Additional copies are $2 each. Orders

should be sent to the following address,

accompanied by a check or money order made

out to the Superintendent of Documents, when

necessary. Orders for 100 or more copies to be

mailed to a single address are discounted

25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 6015

Gaithersburg, MD 20884-6015

or visit:

Room 1100

700 4th St. NW (corner of 4th & G Sts. NW)

U.S. General Accounting Office

Washington, DC

Orders may also be placed by calling

(202) 512-6000 or by using fax number

(301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available

reports and testimony. To receive facsimile

copies of the daily list or any list from the past

30 days, please call (301) 258-4097 using a

touchtone phone. A recorded menu will provide

information on how to obtain these lists.

United States

General Accounting Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Correction Requested

Bulk Mail

Postage & Fees Paid

GAO

Permit No. G100

GAO Technology Division - Government Accountability Office

Documents

Transcript of GAO Technology Division - Government Accountability Office