GAO Report: HEALTHCARE.GOV - Ineffective Planning and Oversight Practices Underscore the Need for...

8/12/2019 GAO Report: HEALTHCARE.GOV - Ineffective Planning and Oversight Practices Underscore the Need for Improved Contract Management

1/66


2/66

United States Government Accountability Office

Highlights ofGAO-14-694,a report tocongressional requesters

July 2014

HEALTHCARE.GOV

Ineffective Planning and Oversight PracticesUnderscore the Need for Improved ContractManagement

Why GAO Did This Study

In March 2010, the Patient Protectionand Affordable Care Act required theestablishment of health insurancemarketplaces by January 1, 2014.Marketplaces permit individuals tocompare and select insurance plansoffered by private insurers. For statesthat elected not to establish amarketplace, CMS was responsible for

developing a federal marketplace. InSeptember 2011, CMS contracted forthe development of the FFM, which isaccessed through Healthcare.gov.

When initial enrollment began onOctober 1, 2013, many usersencountered challenges accessing andusing the website. GAO was asked toexamine various issues surroundingthe launch of the Healthcare.govwebsite. Several GAO reviews areongoing.

This report assesses, for selected

contracts, (1) CMS acquisition planningactivities; (2) CMS oversight of cost,schedule, and system capabilitychanges; and (3) CMS actions toaddress contractor performance. GAOselected two task orders and onecontract that accounted for 40 percentof CMS spending and were central tothe website. For each, GAO reviewedcontract documents and interviewedCMS program and contract officials aswell as contractors.

What GAO Recommends

GAO recommends that CMS takeimmediate actions to assess increasingcontract costs and ensure thatacquisition strategies are completedand oversight tools are used asrequired, among other actions. CMSconcurred with four recommendationsand partially concurred with one.

What GAO Found

The Centers for Medicare & Medicaid Services (CMS) undertook thedevelopment of Healthcare.gov and its related systems without effective planningor oversight practices, despite facing a number of challenges that increased boththe level of risk and the need for effective oversight. CMS officials explained thatthe task of developing a first-of-its-kind federal marketplace was a complex effortwith compressed time frames. To be expedient, CMS issued task orders todevelop the federally facilitated marketplace (FFM) and federal data services hub(data hub) systems when key technical requirements were unknown, includingthe number and composition of states to be supported and, importantly, thenumber of potential enrollees. CMS used cost-reimbursement contracts, whichcreated additional risk because CMS is required to pay the contractors allowablecosts regardless of whether the system is completed. CMS program staff alsoadopted an incremental information technology development approach that wasnew to CMS. Further, CMS did not develop a required acquisition strategy toidentify risks and document mitigation strategies and did not use availableinformation, such as quality assurance plans, to monitor performance and informoversight.

CMS incurred significant cost increases, schedule slips, and delayed systemfunctionality for the FFM and data hub systems due primarily to changingrequirements that were exacerbated by oversight gaps. From September 2011 toFebruary 2014, FFM obligations increased from $56 million to more than $209

million. Similarly, data hub obligations increased from $30 million to nearly $85million. Because of unclear guidance and inconsistent oversight, there wasconfusion about who had the authority to approve contractor requests to expendfunds for additional work. New requirements and changing CMS decisions alsoled to delays and wasted contractor efforts. Moreover, CMS delayed keygovernance reviews, moving an assessment of FFM readiness from March toSeptember 2013just weeks before the launchand did not receive requiredapprovals. As a result, CMS launched Healthcare.gov without verification that itmet performance requirements.

Late in the development process, CMS identified major performance issues withthe FFM contractor but took only limited steps to hold the contractor accountable.In April and November 2013, CMS provided written concerns to the contractorabout product quality and responsiveness to CMS direction. In September 2013,

CMS program officials became so concerned about the contractors performancethat they moved operations to the FFM contractors offices to provide on-sitedirection. At the time, CMS chose to forego actions, such as withholding thepayment of fee, in order to focus on meeting the website launch date. Ultimately,CMS declined to pay about $267,000 in requested fee. This represents about 2percent of the $12.5 million in fees paid to the FFM contractor. CMS awarded anew contract to another firm for $91 million in January 2014 to continue FFMdevelopment. As of June 2014, costs on the contract had increased to over $175million due to changes such as new requirements and other enhancements,while key FFM capabilities remained unavailable. CMS needs a mitigation plan toaddress these issues. Unless CMS improves contract management and adheresto a structured governance process, significant risks remain that upcoming openenrollment periods could encounter challenges.

ViewGAO-14-694.For more information,contact William T. Woods at (202) 512-4841 [email protected].
http://www.gao.gov/products/GAO-14-694http://www.gao.gov/products/GAO-14-694http://www.gao.gov/products/GAO-14-694http://www.gao.gov/products/GAO-14-694http://www.gao.gov/products/GAO-14-694http://www.gao.gov/products/GAO-14-694mailto:[email protected]:[email protected]:[email protected]://www.gao.gov/products/GAO-14-694http://www.gao.gov/products/GAO-14-694


3/66

Page i GAO-14-694 Healthcare.gov Contracts

Letter 1

Background 3Oversight Weaknesses and Lack of Adherence to Planning

Requirements Compounded Acquisition Planning Challenges 11Changing Requirements and Oversight Gaps Contributed to

Significant Cost Growth, Schedule Delays, and ReducedCapabilities during FFM and Data Hub Development 19

CMS Identified Significant Contractor Performance Issues for theFFM Task Order but Took Limited Action 31

Conclusions 39

Recommendations for Executive Action 40Agency Comments, Third-Party Views, and Our Evaluation 40

Appendix I Objectives, Scope, and Methodology 47

Appendix II Cumulative Cost Increases for the Task Orders for Developing

the Federally Facilitated Marketplace System and Federal Data

Services Hub Task Orders 51

Appendix III Comments from the Department of Health and Human Services 53

Appendix IV GAO Contact and Staff Acknowledgments 60

Figures

Figure 1: Timeline of Key Healthcare.gov Events 5Figure 2: Overview of Healthcare.gov and Selected Supporting

Systems 8Figure 3: Key Contract Phases and Selected Activities 10Figure 4: Cumulative Obligation Increases for the Task Orders for

Developing the Federally Facilitated Marketplace Systemand Federal Data Services Hub 20

Figure 5: Planned Schedule of Development Milestone Reviews inthe Federally Facilitated Marketplace System andFederal Data Services Hub Task Orders 24

Contents


4/66

Page ii GAO-14-694 Healthcare.gov Contracts

Figure 6: Completion Status of Federally Facilitated MarketplaceSystem Modules at the End of the Task Order, February2014 26

Figure 7: Federally Facilitated Marketplace System ContractorPerformance during Development 33

Figure 8: Cumulative Obligations for Accenture Federal ServicesContract to Continue FFM Development as of June 5,2014 38


5/66

Page iii GAO-14-694 Healthcare.gov Contracts

Abbreviations

CCIIO Center for Consumer Information andInsurance OversightCHIP State Childrens Health Insurance ProgramCGI Federal CGI Federal Inc.CMS Centers for Medicare & Medicaid ServicesCOR contracting officers representativedata hub federal data services hubDOD Department of DefenseFAR Federal Acquisition RegulationFFM federally facilitated marketplaceGTL government task leaderHHS Department of Health and Human Services

HHSAR Department of Health and Human ServicesAcquisition Regulation

IT Information technologyIRS Internal Revenue ServiceOAGM Office of Acquisition and Grants ManagementOCIIO Office of Consumer Information and

Insurance OversightOIS Office of Information ServicesOPM Office of Personnel ManagementPPACA Patient Protection and Affordable Care ActQSSI QSSI, Inc.VA Department of Veterans Affairs

This is a work of the U.S. government and is not subject to copyright protection in theUnited States. The published product may be reproduced and distributed in its entiretywithout further permission from GAO. However, because this work may containcopyrighted images or other material, permission from the copyright holder may benecessary if you wish to reproduce this material separately.


6/66

Page 1 GAO-14-694 Healthcare.gov Contracts

441 G St. N.W.Washington, DC 20548

July 30, 2014

Congressional Requesters

The Patient Protection and Affordable Care Act (PPACA), enacted inMarch 2010, made fundamental changes to the availability andaffordability of health insurance coverage.1A central provision of the law

required the establishment of state health insurance exchanges, nowcommonly referred to as marketplaces, by January 1, 2014. Marketplacespermit individuals to compare and select private health insurance plans.

For states that elected not to establish a marketplace, PPACA requiredthe federal government to establish and operate a federal marketplace.2

The Centers for Medicare & Medicaid Services (CMS) within theDepartment of Health and Human Services (HHS) was responsible fordesigning, developing, and implementing the information technology (IT)systems needed to support the federal marketplace which users accessvia the Healthcare.gov website. CMS largely relied on contractors todevelop, build, and operate the necessary information technologysystems. When initial enrollment began on October 1, 2013, many userswere unable to successfully access and use the Healthcare.gov website

to obtain health insurance information due to problems such as websitefailures, errors, and slow response times.

Given the high degree of congressional interest in examining thedevelopment, launch, and other issues associated with accessing thefederal marketplace through the Healthcare.gov website, GAO isconducting a body of work in order to assist Congress with its oversightresponsibilities. This report examines selected contracts and task orderscentral to the development and launch of the Healthcare.gov website byassessing (1) CMS acquisition planning activities; (2) CMS oversight ofcost, schedule, and system capability changes; and (3) actions taken byCMS to identify and address contractor performance issues.

1Pub. L. No. 111-148, 124 Stat. 119 (2010).

2PPACA also requires the creation of Small Business Health Options Program exchangeswhere small businesses can shop for and purchase health coverage for their employees.


7/66


To address these objectives, we reviewed the Federal Procurement DataSystem-Next Generation, which is the governments procurementdatabase, to identify CMS contracts and task orders related to the ITsystems supporting the Healthcare.gov website and amounts obligatedfrom fiscal year 2010 through March 2014. We performed data reliabilityassessments and confirmed that the data were sufficiently reliable for ourpurposes. Based on this information as well as interviews with CMScontracting and program officials, we selected one contract and two taskorders issued under an existing 2007 contract for our review.3

To assess CMS acquisition planning activities, we reviewed the FederalAcquisition Regulation (FAR) and relevant HHS/CMS policies andguidance and evaluated contract file documents. To assess CMS

oversight of cost, schedule, and system capability changes, we reviewedcontract modifications, contract deliverables, contractor monthly statusreports, and other documents. To assess actions taken by CMS toidentify and address contractor performance issues, we identifiedmonitoring requirements and analyzed contract file documentation. Tosupport work on all three objectives, we interviewed contracting officials inCMSs Office of Acquisition and Grants Management and programofficials in CMSs Office of Information Services. In addition, weinterviewed the contractors to obtain their perspective on CMSs oversightof cost, schedule, and system capabilities. Appendix I provides additionaldetails about our scope and methodology.

The

contract and task orders combined accounted for more than 40 percent ofthe total CMS reported obligations related to the development of

Healthcare.gov and its supporting systems as of March 2014. Specificallywe selected the task orders issued to CGI Federal Inc. (CGI Federal) forthe development of the federally facilitated marketplace (FFM) systemand to QSSI, Inc. (QSSI) for the development of the federal data serviceshub (data hub) in September 2011and the contract awarded to

Accenture Federal Services in January 2014 to complete FFMdevelopment and enhance existing functionality.

3The existing contract is a multiple-award, indefinite-delivery, indefinite-quantity contract(hereinafter referred to as the 2007 contract). This contract type provides for an indefinitequantity, within stated limits, of supplies or services during a fixed period. TheGovernment places orders for individual requirements. Quantity limits may be stated asnumber of units or as dollar values. FAR 16.504.


8/66


We conducted this performance audit from January 2014 to July 2014, inaccordance with generally accepted government auditing standards.Those standards require that we plan and perform the audit to obtainsufficient, appropriate evidence to provide a reasonable basis for ourfindings and conclusions based on our audit objectives. We believe thatthe evidence obtained provides a reasonable basis for our findings andconclusions based on our audit objectives.

Each marketplace created under PPACA is intended to provide aseamless, single point of access for individuals to enroll in qualified health

plans,4apply for income-based financial subsidies established under thelaw and, as applicable, obtain an eligibility determination for other healthcoverage programs, such as Medicaid or the State Childrens HealthInsurance Program (CHIP).5To obtain health insurance offered through

the marketplace, individuals must complete an application and meetcertain eligibility requirements defined by PPACA, such as being a U.S.citizen or legal immigrant. For those consumers determined eligible, themarketplaces permit users to compare health plans and enroll in the planof their choice. States had various options for marketplace participation,including (1) establishing their own state-based marketplace, (2) deferringto CMS to operate the federal marketplace in the state, or (3) participating

in an arrangement called a partnership marketplace in which the stateassists with some federal marketplace operations.6

In our June 2013 report on CMS efforts to establish the federalmarketplace, we concluded that certain factorssuch as the evolvingscope of marketplace activities required in each statesuggested the

4A qualified health plan is an insurance plan that is certified by a marketplace to offercoverage through that marketplace.

5Medicaid is a joint federal-state program that finances health care coverage for certain

low-income individuals. CHIP is a federal-state program that provides health carecoverage to children 18 years of age and younger living in low-income families whoseincomes exceed the eligibility requirements for Medicaid.

6States seeking to operate a state-based marketplace were required to submit anapplication to CMS in December 2012. States electing not to establish a state-basedmarketplace, but seeking to participate in a partnership marketplace were required tocomplete an abbreviated version of that application by February 2013. States electing notto establish a state-based exchange or participate in a partnership exchange were notrequired to submit an application to CMS.

Background


9/66


potential for implementation challenges going forward.7

In comments on a

draft of that report, HHS emphasized the progress it had made sincePPACA became law and expressed its confidence that marketplaceswould be open and functioning in every state on October 1, 2013.

PPACA required the establishment of marketplaces in each state byJanuary 2014. Based on the expectation that individuals and familieswould need time to explore their coverage options and plan issuers wouldneed time to process plan selections, HHS established October 1, 2013,as the beginning of the enrollment period for all marketplaces, including

the federal marketplace.8

7GAO, Patient Protection and Affordable Care Act: Status of CMS Efforts to EstablishFederally Facilitated Health Insurance Exchanges,

Figure 1 shows a timeline of major contracting,legal or regulatory, and organizational events during that developmentperiod, as well as future milestones through the beginning of openenrollment for 2015.

GAO-13-601(Washington, D.C.: June19, 2013).

8HHS proposed October 1, 2013, as the start of the initial open enrollment period in a July2011 proposed rule and included this date in the statement of work for both the FFM anddata hub task orders. 76 Fed. Reg. 41866 (July 15, 2011). CMS issued a final ruleadopting this date in March 2012. 77 Fed. Reg. 18310 (Mar. 27, 2012) (codified at 45C.F.R. 155.410(b)).

Timeline of Key Events
http://www.gao.gov/products/GAO-13-601http://www.gao.gov/products/GAO-13-601http://www.gao.gov/products/GAO-13-601http://www.gao.gov/products/GAO-13-601


10/66


Figure 1: Timeline of Key Healthcare.gov Events

Notes:aA letter contract is a written preliminary contractual instrument that authorizes the contractor to beginwork immediately. FAR 16.603.


11/66


b

A contract is considered definitized when final agreement on contract terms and conditions isreached.

The Healthcare.gov website is supported by several systems, includingthe FFM and the federal data services hub. Additional componentsinclude the Enterprise Identity Management System that confirms theconsumers identity when entering the system.9

Healthcare.gov is the Internet address of a federal government-operatedwebsite that serves as the online user interface for the federal

marketplace. The website allows the consumer to create an account,input required information, view health care plan options and make a planselection.

The FFM accepts and processes data entered through the website andwas intended to provide three main functions:

Eligibility and enrollment.This module guides applicants through astep-by-step process to determine their eligibility for coverage andfinancial assistance, after which they are shown applicable coverageoptions and have the opportunity to enroll.

Plan management.This module interacts primarily with stateagencies and health plan issuers. The module is intended to provide asuite of services for activities such as submitting, monitoring, andrenewing qualified health plans.

Financial management.This module facilitates payments to issuers,including premiums and cost-sharing reductions, and collects datafrom state-based marketplaces.

Other FFM functions include services related to system oversight,communication and outreach strategies, and customer service.

The data hub routes and verifies information among the FFM and externa

data sources, including other federal and state sources of information and

9GAO, Patient Protection and Affordable Care Act: Preliminary Results of UndercoverTesting of Enrollment Controls for Health Care Coverage and Consumer SubsidiesProvided Under the Act,GAO-14-705T(Washington, D.C.: July 23, 2014). GAO is alsoconducting additional work that will provide information on Healthcare.gov and itssupporting systems.

Healthcare.gov andSupporting Systems

Healthcare.gov Website

FFM System

Federal Data Services Hub
http://www.gao.gov/products/GAO-14-705Thttp://www.gao.gov/products/GAO-14-705Thttp://www.gao.gov/products/GAO-14-705Thttp://www.gao.gov/products/GAO-14-705T


12/66


13/66


Figure 2: Overview of Healthcare.gov and Selected Supporting Systems


14/66


15/66


Figure 3: Key Contract Phases and Selected Activities

To implement and oversee PPACAs marketplace and private healthinsurance requirements, HHS established the Office of ConsumerInformation and Insurance Oversight (OCIIO) in April 2010 as part of theHHS Office of the Secretary. In January 2011, the OCIIO moved to CMSand became the Center for Consumer Information and InsuranceOversight (CCIIO). Within CMS, establishment of the federal marketplacewas managed by CCIIO, with responsibilities shared with the Office ofInformation Services (OIS), and the Office of Acquisition and GrantsManagement (OAGM). HHSs acquisition process for the data hub andFFM task orders involved multiple participants, including:

The contracting officer.The contracting officer has the authority toenter into, administer, and/or terminate contracts and make relateddeterminations. The contracting officer is responsible for ensuringperformance of all necessary actions for effective contracting,ensuring compliance with the terms of the contract, and safeguardingthe interests of the United States in its contractual relationships.

The contracting officers representative (COR).The CORalsoreferred to as the contracting officers technical representativeisdesignated in writing by the contracting officer to perform specifictechnical or administrative functions. Unlike the contracting officer, aCOR has no authority to make any commitments or changes that

affect price, quality, quantity, delivery, or other terms and conditions ofthe contract and cannot direct the contractor or its subcontractors tooperate in conflict with the contract terms and conditions.

The government task leader (GTL).The GTL is a representative ofthe program office who assists the COR and is responsible for day-to-day technical interaction with the contractor. The GTL is alsoresponsible for monitoring technical progress, including the


16/66


surveillance and assessment of performance, and performingtechnical evaluations as required, among other responsibilities.

CMS undertook the development of Healthcare.gov and its relatedsystems without effective planning or oversight practices, despite facing anumber of challenges that increased both the level of risk and the needfor oversight. According to CMS program and contracting officials, thetask of developing a first-of-its-kind federal marketplace was a complexeffort that was exacerbated by compressed time frames and changingrequirements. CMS contracting officials explained that meeting project

deadlines was a driving factor in a number of acquisition planningactivities, such as the selection of a cost-reimbursement contract, thedecision to proceed with the contract award process before requirementswere stable, and the use of a new IT development approach. Theseactions increased contract risks, including the potential for cost increasesand schedule delays, and required enhanced oversight. However, CMSdid not use information available to provide oversight, such as qualityassurance surveillance plans. CMS also missed opportunities to considerthe full range of risks to the acquisition by not developing a writtenacquisition strategy, even though the agency was required to do so. As aresult, key systems began development with risks that were not fullyidentified and assessed.

Meeting project deadlines was a driving factor in a number of acquisitionplanning activities. HHS had 15 months between enactment of PPACAand the agencys request for proposal to develop requirements for theFFM and data hub. In a prior report on acquisition planning at severalagencies, including HHS, we found that the time needed to completesome pre-solicitation planning activitiessuch as establishing the needfor a contract, developing key acquisition documents such as therequirements document, the cost estimate, and, if required, theacquisition plan; and obtaining the necessary review and approvals

could be more than 2 years. The time needed depended on factors thatwere present for this acquisition including complexity of the requirements,

OversightWeaknesses andLack of Adherenceto Planning

RequirementsCompoundedAcquisition PlanningChallenges

Acquisition PlanningActivities Carried HighLevels of Risk for theGovernment


17/66


political sensitivity, and funding.13

The FFM and data hub task orders were issued under an existing 2007contract for enterprise system development. This approach wasreasonable in these circumstances because, according to contractingofficials, the task orders could be issued more quickly than using a fulland open competitive approach. The 2007 contract had been awarded to16 vendors who were then eligible to compete for individual task orders.The 2007 contract was specifically established to improve efficiency when

new IT requirement arosesuch as the federal marketplacedevelopment. The 16 eligible contractors had experience with CMSs ITarchitecture and could come up to speed quickly. The solicitation for the2007 contract sought contractors with experience in software design,development, testing and maintenance in complex systems environmentsto provide a broad range of IT services including planning, design,development, and technical support, among others. Of the 16 eligiblecontractors, four contractors responded with proposals for each system.

CMS program officials noted challenges

developing requirements for a complex, first-of-its-kind system in thesecompressed time frames and indicated that more time was needed.

CMS used a source selection process that considered both cost and non-cost factors. This type of source selection process is appropriate when it

may be in the best interest of the agency to consider award to other thanthe lowest priced offer or the highest technically rated offer.14

13In an August 2011 report, GAO recommended that HHS collect information about the

time frames needed for pre-solicitation acquisition planning activities to establish timeframes for when program officials should begin acquisition planning. This recommendationhas not yet been implemented. A second recommendation from this reportthat HHSensure that agency and component guidance clearly define the role of cost estimating andincorporating lessons learned in acquisition planning, as well as specific requirements forwhat should be included in documenting these elements in the contract filehas beenimplemented. See GAO,Acquisition Planning: Opportunities to Build Strong Foundationfor Better Services Contracts,

In this case,

the request for proposals indicated that cost and non-cost factors wereweighted equally. The non-cost factors for technical evaluation includedlogical and physical design, project plan, and staffing plan, among others.In addition, CMS considered contractor past performance, but did notinclude that factor in the technical evaluation. CMS determined that theselected contractors for both task orders offered the most advantageouscombination of technical performance and cost.

GAO-11-672(Washington, D.C.: Aug. 9, 2011).

14FAR 15.101-1(a).


18/66


The FAR requires that agencies ensure that requirements for services areclearly defined.15In addition, in our August 2011 review of opportunities to

build strong foundations for better services contracts, we found that well-defined requirements are critical to ensuring the government gets what itneeds from service contractors. We also found that program andcontracting officials at the four agencies we reviewedwhich includedHHSnoted that defining requirements can be a challenging part ofacquisition planning and is a shared responsibility between program andcontracting officials.16Further, our March 2004 report on software-

intensive defense acquisitions found that while requirements for a projectcan change at any point, officials must aggressively managerequirements changes to avoid a negative effect on project results, suchas cost increases and schedule delays.17

In order to begin work quickly, CMS proceeded with the award processbefore FFM contract requirements, which included general technicalrequirements for system development, were finalized. For example, at thetime the task order was issued, CMS did not yet know how many stateswould opt to develop their own marketplaces and how many wouldparticipate in the federally facilitated marketplace, or the size of theiruninsured populations.

18

15FAR 37.503(a).

CMS also had not completed rulemaking

necessary to establish key marketplace requirements. The statement of

work for the FFM acknowledged a number of these unknownrequirements, for example, stating that requirements for state supportwere not fully known and the FFM system must be sufficiently robust toprovide support of state exchange requirements at any point in the lifecycle. In addition, the FFM statement of work noted that therequirements related to a number of FFM services would be finalized aftercontract award, including services related to all three main functionalareaseligibility and enrollment, financial management, and planmanagementas well as system oversight, communication, andcustomer service.

16GAO-11-672.

17GAO, Defense Acquisitions: Stronger Management Practices Are Needed to ImproveDODs Software-Intensive Weapon Acquisitions.GAO-04-393(Washington, D.C.: Mar. 1,2004).

18Under PPACA, states had to obtain CMS approval to establish and operate their ownmarketplaces for 2014 by January 1, 2013. 42 U.S.C. 18041(c)(1)(B).

Requirements for Developingthe FFM System Were NotWell Defined When the TaskOrder Was Issued
http://www.gao.gov/products/GAO-11-672http://www.gao.gov/products/GAO-11-672http://www.gao.gov/products/GAO-11-672http://www.gao.gov/products/GAO-04-393http://www.gao.gov/products/GAO-04-393http://www.gao.gov/products/GAO-04-393http://www.gao.gov/products/GAO-04-393http://www.gao.gov/products/GAO-11-672


19/66


The technical requirements for both the FFM and data hub weredeveloped by CMS staff with contractor support19and documented in astatement of work for each task order.20Both statements called for the

contractor to design a solution that is flexible, adaptable, and modular toaccommodate the implementation of additional functional requirementsand services. However, according to CMS program officials,requirements for data hub development were more clearly defined at thetime that task order was issued than FFM requirements. These officialsalso stated that, prior to issuing the task order, CMS was able to developa prototype for the data hub and a very clear technical framework to guidethe contractor, but due to still-changing requirements, CMS could notprovide the same guidance for FFM development. We have previouslyfound that unstable requirements can contribute to negative contractoutcomes, including cost overruns and schedule delays.21

In response to unsettled requirements, CMS contracting officials selecteda type of cost reimbursement contract known as a cost-plus-fixed-feecontract for both the FFM and data hub task orders. According to theFAR, these contracts are suitable when uncertainties in requirements orcontract performance do not permit the use of other contract types.

22

19The Program Support Center in the Office of the Secretary awarded a contract inSeptember 2010 on behalf of OCIIO to develop the business architecture for the FFM anddata hub. This contract was transferred to CMS when OCIIO became CCIIO within CMS.

Under a cost reimbursement contract, the government pays all of thecontractors allowable incurred costs to the extent prescribed in the

contract. These contracts are considered high risk for the governmentbecause of the potential for cost escalation and because the governmentpays a contractors allowable cost of performance regardless of whetherthe work is completed. In recent years, the federal government has taken

20According to CMS contracting and program officials, requirements development was

done simultaneously for the two task orders, with the potential for both task orders to beawarded to the same contractor.

21See, for example,GAO-11-672and GAO, Department of Homeland Security: BetterPlanning and Assessment Needed to Improve Outcomes for Complex Service

Acquisitions,GAO-08-263(Washington, D.C.: Apr. 22, 2008). In this report GAO madethree recommendations to the Secretary of Homeland Security to achieve improvedoutcomes for its service acquisitions.

22FAR 16.301-2(a)(1) & (2).

CMS Used a Contract TypeThat Carried Risk for theGovernment and Required

Additional Oversight


20/66


steps to minimize the use of cost reimbursement contracts.23

In our November 2007 report on internal control deficiencies at CMS, wefound that certain contracting practices, such as the frequent use of costreimbursement contracts, increased cost risks to CMS because CMS didnot implement sufficient oversight for cost reimbursement contracts atthat time.

While

CMSs use of the cost-plus-fixed-fee contract type may have been areasonable choice under the circumstances, the related risks increasedthe need for oversight.

24However, in planning documents for the two task orders, CMS

acknowledged the increased responsibilities and risks associated withmanaging a cost reimbursement contract and included a number ofoversight elements in the task orders to support contract oversight andmanage risks. These elements included contract deliverables such asearned value management reports,25monthly financial and project status

reports, and a quality assurance surveillance plan. 26

Both task orders required that a quality assurance surveillance plan beprovided within 45 days after award. This plan is intended to ensure thatsystematic quality assurance methods are used in administration of thecontract and provides for government oversight of the quality, quantity,and timeliness of contractor performance. The FAR requires that contract

quality assurance be performed as may be necessary to determine that

23In 2009, the President released a Memorandum (M-09-25) calling for a reduction in theuse of high-risk contracts. In 2012, DOD, GSA, and NASA adopted as final rule amendingthe FAR to implement a section of the Duncan Hunter National Defense Authorization Actfor Fiscal Year 2009 that addresses the use and management of cost-reimbursementcontracts. 77 Fed. Reg. 12925 (Mar. 2, 2012).

24See GAO, Centers for Medicare and Medicaid Services: Internal Control DeficienciesResulted in Millions of Dollars of Questionable Contract Payments,GAO-08-54(Washington, D.C.: Nov. 15, 2007). We made nine recommendations to the Administratorof CMS to improve internal control and accountability in the contracting process and

related payments to contractors. All nine recommendations have been implemented.25Earned value management is a project management tool that integrates project scopewith cost, schedule and performance elements for purposes of project planning andcontrol. FAR 2.101.

26The task orders also required additional oversight mechanisms, such as CMSgovernance milestone reviews. These included a Project Baseline Review intended toassess the project plans scope, schedule and risk, and an Operational Readiness Reviewto determine if the product was ready to support business operations.


21/66


the supplies or services conform to contract requirements.27

To help manage compressed time frames for FFM and data hubdevelopment, CMS program officials adopted an iterative IT developmentapproach called Agile that was new to CMS. Agile development is amodular and iterative approach that calls for producing usable software insmall increments, sometimes referred to as sprints, rather than producinga complete product in longer sequential phases.

However, we

found that the quality assurance surveillance plans were not used toinform oversight. For example, contracting and program officials,including the COR and contracting officer, were not sure if the qualityassurance surveillance plan had been provided as required by the FFMand data hub task orders. Although a copy was found by CMS staff inJune 2014, officials said they were not aware that the document had beenused to review the quality of the contractors work. Instead, CMS programofficials said they relied on their personal judgment and experience todetermine quality.

28The Office of

Management and Budget issued guidance in 2010 that advocated the useof shorter delivery time frames for federal IT projects, an approachconsistent with Agile.29However, CMS program officials acknowledged

that when FFM and data hub development began in September 2011,they had limited experience applying an Agile approach to CMS IT

projects. In 2011, CMS developed updated guidance to incorporate theAgile IT development approach with its IT governance model, but thatmodel still included sequential reviews and approvals and requireddeliverables at pre-determined points in the project. In our July 2012report, we found a number of challenges associated with introducing Agilein the federal environment.30

27FAR 46.401.

Specifically, we found that it was difficult to

ensure that iterative projects could follow a standard, sequential approach

28In 2012, GAO reported on the use of Agile methods in the Federal government. See

GAO, Software Development: Effective Practices and Federal Challenges in ApplyingAgile Methods,GAO-12-681(Washington, D.C.: July 27, 2012). In this report we madeone recommendation to the Federal CIO Council to encourage the sharing of thesepractices.

29OMB, 25 Point Implementation Plan to Reform Federal Information TechnologyManagement(Washington, D.C.: Dec. 9, 2010) and Immediate Review of FinancialSystems IT Projects, M-10-26 (Washington, D.C.: June 28, 2010).

30GAO-12-681.

CMS Selected a New ITDevelopment Approachto Save Time, butIncreased Risks


22/66


and that deviating from traditional procedural guidance to follow Agilemethods was a challenge. We also reported that new tools and trainingmay be required, as well as updates to procurement strategies.Therefore, the new approach that CMS selected in order to speed workalso carried its own implementation risks.

While a number of CMSs acquisition planning actions were taken in aneffort to manage acquisition challenges, CMS missed opportunities tofully identify and mitigate the risks facing the program. HHS acquisitionpolicy requires the development of a written acquisition strategy for major

IT investments, such as the FFM system.31According to HHS policy, anacquisition strategy documents the factors, approach, and assumptionsthat guide the acquisition with the goal of identifying and mitigating risks.32

According to program officials, the acquisition planning process for theFFM and data hub task orders began in 2010, prior to HHSs decision tomove its Office of Consumer Information and Insurance Oversight(OCIIO) to CMS, and continued into early 2011. Program officials stated

that the planning process included discussions of an acquisition strategy.However, CMS program and contracting staff did not complete therequired acquisition strategy for FFM and data hub development.

According to contracting and program officials, CMS has not beenpreparing acquisition strategies for any of its major IT acquisitions, not

just those related to systems supporting Healthcare.gov. This is alongstanding issue. In November 2009 we found deficiencies in CMScontract management internal controls practices such as the failure tofollow existing policies and the failure to maintain adequate

HHS provides a specific acquisition strategy template that requiresdetailed discussion and documentation of multiple strategy elements,including market factors and organizational factors, among others.

31HHS defines a major IT investment as an IT investment that involves one or more of thefollowing: (1) has total planned outlays of $10 million or more in the budget year; (2) is forfinancial management and obligates more than $500,000 annually; (3) is otherwisedesignated by the HHS CIO as critical to the HHS mission or to the administration of HHSprograms, finances, property or other resources; (4) has life-cycle costs exceeding $50million.

32HHS Acquisition Policy Memorandum 2009-05, Attachment A.

CMS Did Not Fully Adhereto HHS AcquisitionPlanning Requirements

and Missed Opportunitiesto Capture and ConsiderRisks Important to thePrograms Success


23/66


documentation in contract files.33

Contracting officials from OAGM explained that at CMS the majority ofacquisition planning is done by the program office and OAGM begandiscussions of the upcoming task orders related to Healthcare.gov and itssupporting systems with program officials in February 2011. In June2011, OAGM accepted a Request for Contract packagea set ofdocuments used to request and approve a contract actionfrom the

program office. The package documents some elements of an acquisitionstrategy. Specifically, it indicated the type of contract to be used and theselected contract approach; however, the documents do not include therationale for all decisions and did not address a number of planningelements required in HHS acquisition strategy, such as organizationalfactors, technological factors, and logistics.

According to CMS contracting officials,

CMS is planning steps to strengthen the agencys program and projectmanagement, including training related to the acquisition strategyrequirement.

In the absence of an acquisition strategy, key risks and plans to managethem were not captured and considered as required. The acquisitionstrategy provides an opportunity to highlight potential risk areas andidentify ways to mitigate those risks. For example, the strategy guidance

requires the consideration of organizational factors that includemanagement and their capabilities, available staff and their skills, andrisks associated with the organizational structure. Organizational factorswere a potential risk area for these projects because the CMSorganizations responsible for the FFM and data hub experiencedsignificant changes just prior to and during the planning period.Specifically, OCIIO was established in 2010 and integrated into CMS inJanuary 2011, just prior to the beginning of planning discussions withOAGM. According to CMS contracting and program officials, some of the246 OCIIO staff transitioned to the new CCIIO and others joined CMSsOffice of Information Services (OIS) and OAGM. In the context of theseorganizational changes and the other considerable project risks, the

33GAO, Centers for Medicare and Medicaid Services: Deficiencies in ContractManagement Internal Control Are Pervasive,GAO-10-60(Washington, D.C.: Oct. 23,2009) andGAO-08-54.InGAO-10-60we made 10 recommendations to the Administratorof CMS, OAGM management, and the Secretary of HHS to ensure adherence to FARrequirements and other control objectives. Nine of the 10 recommendations have beenimplemented.
http://www.gao.gov/products/GAO-10-60http://www.gao.gov/products/GAO-10-60http://www.gao.gov/products/GAO-10-60http://www.gao.gov/products/GAO-08-54http://www.gao.gov/products/GAO-08-54http://www.gao.gov/products/GAO-08-54http://www.gao.gov/products/GAO-10-60http://www.gao.gov/products/GAO-10-60http://www.gao.gov/products/GAO-10-60http://www.gao.gov/products/GAO-10-60http://www.gao.gov/products/GAO-08-54http://www.gao.gov/products/GAO-10-60


24/66


acquisition strategy could have been a powerful tool for risk identificationand mitigation. By failing to adhere to this requirement, CMS missedopportunities to explain the rationales for acquisition planning activitiesand to fully capture and consider risks important to the success of theprogram.

CMS incurred significant cost increases, schedule slips, and reducedsystem functionality in the development of the FFM and data hubsystemsprimarily attributable to new and changing requirementsexacerbated by inconsistent contract oversight. From September 2011 to

February 2014, estimated costs for developing the FFM increased froman initial obligation of $56 million to more than $209 million; similarly, datahub costs increased from an obligation of $30 million to almost $85million. New and changing requirements drove cost increases during thefirst year of development, while the complexity of the system and reworkresulting from changing CMS decisions added to FFM costs in the secondyear. In addition, required design and readiness governance reviewswere either delayed or held without complete information and CMS didnot receive required approvals. Furthermore, inconsistent contractoroversight within the program office and unclear roles and responsibilitiesled CMS program staff to inappropriately authorize contractors to expendfunds.

Obligations for both the FFM and data hub rose significantly during thetwo-and-a-half-year development period, with the FFM task orderincreasing almost four-fold, from $55.7 million obligated when issued inlate 2011 to more than $209 million obligated by February 2014. Similarlythe data hub task order almost tripled, increasing from $29.9 million to$84.5 million during the same period.34

Figure 4 shows FFM and data hub

obligation growth during this time.

34As of April 2014, CMS had obligated more than $103 million for the data hub, whichincludes post-development operational and maintenance functions.

ChangingRequirements and

Oversight GapsContributed toSignificant CostGrowth, ScheduleDelays, and ReducedCapabilities duringFFM and Data HubDevelopment

FFM and Data Hub TaskOrders ExperiencedSignificant Increases


25/66


Figure 4: Cumulative Obligation Increases for the Task Orders for Developing the Federally Facilitated Marketplace Systemand Federal Data Services Hub

Source: GAO analysis of Centers for Medicare & Medicaid Services data. | GAO-14-694

Dollars obligated

Interactive Graphic Rollover green and light blue circles for more information. Please see appendix II for the print version.

09/2011$55.7 million

8/2012$91.5 million

04/2013

$119.2 million

09/2013

$196 million

09/2011

$29.9 million01/2012

$25.7 million

9/2012$48.7 million

06/2013$53.7 million

09/2013$84.5 million

02/2014

$209.6 million

2012 2013 2014

2012 2013 2014

Federally Facilitated Marketplace System

Federal data services hub


26/66


Development cost increases for the FFM and data hub were due to acombination of factors, including costs associated with adding orchanging requirements. For example, CMS was aware that a number ofkey business requirements for the FFM and data hub would not be knownuntil after the task orders were issued in September 2011, and itacknowledged some of these uncertainties in the statements of work,such as noting that the actual number of states participating in the federalmarketplace and the level of support each state required was notexpected to be known until January 2013. We previously found in March2004 that programs with complex software development experienced costincreases and schedule delays when they lacked controls over their

requirements, noting that leading software companies found changingrequirements tend to be a major cause of poor software developmentoutcomes.35

Subsequent modifications to the FFM and data hub task orders show thecosts associated with adding requirements beyond those initialuncertainties. For example, CMS obligated an additional $36 million to theFFM and $23 million to the data hub in 2012, in large part to addressrequirements that were added during the first year of development, suchas increasing infrastructure to support testing and production and addinga transactional database. Some of these new requirements resulted from

regulations and policies that were established during this period. Forexample, in March 2012, federal rulemaking was finalized for keymarketplace functions, resulting in the need to add services to support thecertification of qualified health plans for partnership marketplace states.Other requirements emerged from stakeholder input, such as a newrequirement to design and implement a separate server to processinsurance issuers claims and enrollment data outside of the FFM. CMSprogram officials said that this resulted from health plan issuers concernsabout storing proprietary data in the FFM. The FFM and data hub taskorders were both updated to include this requirement in 2012, which wasinitially expected to cost at least $2.5 million.

During the second year of development, from September 2012 toSeptember 2013, the number of task order modifications and dollarsobligated for the development of the FFM and data hub continued toincrease. New requirements still accounted for a portion of the costs, but

35GAO-04-393.

New and ChangingRequirements DroveCost Increases throughoutSystem Development

System Complexities andRework Further Added to FFMCosts in the Second Year


27/66


the second-year increases also reflected the previously unknowncomplexities of the original requirements and associated rework,particularly for the FFM. For example, according to the FFM contractor,one of the largest unanticipated costs came from CMS directions topurchase approximately $60 million in software and hardware that wasoriginally expected to be provided by another Healthcare.gov contractor.Most of these costs were added through task order modifications in 2013.

In April 2013, CMS added almost $28 million to the FFM task order tocover work that that was needed because of the increasingly complexrequirements, such as additional requirements to verify income for

eligibility determination purposes. The FFM contractor said some of thesecosts resulted from CMSs decisions to start product development beforeregulations and requirements were finalized, and then to change the FFMdesign as the project was ongoing, which delayed and disrupted thecontractors work and required them to perform rework. In addition, CMSdecisions that appeared to be final were reopened, requiring work thathad been completed by the contractor to be modified to account for thenew direction. This included changes to various templates used in theplan management module and the application used by insurance issuers,as well as on-going changes to the user interface in the eligibility andenrollment module. According to the FFM contractor, CMS changed thedesign of the user interface to match another part of the system aftermonths of work had been completed, resulting in additional costs anddelays. In November 2012, the contractor estimated that the additionalwork in the plan management module alone could cost at least $4.9million.

By contrast, CMS program officials explained that the data hub generallyhad more stable requirements than the FFM, in part due to its functionsbeing less technically challenging and because CMS had had more timeto develop the requirements. While the obligations for the data hub alsoincreased at the same rate as the FFM in the first year of development,they did so to a lesser degree during the second year. According to the

data hub contractor, these increases were due to CMS-requestedchanges in how the work was performed, which required additionalservices, as well as hardware and software purchases.


28/66


In addition to increased costs, the FFM and data hub experiencedschedule delays, which contributed to CMS holding incompletegovernance oversight reviews and eventually reduced the capabilities itexpected the FFM contractor to produce by the October 1, 2013,deadline.

CMS initially established a tight schedule for reviewing the FFM and data

hub development in order to meet the October 1, 2013, deadline forestablishing enrollment through the website. Each task order lists the keygovernance reviews that the systems were required to meet as theyprogressed through development.

The FFM and data hub task orders initially required the contractors to beprepared to participate in most of the CMS governance reviewsincluding a project baseline and final detailed design reviewswithin thefirst 9 months of the awards. This would allow CMS to hold the finalreview needed to implement the systemsoperational readinessatleast 6 months before the Healthcare.gov launch planned for October 1,2013. In April 2013, CMS extended the requirements analysis and designphase. According the CMS program officials, requirements were stillchanging and more time was needed to finalize the FFM design. As aresult, CMS compressed time frames for conducting reviews for thetesting and implementation phases. Under the revised schedule, thecontractor had until the end of September 2013immediately prior to thedate of the planned launchto complete the operational readinessreview, leaving little time for any unexpected problems to be addresseddespite the significant challenges the project faced. Figure 5 shows theschedule of planned and revised development milestone reviews in theFFM and data hub task orders.

CMS ExperiencedSchedule Delays,Conducted IncompleteGovernance OversightReviews, and DelayedSome Capabilities for theFFM and Data Hub

CMS Delayed ScheduledGovernance Reviews,Reducing Time Available forFFM and Data Hub Testingand Implementation Reviews


29/66


Figure 5: Planned Schedule of Development Milestone Reviews in the Federally Facilitated Marketplace System and FederalData Services Hub Task Orders


30/66


The four reviews shown in figure 5architecture, project baseline, finaldetailed design, and operational readiness are among those requiredunder the exchange life cycle framework, the governance model CMSspecifically designed to meet the need to quickly develop the FFM anddata hub using the Agile development approach.36

Despite the revised FFM schedule, it is not clear that CMS held all of thegovernance reviews for the FFM and data hub or received the approvalsrequired by the life cycle framework. The framework was developed toaccommodate multiple development approaches, including Agile. Asenior CMS program official said that although the framework was usedas a foundation for their work, it was not always followed throughout thedevelopment process because it did not align with the modified Agile

approach CMS had adopted. CMS program officials explained that theyheld multiple reviews within individual development sprintsthe shortincrements in which requirements are developed and software isdesigned, developed, and tested to produce a building block for the finalsystem. However, CMS program officials indicated that they were focusedon responding to continually changing requirements which led to themparticipating in some governance reviews without key information beingavailable or steps completed. Significantly, CMS held a partial operationalreadiness review for the FFM in September 2013, but development andtesting were not fully completed and continued past this date. As a result,CMS launched the FFM system without the required verification that itmet performance requirements.

The life cycle

framework requires technical reviews at key junctures in the developmentprocess, such as a final detailed design review to ensure that the designmeets requirements before it is developed and tested. To accommodatedifferent development approaches, the life cycle framework allowsprogram offices leeway regarding how some reviews are scheduled andconducted, permitting more informal technical consultations when holding

a formal review would cause delays. However, the framework requiresthat the four governance or milestone reviews be approved by a CMSgovernance board.

36The Exchange Life Cycle framework was also designed to support other IT efforts forthe marketplaces, such as state-based exchanges. This framework was derived fromCMSs Integrated IT Investment & System Life Cycle Framework and HHSs EnterprisePerformance Life Cycle. During the course of the contracts, the Exchange Life CycleFramework was replaced with CMSs Expedited Life Cycle process.

Some Governance ReviewsWere Not Fully Conductedor Approved


31/66


Furthermore, the life cycle framework states that CMS must obtaingovernance-board approval before the systems proceed to the nextphase of development, but we did not see evidence that any approvalswere provided. CMS records show that CMS held some governancereviews, such as design readiness reviews. However, the governanceboards findings identified outstanding issues that needed to beaddressed in subsequent reviews and they were not approved to move tothe next stage of development.

By March 2013, CMS recognized the need to extend the task ordersperiods of performance in order to allow more time for development. CMS

contract documents from that time estimated that only 65 percent of theFFM and 75 percent of the data hub would be ready by September 2013,when development was scheduled to be completed. Recognizing thatneither the FFM nor the data hub would function as originally intended bythe beginning of the initial enrollment period, CMS made trade-offs in anattempt to provide necessary system functions by the October 1, 2013,deadline. Specifically, CMS prioritized the elements of the system neededfor the launch, such as the FFM eligibility and enrollment module, andpostponed the financial module, which would not be needed until post-enrollment. CMS also delayed elements such as the Small BusinessHealth Options Program marketplace, initially until November 2013, andthen until 2015. See figure 6 for the modules completion status as of theend of the task order in February 2014.

Figure 6: Completion Status of Federally Facilitated Marketplace System Modules atthe End of the Task Order, February 2014

CMS Postponed Some FFMCapabilities to Meet Deadlines


32/66


In September 2013, CMS extended the amount of time allotted fordevelopment under the FFM and data hub task orders, which accountedfor the largest modifications. The additional obligations$58 million forthe FFM and $31 million for the data hubincluded some new elements,such as costs associated with increasing FFM capacity needed to supportanticipated internet traffic, but our review of the revised statements ofwork show that the additional funding was primarily for the time needed tocomplete development work rather than new requirements.

After the FFM was launched on October 1, 2013, CMS took a number ofsteps to respond to system performance issues through modifications to

the FFM task order. These efforts included adding more than $708,000 tothe FFM task order to hire industry experts to assess the existing systemand address system performance issues. CMS also greatly expanded thecapacity needed to support internet users, obligating $1.5 million toincrease capacity from 50 terabytes to 400 terabytes for the remainder ofthe development period. While CMS program officials said that thewebsites performance improved, only one of the three key componentsspecified in the FFM task order was completed by the end of the taskorders development period. (See figure 6.) According to programofficials, the plan management module was complete, but only some ofthe elements of the eligibility and enrollment module were provided andthe financial management remained unfinished.

We identified approximately 40 instances during FFM development inwhich CMS program staff inappropriately authorized contractors toexpend funds totaling over $30 million because those staff did not adhereto established contract oversight roles and responsibilities. Moreover,CMS contract and program staff inconsistently used and reviewedcontract deliverables on performance to inform oversight.

The FFM task order was modified in April 2013 to add almost $28 millionto cover cost increases that had been inappropriately authorized by CMS

program officials in 2012.37

37The cost increase was originally estimated to be $32 million in December 2012, but wasnegotiated to the lesser figure in the subsequent contract modification.

This issue also affected the data hub taskorder, which had an estimated $2.4 million cost increase over the sameperiod. In November 2012, the FFM contractor informed CMS of a

Unclear ContractOversight ResponsibilitiesExacerbated FFM andData Hub Cost Growth

CMS Staff InappropriatelyAuthorized Contractors to

Expend Funds


33/66


potential funding shortfall due to work and hardware that CMS programofficials had directed the contractor to provide. The FAR provides that thecontracting officer is the only person authorized to change the terms andconditions of the contract. Further, other government personnel shall notdirect the contractor to perform work that should be the subject of acontract modification.38The federal standards for internal control also

state that transactions and significant events need to be authorized andexecuted by people acting within the scope of their authority, to ensurethat only valid transactions to commit resources are initiated.39

CMS documents show that the cost growth was the result of at least 40instances in which work was authorized by various CMS programofficials, including the government task leader (GTL)who is responsiblefor day-to-day technical interaction with the contractorand other staffwith project oversight responsibilities, who did not have the authority toapprove the work. This was done without the knowledge of thecontracting officer or the contracting officers representative. Thisinappropriately authorized work included adding features to the FFM anddata hub, changing designs in the eligibility and enrollment module, andapproving the purchase of a software license. CMS later determined thatthe work was both necessary and within the general scope of the taskorder but the cost of the activities went beyond the estimated cost amount

established in the order and thus required a modification.

A senior CMS program official described a three-pronged approach tocontract oversight that involved various CMS offices, including the CORand GTL in the program offices, and the contracting officer in OAGM. TheCOR and GTL were assigned overlapping responsibilities for monitoringthe contractors technical performance, but CMSs guidance to clarify theirroles did not fully address the need to ensure that directions given tocontractors were appropriate. CMS program officials said the guidancewas issued in 2006, several years before the FFM and data hub taskorders were issued. The guidance generally noted that CORs areresponsible for financial and contractual issues while GTLs have day-to-

day technical interactions with the contractors. However, the guidance didnot clarify the limitations on CORs and GTLs authorities, such as not

38FAR 43.102(a).

39GAO, Standards for Internal Control in the Federal Government.GAO/AIMD-00-21.3.1(Washington, D.C.: November 1999).

Inappropriate AuthorizationsDue to Unclear OversightResponsibilities
http://www.gao.gov/products/GAO/AIMD-00-21.3.1http://www.gao.gov/products/GAO/AIMD-00-21.3.1http://www.gao.gov/products/GAO/AIMD-00-21.3.1http://www.gao.gov/products/GAO/AIMD-00-21.3.1


34/66


providing contractors with technical direction to perform work outside thescope of the contract.

CMS program officials also described difficulties clarifying oversightresponsibilities in organizations that were new to CMS, which contributedto the inappropriately authorized work. Program responsibilities wereshared between CCIIO, which was primarily responsible for developingbusiness requirements, and the information technology staff in OIS,where the GTL and COR were located. CCIIO was relatively new to CMShaving been incorporated shortly before the FFM and data hub taskorders were issued. OIS program officials explained that CCIIO was not

as experienced with CMSs organization and did not strictly follow theirprocesses, including for oversight. CMS documents show that there wereconcerns about inappropriate authorizations prior to the cost growthidentified in late 2012, as officials in the OIS acquisition group hadrepeatedly cautioned other OIS and CCIIO staff about inappropriatelydirecting contractors.

Furthermore, CMS program officials said that CCIIO staff did not alwaysunderstand the cost and schedule ramifications associated with thechanges they requested. As the FFM in particular was in the phase ofdevelopment in which complexities were emerging and multiple changeswere needed, there were a series of individual directions that, in sum,exceeded the expected cost of the contract. As a result of theunauthorized directions to contractors, the CMS contracting officer had toreact to ad hoc decisions made by multiple program staff that affectedcontract requirements and costs rather than directing such changes byexecuting a contract modification as required by the FAR.

In April 2013, shortly after the inappropriate authorizations and relatedcost increases for the FFM and data hub task orders were identified, asenior contracting official at CMS sent instructions on providing technicaldirections to contractors to the program offices that had been involved inthe authorizations and to CMS directors in general. Specifically, the

program offices were reminded to avoid technical direction tocontractorsparticularly when there is an immediate need for criticalfunctionswhich might constitute unauthorized commitments by thegovernment. This instruction has not been incorporated into existingguidance on the roles and responsibilities of the CORs and GTLs. CMScontracting and program officials also reported additional steps to bolstercontract oversight such as reminding the FFM contractor not to undertakeactions that result in additional costs outside of the statement of workwithout specific direction from the contracting officer.


35/66


It was not always clear which CMS officials were responsible forreviewing and accepting contractor deliverables, including items such asthe required monthly status and financial reports and the qualityassurance surveillance plan that aid the government in assessing thecosts and quality of the contractors work. According to contractingofficials, reviewing such deliverables helped to provide the additionaloversight that cost-reimbursable task orders require per the FAR toreduce risks of cost growth. However, particularly in the first year of FFMdevelopment, contract documentation shows repeated questions aboutwho was responsible for reviewing the deliverables and difficulties findingthe documents. Both task orders were ultimately modified to require that

deliverables be provided to the contracting officer, who had previouslyjust been copied on transmittal letters, in addition to the program office.

In September 2012, the COR oversight function transferred to theacquisition group within CMSs OIS and a new COR was assigned tomanage both the FFM and data hub task orders. A CMS program officialexplained that the acquisition group typically fulfills the COR role for CMScontracts and that it had been unusual for those functions to be providedby another office. Upon assuming oversight responsibilities, the new CORcould not locate a complete set of FFM and data hub deliverables and theoriginal COR was unable to provide them. Instead, the new COR had torequest all monthly status and financial reports directly from thecontractors. When the new COR began reviewing the reports in the fall of2012, he said he noticed that the FFM contractor had not been projectingthe burn rate, a key measure that shows how quickly money is beingspent. The COR asked the contractor to provide the figures in November2012, at which point the cost growth was identified, even though thecontract had been modified in August 2012 to add almost $36 million tothe task order. We found that the burn rate was not included in earlierreports, but its absence had gone unnoticed due to ineffective contractoversight. In November 2007, we had found internal control deficienciesat CMS related to the inadequate review of contractor costs.40

40GAO-08-54

CMS Provided InconsistentOversight of ContractPerformance


36/66


CMS took limited action to address significant FFM contractorperformance issues as the October 1, 2013, deadline for establishingenrollment through the website neared, and ultimately hired a newcontractor to continue FFM development. Late in the developmentprocess, CMS became increasingly concerned with CGI Federalsperformance. In April and November 2013, CMS provided writtenconcerns to CGI Federal regarding its responsiveness to CMSs directionand FFM product quality issues. In addition, in August 2013, CMS wasprepared to take action to address the contractors performance issuesthat could have resulted in withholding of fee; however, CMS ultimatelydecided to work with CGI Federal to meet the deadline. CMS contracting

and program officials stated that the contract limited them to onlywithholding fee as a result of rework. Ultimately, CMS declined to payonly about $267,000 of requested fee. This represented about 2 percentof the $12.5 million in fee paid to CGI Federal. Rather than pursue thecorrection of performance issues with CGI Federal, in January 2014 CMSawarded a new one-year contract to Accenture Federal Services for $91million to continue FFM development. This work also has experiencedcost increases due to new requirements and other enhancements, withcosts increasing to over $175 million as of June 2014.

CMS generally found CGI Federal and QSSIs performance to besatisfactory in September 2012, at the end of the first year ofdevelopment. CMS noted some concerns related to FFM contractorperformance, such as issues completing development and testing ontime; however, CMS attributed these issues to the complexity of the FFMand CMSs changing requirements and policies.41

41CMS reported this information in the Contractor Performance Assessment ReportingSystem the government-wide evaluation reporting tool for all past performance reportson contracts and orders. This report card assesses a contractors performance andprovides a record, both positive and negative, on a given contractor during a specificperiod of time. Each assessment is based on objective facts and supported by programand contract management data, such as cost performance reports, customer comments,quality reviews, technical interchange meetings, financial solvency assessments,construction/production management reviews, contractor operations reviews, functionalperformance evaluations, and earned contract incentives.

Further, according to

program officials, during the first year of FFM development, few definedproducts were to be delivered as requirements and the systems designwere being finalized. For example, as previously identified in this report,under the revised FFM development schedule the final detailed design

CMS IdentifiedSignificant ContractorPerformance Issuesfor the FFM TaskOrder but TookLimited Action

CMS Deemed EarlyContractor PerformanceSatisfactory and TookLimited Action to AddressSignificant ContractorPerformance Issues asthe Deadline Neared


37/66


review for the FFMa key development milestone review to ensure thatthe design meets requirements before it is developed and tested, wasdelayed from June 2012 to March 2013. Therefore, CMS had limitedinsight into the quality of CGI Federals deliverables during the first yearas development and testing of certain FFM functionality had not yet beencompleted. CMS found QSSIs performance satisfactory in September2012. CMS program officials told us that they did not identify significantcontractor performance issues during data hub development, and that thedata hub generally worked as intended when Healthcare.gov waslaunched on October 1, 2013.

During the second year of development, which began in September 2012,CMS identified significant FFM contractor performance issues as theOctober 1 deadline approached (see figure 7). In April 2013, CMSidentified concerns with CGI Federals performance, including notfollowing CMSs production deployment processes and failing to meetestablished deadlines, as well as continued communication andresponsiveness issues. To address these issues, the contracting officersrepresentative (COR) sent an email to CGI Federal outlining CMSsconcerns and requesting that CGI Federal provide a plan for correctingthe issues moving forward. CMS accepted CGI Federals mitigation plan.The plan included changes, according to CGI Federal officials, toaccommodate CMS communication practices, which CGI Federalbelieved to be the root cause of some of the CMS-identified issues. CMScontracting officials said that they were satisfied with CGI Federalsoverall mitigation approach, which seemed to address the performanceissues that CMS had identified at that time.

CMS Identified Significant FFMContractor Performance Issuesas the Deadline Approached,but CMS Opted Against TakingRemedial Contractual Actionsat That Time


38/66


Figure 7: Federally Facilitated Marketplace System Contractor Performance during Development

Notes:a

The development period of performance ended in February 2014, and CMS chose not to exerciseoption years provided for in the task order.

According to CMS program officials, they grew increasingly concernedwith CGI Federals performance late in the development process in Juneand July 2013 as the scheduled launch date approached. Specifically,CMS program officials identified concerns with FFM technical and codequality during early testing of the enrollment process. The initial task orderschedule had called for the FFMs development and test phase to becomplete by this point, but these efforts were delayed in the revisedschedule. CMS program officials explained that they identif ied issuessuch as inconsistent error handling, timeouts, and pages going blank.Overall, more than 100 defects were identified, which resulted in delayswhile CGI Federal worked to correct them. According to CGI Federalofficials, the code reflected the instability of requirements at that time.However, once requirements were more stable, after October 2013, the


39/66


contractor was able to quickly make improvements to the FFMsperformance.

In August 2013, CMS contracting and program officials decided to takeformal action to address their concerns with CGI Federals performanceby drafting a letter to the contractor. Specifically, CMS identified concernswith the contractors code quality, testing, failure to provide a keydeliverable, and scheduled releases not including all agreed uponfunctionality. The letter further stated that CMS would take aggressiveaction, such as withholding fee in accordance with the FAR, if CGIFederal did not improve or if additional concerns arose. However, the

contracting officer withdrew the letter one day after it was sent to CGIFederal, after being informed that the CMS Chief Operating Officerpreferred a different approach. CMS contracting and program officials toldus that, rather than pursue the correction of performance issues, theagency determined that it would be better to collaborate with CGI Federalin completing the work needed to meet the October 1, 2013, launch. CMScontracting officials told us that the agency did not subsequently take anyremedial actions to address the issues outlined in the August 2013 letter.

By early September 2013, CMS program officials told us that theybecame so concerned about the contractors performance that CMSprogram staff moved their operations to CGI Federals location inHerndon, Virginia to provide on-site direction leading up to the FFMlaunch. CMS had identified issues such as deep-rooted problems withcritical software defects during testing and demonstration of the productand CGI Federals inability to perform quality assurance adequatelyincluding full testing of software. According to CMS program officials,CMS staff members worked on-site with CGI Federal for several weeks toget as much functionality available by October 1, 2013, as possible,deploying fixes and new software builds daily.

After the Healthcare.gov launch on October 1, 2013, CMS contracting

officials began preparing a new letter detailing their concerns regardingcontractor performance which was sent to CGI Federal in November2013. In its letter, CMS stated that CGI Federal had not met certainrequirements of the task order statement of work, such as FFMinfrastructure requirements including capacity and infrastructureenvironments, integration, change management, and communicationissuessome of which had been previously expressed in writing to CGIFederal. In addition, CMS stated that some of these issues contributed toproblems that Healthcare.gov experienced after the October 1, 2013

CMS Took Some

Actions to Hold the FFMContractor Accountableafter the Healthcare.govLaunch


40/66


launch. CMSs letter also requested that CGI Federal provide a plan toaddress these issues. CGI Federal responded in writing, stating that itdisagreed with CMSs assertion that CGI Federal had not met therequirements in the FFM statement of work. In its letter, CGI Federalstated that delays in CMSs establishment and finalization ofrequirements influenced the time available for development and testing ofthe FFM. CGI Federal further stated that disruptions to its performance asa result of delays in finalizing requirements were compounded by thescheduled launch date, which resulted in CMS reprioritizing tasks andcompressing time frames to complete those tasks. CGI Federal officialssaid they did not provide a formal plan for addressing CMSs concerns

because they regarded them as unfounded, but agreed to work with CMSto avoid future issues and optimize the FFMs performance.42

In addition, after the October 1, 2013, launch, CMS contracting officialstold us that they provided additional FFM oversight by participating indaily calls with CGI Federal on the stability of the FFM and the status ofCGI Federals work activities. Contracting officials told us that theincreased oversight of FFM development helped to fix things morequickly. Further, the COR increasingly issued technical direction letters toclarify tasks included in the FFM statement of work and focus CGIFederals development efforts. 43

CMS contracting and program officials explained that they found it difficultto withhold the contractors fee under FAR requirements. As discussedearlier in this report, the development work for the FFM was conductedthrough a cost-plus-fixed-fee task order, through which the government

For example, CMS issued several

technical direction letters to CGI Federal in October 2013, directing CGIFederal to follow the critical path for overall performance improvement ofthe FFM, purchase software licenses, and collaborate with otherstakeholders, among other things. According to program officials, writtentechnical direction letters issued by the COR had more authority thantechnical direction provided by the GTL.

42CMS and CGI Federal exchanged a series of letters regarding CGI Federalsperformance under the FFM task order in November 2013. In its initial response to CMSsNovember 2013 letter, CGI Federal addressed each issue identified by CMS

GAO Report: HEALTHCARE.GOV - Ineffective Planning and Oversight Practices Underscore the Need for...

Documents

Transcript of GAO Report: HEALTHCARE.GOV - Ineffective Planning and Oversight Practices Underscore the Need for...