Applying a Life-Applying a Life-Cycle Approach to Cycle Approach to

Information QualityInformation QualityARRA Performance Data ARRA Performance Data

Office of the Chief Financial OfficerOffice of the Chief Financial Officerand theand the

Office of Environmental Office of Environmental InformationInformation

May 2009May 2009

What are performance What are performance data?data?

Data collected/used to monitor and Data collected/used to monitor and report a program’s accomplishments report a program’s accomplishments (its activities, outputs, outcomes) (its activities, outputs, outcomes) relative to a target relative to a target

Performance data can be Performance data can be environmental, administrative environmental, administrative (count), activity, survey, you name it(count), activity, survey, you name it

ARRA of 2009ARRA of 2009

OMB’s guidance regarding quality OMB’s guidance regarding quality assuring ARRA-performance assuring ARRA-performance

information:information: “…“…agencies are responsible for pre-agencies are responsible for pre-

dissemination review of all information that dissemination review of all information that will appear on Recovery.gov. All agencies will appear on Recovery.gov. All agencies must ensure all reporting related to ARRA must ensure all reporting related to ARRA funding is complete and accurate and funding is complete and accurate and complies with the agency's Information complies with the agency's Information Quality Act guidelines. Each agency will Quality Act guidelines. Each agency will provide its point-of-contact for quality provide its point-of-contact for quality information on its Recovery.gov page.”information on its Recovery.gov page.”

Other accountability Other accountability objectives in ARRA:objectives in ARRA:

Public benefits of funds reported Public benefits of funds reported clearly, accurately, in timely mannerclearly, accurately, in timely manner

Funds used for authorized purposeFunds used for authorized purpose No unnecessary project delays and No unnecessary project delays and

cost overrunscost overruns Programs meet their goals and Programs meet their goals and

targets, and contribute to targets, and contribute to improvement of broad economic improvement of broad economic indicatorsindicators

OMB Guidance on OMB Guidance on Performance Measures Performance Measures

under ARRA under ARRA Performance measures are expected quantifiable Performance measures are expected quantifiable

outcomes…with each outcome supported by a outcomes…with each outcome supported by a corresponding quantifiable output(s)—to indicate corresponding quantifiable output(s)—to indicate incremental change against present level of incremental change against present level of performanceperformance

Programs must specify results reporting period Programs must specify results reporting period (e.g., monthly, quarterly), measurement (e.g., monthly, quarterly), measurement methodology, and how the results will be made methodology, and how the results will be made accessible to public. accessible to public.

Use of existing measures will allow the public to Use of existing measures will allow the public to see the marginal performance impact of the see the marginal performance impact of the Recovery Act investments and will reduce burden.Recovery Act investments and will reduce burden.

ARRA Performance Measures—ARRA Performance Measures—Available in Draft:Available in Draft:

Developed by Programs receiving ARRA fundsDeveloped by Programs receiving ARRA funds Mix of Outcomes/OutputsMix of Outcomes/Outputs[1][1] Many are existing planning and annual commitment Many are existing planning and annual commitment

measuresmeasures ““Green” measures and economic efficiency measures (e.g., Green” measures and economic efficiency measures (e.g.,

jobs created/preserved)jobs created/preserved) Measures developed by OIG to detect waste, abuse and Measures developed by OIG to detect waste, abuse and

fraudfraud

[1][1] Number of existing heavy duty diesel engines (including Number of existing heavy duty diesel engines (including school bus engines) that have been retrofitted, replaced or school bus engines) that have been retrofitted, replaced or retired (DERA); Number of states that have awarded all of retired (DERA); Number of states that have awarded all of the 20% green project reserve (CW SRF); Number of jobs the 20% green project reserve (CW SRF); Number of jobs leveraged (Brownfields); Number of criminal, civil and leveraged (Brownfields); Number of criminal, civil and administrative actions (OIG)administrative actions (OIG)

Performance Measure/Data Quality Performance Measure/Data Quality

PlanPerformance Measures Development•Program performance questions•Data/information sources & definitions •Acceptance criteria/data quality objectives (award conditions)

Data/Info QA Plan•Sampling & lab analysis plans/protocols•Reporting & record keeping•Database design and operation•Data input protocol•Statistical analysis & presentation

Roles & responsibilities

CheckCheck quality processes for adequacy

Review oversight results for compliance with QA plans

Document issues, problems & deviations

Characterize validity of a priori assumptions about measure/data quality

Assess implications for intended use of/goals for performance measure/data

Identify options to improve performance measure/data quality

ActEvaluate measure/data quality options

Assess costs & benefits of options

Revise measures & QA plans as warranted

DoData/info collection & analysis•Note issues/deviations

Data recording, storage, access, aggregation, etc.

Oversight, corrective actions & implications

Analyze & interpret

IQG pre-dissemination review

Present/communicate

Management Action Plan IManagement Action Plan I

Explains EPA’s Quality Program’s role under Explains EPA’s Quality Program’s role under ARRAARRA

References existing grants, contracts, and References existing grants, contracts, and IAs receiving ARRA funds and specifies IAs receiving ARRA funds and specifies which have which have environmentalenvironmental data collection data collection components with Quality Management Plans components with Quality Management Plans and/or Quality Assurance Project Plansand/or Quality Assurance Project Plans

Good check for ARRA environmental Good check for ARRA environmental measures that have robust Plan/Do quality measures that have robust Plan/Do quality assuranceassurance

Management Action Plan IIManagement Action Plan II Broadens definition of performance data to include Broadens definition of performance data to include

administrative data and other typesadministrative data and other types Lays out “State of QA” for each performance measureLays out “State of QA” for each performance measure Inventory data standards and definitions, management Inventory data standards and definitions, management

controls for reporting and oversight, data flowcontrols for reporting and oversight, data flow Identifies gaps in QA processes & responsibilitiesIdentifies gaps in QA processes & responsibilities Requires programs to self-certify completeness of data Requires programs to self-certify completeness of data

quality processes through a signed Pre-dissemination quality processes through a signed Pre-dissemination ReviewReview

Bottom line purpose (of PDCA, too): Bottom line purpose (of PDCA, too): To minimize To minimize end-of-pipe errors in reporting performance end-of-pipe errors in reporting performance results!results!

Pat’s topicsPat’s topics

Where we are/were headed when Where we are/were headed when ARRA intervened – larger ARRA intervened – larger vision/contextvision/context

What MAP II represents compared to What MAP II represents compared to the whole effort: limited scope but the whole effort: limited scope but progress in involving the PM progress in involving the PM communitycommunity

Progress outside of MAP IIProgress outside of MAP II Opportunities for participationOpportunities for participation

Performance Performance MeasurementMeasurement

Goal – to further QA in all OMB-related Goal – to further QA in all OMB-related performance measures for EPAperformance measures for EPA

Beyond performance data reporting Beyond performance data reporting from ARRA-funded programsfrom ARRA-funded programs

Broaden to an Agency-wide QA contextBroaden to an Agency-wide QA context Plan, Do, Check, Act – quality life cyclePlan, Do, Check, Act – quality life cycle Information Quality Guidelines (IQGs)Information Quality Guidelines (IQGs) CIO 2106 – new Quality PolicyCIO 2106 – new Quality Policy

ARRA ProgressARRA Progress

MAP I QA of environmental data in MAP I QA of environmental data in ARRA financial instrumentsARRA financial instruments

MAP II QA adds MAP II QA adds performance data performance data and informationand information for ARRA measures for ARRA measures

Communication with and input from:Communication with and input from: NPO & Regional QA communitiesNPO & Regional QA communities IT/IM community (some affected by IT/IM community (some affected by

ARRA)ARRA) Stimulus Steering Committee - planners Stimulus Steering Committee - planners

from 5 program officesfrom 5 program offices

What’s NOT in MAP IIWhat’s NOT in MAP II QA of performance QA of performance measuresmeasures in the life cycle in the life cycle

framework (ARRA measures are framework (ARRA measures are setset, ICR , ICR issue)issue)

IT/IM QA for IT/IM QA for allall types of data bases types of data bases Enterprise systems (CERCLIS, SDWIS)Enterprise systems (CERCLIS, SDWIS) End-user systems (spreadsheets)End-user systems (spreadsheets)

Guidance from IT/IM QA standards referencesGuidance from IT/IM QA standards references GAO auditing protocols (FISCAM)GAO auditing protocols (FISCAM) Corporate accounting firmsCorporate accounting firms OEI policyOEI policy

QA review/approval of the proceduresQA review/approval of the procedures

Performance Measurement Performance Measurement QAQA

Plan with stakeholders, internal & Plan with stakeholders, internal & externalexternal

Apply the QA graded approach to assureApply the QA graded approach to assure Adequate QA resources for oversightAdequate QA resources for oversight Performance measure & data are adequate Performance measure & data are adequate

for intended usefor intended use Document, validate & verify data flow Document, validate & verify data flow

processesprocesses Use life cycle “Check” and “Act” for Use life cycle “Check” and “Act” for

evolution to environmental outcomesevolution to environmental outcomes

Opportunities for InputOpportunities for Input IT/IM Managers: Performance data quality IT/IM Managers: Performance data quality

expectations forexpectations for Large systemsLarge systems Small spreadsheet scale systemsSmall spreadsheet scale systems

QA Managers: QA adequacy ofQA Managers: QA adequacy of Non-environmental data and information used Non-environmental data and information used

to support the budget processto support the budget process Environmental data with additional intended Environmental data with additional intended

uses in performance reportinguses in performance reporting Performance reporting processPerformance reporting process

More Opportunities for More Opportunities for Management InputManagement Input

Senior managers: views on certificationSenior managers: views on certification Program managersProgram managers

Consult with IT/IM & QA managers on PDCA Consult with IT/IM & QA managers on PDCA application to design performance measures application to design performance measures more efficiently & apply graded approachmore efficiently & apply graded approach

To ensure performance data are adequateTo ensure performance data are adequate To ensure data flow procedures are effectiveTo ensure data flow procedures are effective To ensure verification & validation are To ensure verification & validation are

transparent and documentedtransparent and documented