NSF 2018 LARGE FACILITIES WORKSHOP

2018 National Science Foundation April 30 – May 2, 2018 Alexandria, VA

6B377BNSF 2018 Large Facilities Workshop

Page ii

TA B L E O F C O N T E N T S

1. INTRODUCTION ............................................................................................................................ 1 1.1. Purpose & Intended Audience ................................................................................................................................. 1 1.2. Format .......................................................................................................................................................................... 1 1.3. Location ........................................................................................................................................................................ 2

2. PRESENTATION HIGHLIGHTS ........................................................................................................ 3 2.1. Monday, April 30, 2018 .......................................................................................................................................... 3 2.2. Tuesday, May 1, 2018 ............................................................................................................................................. 6 2.3. Wednesday, May 2, 2018 ................................................................................................................................... 10

3. PARTICIPANT SUMMARY DATA ................................................................................................. 13

4. SURVEY, IDEA EXCHANGE, AND WORKSHOP DEBRIEF – SUMMARY OF RESULTS & ACTIONABLE RECOMMENDATIONS ........................................................................................... 15 4.1. Key Survey Results .................................................................................................................................................. 15 4.2. Consolidated Actionable Recommendations – Topical Sessions, Idea Exchange, Workshop Debrief, and

Survey ........................................................................................................................................................................ 16

5. OVERALL CONCLUSIONS AND ACTIONABLE RECOMMENDATIONS......................................... 19

APPENDIX A: AGENDA ...................................................................................................................... 20

APPENDIX B: PARTICIPANT INFORMATION ..................................................................................... 26

APPENDIX C: PRESENTATION SLIDES ................................................................................................ 32

6B377BNSF 2018 Large Facilities Workshop

Page 1

6B377BNSF 2018 Large Facilities Workshop 1. INTRODUCTION

1.1. Purpose & Intended Audience The Large Facilities Workshop (LFW) is an annual event hosted by NSF’s Large Facilities Office (LFO). The workshop is a collaborative forum for NSF’s Large Facilities community, intended to provide participants with continuous learning and information sharing opportunities. Desired outcomes to advance the NSF mission of scientific research at Large Facilities include: • Sharing knowledge and experience with best practices and common challenges that arise for both NSF and its

Large Facilities • Discussing new initiatives and engaging the Large Facilities community for input • Demonstrating project management, operations, and business-related tools and techniques • Expanding our community of practice and connecting colleagues from large complex scientific facilities to share

information, experience, and expertise

The workshop targets the following professionals who have a role in the development, planning, construction, or operation of large complex scientific facilities: • Facility and operations managers • Property and maintenance managers • Cooperative agreement and contracting officers • Business professionals • Project managers, estimators, schedulers, and controllers • Principal investigators and scientists • Engineers and construction managers • Policy and oversight stewards

Workshop attendees and speakers may earn credits in the form of continuous learning points towards various certifications, such as AACE International’s Cost Estimating Professional or Earned Value Professional, state Professional Engineering License, Project Management Institute’s Project Management Professional, and Federal Acquisition Certifications for Program and Project Managers and Contracting Officer’s Technical Representative.

1.2. Format The agenda and presentations from the workshop are included in Appendices A and D and available on the 2018 NSF LFW web site. The list of participants and an overview of the range of participants are included in Appendix B and Section 3.

The agenda was structured to provide a diverse selection of relevant topics for the Large Facilities community. Various approaches for information sharing were used, such as plenary presentations and break-out sessions featuring presentations and panel discussions. The workshop was also structured to present NSF’s evolving oversight approach at a high level, followed by more detailed discussions of key elements.

The workshop strived to include topics applicable to all facility life cycle stages, with a majority of sessions focused on operating facilities. Speakers were encouraged to present and elicit “Best Practices” and “Actionable Recommendations,” which were captured by dedicated note takers and compiled in this report.

The session “Idea Exchange Open Forum for Award Recipients” provided an opportunity for facilities to engage one other and freely share ideas, to provide critical feedback to NSF to improve future workshops and large facility efforts.

6B377BNSF 2018 Large Facilities Workshop

Page 2

1.3. Location The 2018 Large Facilities workshop took place in Alexandria, VA, where the new National Science Foundation Headquarters is located, just outside Washington, DC. The workshop is held every year, with even year workshops in the Washington, DC area and odd year workshops in the field at a Large Facility.

6B377BNSF 2018 Large Facilities Workshop

Page 3

2. PRESENTATION HIGHLIGHTS This section provides brief summaries of the presentations, including their purpose and main points. Key takeaways from interactive discussions that can be characterized as “Best Practices” or “Actionable Recommendations” are also noted. Where possible, organizations responsible for following up on any actions are identified. This section is intended to summarize and supplement the detailed expert slide presentations, not to transcribe all comments and discussions. More detailed but rough notes on the various viewpoints expressed during the discussions are available and may be requested from LFO. These notes will be used by LFO to help inform the Action Recommendations.

The NSF Major Facilities Knowledge Sharing Gateway provides links to all workshop presentations. Direct links to each presentation are provided below.

2.1. Monday, April 30, 2018 LFO Opening Remarks – Status, Evolving Oversight & Other Developments

Speaker: Matthew Hawkins, Head, Large Facilities Office (LFO), NSF Description: NSF welcomed participants and provided an update on significant happenings over the

past year related to NSF’s research infrastructure. Link to Slides: https://www.largefacilitiesworkshop.com/wp-

content/uploads/2018/04/OpeningPresentation.pdf

Lessons for the Future Speaker: Adam Cohen, President and CEO, Associated Universities Incorporated

Description: Experience in projects and facilities is a balance sheet of successes and failures. With all the current concerns about budget uncertainty, has the era of several “billion dollar” class R&D facilities progressing together ended? Adam Cohen provided insights on how experiences from the past might provide valuable lessons for the future planning and eventual construction and operations of the next generation of facilities.

Link to Slides: No slides were used during the session. Best Practices: • Mega, high-risk projects should not commit to a total project cost estimate too soon. This

can help avoid having it “cast in stone.” Having to make adjustments later may create an appearance of inadequate planning.

• Delivering projects on-time and on-budget creates a framework of trust and credibility between funding stakeholders and execution agencies/organizations.

• Give project managers the authority to execute the project efficiently and effectively, without unnecessary oversight or constraints.

Fast Radio Bursts Speaker: Nipuni Palliyaguru, Radio Astronomer, Arecibo Observatory

Description: This session discussed fast radio burst observations across several NSF facilities, with focus on the observations of FRB 121102 using the Arecibo Observatory.

Link to Slides: https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/04/FastRadioBursts.pdf

Impacts of Hurricane Maria on the Arecibo Observatory Speaker: Francisco Cordova, Director, Arecibo Observatory

Description: This session focused on describing the impact hurricane Maria had on the Arecibo Observatory. It also discussed best practices for hurricane preparation and response to major weather events.

Link to Slides: https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/ImpactsHurricaneMaria.pdf

6B377BNSF 2018 Large Facilities Workshop

Page 4

Woodrow Wilson Bridge Replacement Project: A Mega-Project Success Story Speaker: Jim Ruddell, Vice President; WSP USA

Description: Jim Ruddell provided an overview of the successful Washington, DC–area Woodrow Wilson Bridge Replacement project. The session highlighted challenges and lessons learned, including application of scheduling best practices from the GAO Schedule Guide.

Link to Slides: https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/04/WoodrowWilsonBridge.pdf

Best Practices: • Manage to schedule and you will manage the budget. • Balance risk between client and contractor. • Consider plotting Early-Late delivery schedule curves to get a visual indication of

schedule float. • Include contract incentives as milestones on project schedule. • Active clients (customer/owner) are needed to resolve issues and roadblocks quickly to

reduce/minimize schedule delay.

Session I: Introduction – Forming, Learning, and Operating in High Functioning Teams: Lessons from the Navy SEALs

Speaker: Larry Yatch, CEO and Founder, Sealed Mindset Description: How do organizations mistake-proof planning and execution? The Navy SEALS plan for

success by identifying possible risks of failure and mitigating the possibilities before they occur. Participants learned how the SEALs use a feedback cycle for systemic planning, after-action assessment, and learning from challenges, to empower their teams and maximize performance. Session I was a motivational introduction to the roles and responsibilities of high-functioning teams and the factors that increase the likelihood of a team’s success. High-functioning teams must master individual performance, team performance, and tactical planning.

Link to Slides: https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/NavySEALSSession1.pdf

Best Practices: • Team development of a desired end state is used for communication and planning to create high functioning teams.

Session II: Overview – Forming, Learning, and Operating in High Functioning Teams: Lessons from the Navy SEALs

Speaker: Larry Yatch, CEO and Founder, Sealed Mindset Description: Larry Yatch presented the five phases of the Navy SEALS Contingency Planning Cycle and

the steps involved for completing each phase. High functioning teams follow this process to define their desired end state, define the courses of action for achieving it, determine contingencies and perform detailed planning. After each mission, they conduct after-action reviews to develop lessons learned for the next iteration.

Link to Slides: https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/NavySEALSSession2.pdf

Best Practices: • Define the Desired End State • Define a course of action. • Determine Contingencies (Planning to fail.) • Conduct detailed planning. Solve problems that do not exist yet. • Conduct an After Action Review (the first thing done after the mission) to develop

lessons learned.

6B377BNSF 2018 Large Facilities Workshop

Page 5

Session III: Hands On Exercise – Forming, Learning, and Operating in High Functioning Teams: Lessons from the Navy SEALs

Speaker: Larry Yatch, CEO and Founder, Sealed Mindset Description: In the third session of this series, Larry Yatch demonstrated how to apply the Contingency

Planning Cycle to real-world problems, either professional or personal. Participants practiced defining Phase 1: The desired end state. This phase strengthens communication among team members and with their management.

Link to Slides: https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/NavySEALSSession3.pdf

Best Practices: • When considering the desired end state consider that why we do something is more

important than how we do it. • Develop a clear concept of operations with measurable effects and the impact. • Clarify the current situation. When we take care of our obligations opportunities arise,

when we do not hold our obligations threats arise. • Define and speak the desired end state.

International Science and G7 Group of Senior Officials (GSO) Framework Speaker: Jim Ulvestad, Chief Officer for Research Facilities, NSF & Matthew Hawkins, Head, Large

Facilities Office, NSF Description: This session introduced the latest Group of Senior Officials (GSO) Global Research

Infrastructure Framework: its history, purpose, and potential benefits for NSF major facilities. Potential case studies around NSF-funded facilities were also discussed.

Link to Slides: https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/InternationalScience.pdf

Enhancing NSF Design and Construction Processes through Spiral Development Speaker: Tony Beasley, Director, National Radio Astronomy Observatory, Lou Lussier National Center

for Atmospheric Research (NCAR), & Tim Cockerill, Texas Advanced Computing Center (TACC)

Description: A panel of practitioners explored the pros and cons of using a “spiral model” to facilitate successful design and construction of major and mid-scale facilities, while working within NSF policies and procedures; including the No Cost Overrun Policy. The discussion included appropriate scaling of Design Stage oversight for mid-scale projects (Section 2.7 of the Large Facilities Manual), as well as major acquisitions.

Link to Slides: https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/EnhancingDesign.pdf

Best Practices: • Use mid-scale as a loop in spiral development of a national facility (e.g., Atacama Large Millimeter/submm Array [ALMA]).

• Use spiral design for software development for scientific web portals (e.g., TACC).

Actionable Recommendations:

• Provide more descriptions of different types or models of development (spiral, waterfall, etc.) but resist the temptation to implement more “guard rails” at lower levels of R&D.

• Use mid-scale projects as a stepping stone, where spirals are relatively fast (on the scale of 4-5 years) per cycle.

• Allow multiple spirals before convening a panel. (What is the appropriate oversight?) • Avoid BSRs on mid-scale recipients. • Be able to articulate goals to reviewers in a spiral development environment. • Implement feedback mechanisms.

6B377BNSF 2018 Large Facilities Workshop

Page 6

• Enable software specialists and scientists to speak the same language. • Be able to move from R&D to MREFC while using spiral development.

Core Competency Requirements for Recipients Speaker: Jeff Zivick, Large Facilities Advisor, NSF

Description: The Federal government is paying greater attention to the core competencies that will be required for Federal program and project managers. This talk highlighted important aspects of some recent initiatives, such as the Program Management Improvement Accountability Act of 2015 (PMIAA) and the National Academy of Public Administration’s (NAPA) recommendations, and how they may impact NSF award recipients. The NAPA report recommended that NSF identify project management requirements for award recipients. OPM is working to develop core competency requirements for Federal Program and Project Managers; NSF is participating on OPM panels. Award recipient management teams are envisioned to include: Principal Investigator, Project Director, Program Manager (PM), Project Controls Manager, and possible Deputy positions. Requirements would be codified in Supplemental Cooperative Agreements and Financial & Administrative Terms and Conditions.

Link to Slides: https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/CoreCompetencyRequirements.pdf

Actionable Recommendations:

• Incorporate competencies in merit review criteria. • Scale PM competencies to the size of the project. • Define competency requirements/recommendations for construction projects vs.

operations.

2.2. Tuesday, May 1, 2018

Federal Budget Outlook Speaker: Beth Blue, Program Analyst, Program Analysis Branch, NSF Budget Division

Description: Beth Blue explained the federal budget process and provided insights on the current budget and an outlook for FY 2019 and FY 2020.

Link to Slides: https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/04/FederalBudgetOutlook.pdf

NSF Office of Inspector General (OIG) – Audit Results & Other Topics Speaker: Elizabeth Goebels, Director of Performance Audits, NSF OIG; Phil Emswiler, Senior

Management Analyst, NSF OIG; Melissa Prunchak, Audit Manager, NSF OIG; & Keith Nackerud, Audit Manager, NSF OIG

Description: Panelists discussed past audit results, risk trends and emerging issues, audit coordination, and the resolution of past issues. A question and answer session was held.

Link to Slides: https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/04/NSFOfficeInspectorGeneral.pdf

Knowledge Management Model for Improving Communication Among Facilities and with NSF

Speaker: Rebecca Yasky, Large Facilities Advisor; LFO, NSF Office of Budget, Finance, & Award Management

Description: This session discussed a proposed model and enhanced tools for sharing best practices and lessons learned, and to strengthen communication among facilities and with NSF. Group discussion provided additional collaborative techniques for knowledge integration:

6B377BNSF 2018 Large Facilities Workshop

Page 7

Knowledge discovery from annual reports and reviews; recipient-peer networking and assistance; and the LFO webpage resources and toolbox.

Link to Slides: https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/04/KnowledgeManagement.pdf

Best Practices: • To encourage people to share lessons learned, start simple with one-page of stories on a mobile-enabled platform.

Actionable Recommendations:

• For future LFO workshop, move up “Ideas” sessions to Day 1. • Make the LFO project list more prominent on the website.

NSF Business Issues I Speaker: Jeff Lupis, Division Director; NSF Division of Acquisition and Cooperative Support, and Bill

Kinser, Branch Chief, Cooperative Support Branch (CSB) Description: This session informed participants of key business issues surrounding: incurred costs audits,

accounting system audits, financial data collection tool, and management fee. Mr. Kinser provided a high-level walkthrough of the Large Facility Financial Data Collection Tool, and discussed plans to establish an FAQ web page and dedicated user support email address. The purposes of incurred cost audits and accounting system audits were explained, along with the criteria and timing for these audits. Recipients were informed of how to resolve findings identified under an NSF-obtained audit. On the topic of NSF fee policy for major facility cooperative agreements, Mr. Lupis explained the context for strengthened NSF business practices; reasons for the new fee policy; major aspects of new fee policy; and next steps.

Link to Slides: https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/CostEstimating.pdf https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/NSFBuisnessIssues1a.pdf

Actionable Recommendations:

• NSF should examine its scoping process for incurred costs audits and accounting system audits. Several awardees noted problems and expressed their perception on the ineffective approaches to scoping which has resulted in: selection of lengthy award periods, non-grouping of related programs, partial year selection over stages (construction vs. operations), and cumbersome mapping against the NSF budget categories.

Society for Science at User Research Facilities (SSURF, formerly NUFO) Speaker: Rich Leonard, Vice-President, Ecology Business, Battelle

Description: This session introduced SSURF’s mission and explained how the society supports the research community across a broad range of disciplines. SSURF focuses on facilities that deliver “Big Science” and the investments required to support them. Representing a community of communities (administrators, users, user committees, facility managers, and government affairs and communications professionals), SSURF delivers a unified message at a national level.

Link to Slides: https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/SocietyScience.pdf

Best Practices: • Conduct training reciprocity pilot program jointly with Office of Science facilities. • Document commonalities among major research facilities.

NSF Proposal and Award Policy Update Speaker: Jean Feldman, Head, Policy Office, Office of Budget, Finance & Award Management;

Division of Institution and Award Support (DIAS)

6B377BNSF 2018 Large Facilities Workshop

Page 8

Description: Jean Feldman provided an update on Federal and NSF policies related to grants management, oversight, procurement, subawards and contracts. She also explained recent changes to the Proposal and Awards Policies and Procedures Guide, new and revised Terms & Conditions and proposal system modernization/FastLane/Grant.gov.

Link to Slides: https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/NSFGrantsPolicyUpdate.pdf

User Engagement and Support Speaker: Forrest Masters, Associate Dean, Professor, Research & Facilities, Department of Civil &

Coastal Engineering, University of Florida Description: This session was a community-led discussion sharing best ways to recruit, engage, and

support users. Link to Slides: https://www.largefacilitiesworkshop.com/wp-

content/uploads/2018/05/UserEngagement.pdf Best Practices: • Facilitate interactions that cross boundaries among people from different fields.

• Increase interactions among users. Actionable

Recommendations: • Highlight large facilities on the NSF home page. • NSF should produce information that facilities can provide to new faculty at orientation

and to graduate students and post-docs. • NSF should produce a highly accessible slide deck on large facilities. Facilities can tailor

it for local needs.

Facility Performance Metrics Speaker: Rich Leonard, Vice-President, Ecology Business, Battelle; Jamie Allan, NSF Program Director,

International Ocean Discovery Program; John LaFemina, Director, Laboratory Planning and Performance Management, Pacific Northwest National Laboratory; Gina Taberski, Interim VP Finance & Administration, University Corporation for Atmospheric Research/National Center for Atmospheric Research (UCAR/NCAR)

Description: Performance metrics are more than a scorecard. Properly implemented, they can inform decisions on how best to keep research infrastructure on the forefront of the scientific endeavor. The session featured examples of operational performance metrics that can be used to describe facility outputs and outcomes, track operational performance (uptime), track operations and maintenance costs, and support budget requests for facility sustainment and improvement.

Link to Slides: https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/FacilitiesPerformanceMetrics.pdf https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/04/FacilityPerformanceMetrics.pdf

Best Practices: • One external source to consider for metrics is the International Facilities Management Organization (http://www.ifma.org/).

• Collected metrics should inform decisions and not just be “interesting.” • Define the questions you want answered first and then design the metrics to answer

those questions. • Use surveys (any medium) to solicit user satisfaction.

NSF Business Issues II Speaker: Eddie Whitehurst, Deputy Branch Chief, Cooperative Support Branch; Matthew Hawkins,

LFO Head, Large Facilities Office Description: This session raised awareness among participants on the terms and conditions and draft

requirements associated with branding. It explained recent updates to cooperative

6B377BNSF 2018 Large Facilities Workshop

Page 9

agreement modifications and supplemental financial & administrative terms and conditions (CA-MSFATC) for major multi-user research facility projects and federally-funded research and development centers. The second topic, signage and branding of NSF major facilities, included a proposed modification of Article 28, Publications, in the CA-FATC, regarding acknowledgment of NSF support; public relations activities and news releases; disclaimer statements; and facility signage.

Link to Slides: https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/NSFBusinessIssuesII.pdf

Actionable Recommendations:

• In the proposed modification, NSF should clarify language on branding in various places (captured in detailed Q&A notes). Attendees provided a helpful read on draft text.

• NSF’s Cooperative Support Branch (CSB) will take a turn on related terms, and circulate via the LF workshop mailing list in the next two weeks.

• CSB to follow-up on cooperative R&D agreements (CRADA) question and NSF process for awardees who are interested in applicability to NSF major facility awards.

Energy Improvements and Alternative Funding Options

Speaker: Scott Wolf, Federal Project Executive, DOE Federal Energy Management Program (FEMP), Oak Ridge National Laboratory; Tom Hattery, Federal Project Executive, DOE FEMP, Oak Ridge National Laboratory; Richard Oram, Operations Manager; California Institute of Technology- LIGO Livingston Observatory; Rob Reid, Facility Management & SAS Director, National Center for Atmospheric Research (NCAR); & Kevin Porter, Large Facilities Advisor, LFO, NSF Office of Budget, Finance, & Award Management

Description: This session discussed how DOE’s successful Federal Energy Management Program can finance energy saving initiatives without up-front capital costs. Benefits to NSF major facilities were cited. NSF major facility case studies were discussed, including examples funded via NSF Research and Related Activity funding for facility Operations and Maintenance versus DOE’s Energy Savings Performance Contracts (ESPCs).

Link to Slides: https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/EnergyImprovements.pdf

Cyberinfrastructure for Facilities Speaker: Manish Parashar, Office Director, NSF Office of Advanced Cyberinfrastructure; Bill Miller,

Science Advisor, NSF Office of Advanced Cyberinfrastructure; Ivan Rodero, Associate Director, Associate Research Professor, Rutgers Discovery Informatics Institute

Description: This interactive session covered: (1) findings and reports from the 2017 Workshop on Cyberinfrastructure for Large Facilities; (2) updates on the NSF Office of Advanced Cyberinfrastructure; and (3) defining expectations/best practices for CI in large facility projects.

Link to Slides: https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/CyberinfrastructureFacilities1.pdf https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/CyberinfrastructureFacilities2.pdf

Best Practices: • Use systems engineering to manage the CI lifecycle and interfaces. • Bake in redundancy to provide high availability. • Lessons learned – be able to trace CI features to requirements and business needs.

6B377BNSF 2018 Large Facilities Workshop

Page 10

2.3. Wednesday, May 2, 2018 Major Facility Innovations and Technology Transfer

Speaker: Tony Beasley, Director, National Radio Astronomy Observatory; Rich Leonard, Vice-President, Ecology Business, Battelle; Drew Weisenberger, Chief Technology Officer, Jefferson Lab; & Matthew Hawkins, Head, Large Facilities Office, NSF

Description: This community-led discussion identified a path forward to develop good practices for conveying the socio-economic benefit from NSF Research Infrastructure investments in the broader U.S. innovation ecosystem and the economy. This included follow-up discussion from the 2017 Large Facilities Workshop on a tailored questionnaire for the academic institutions and non-profits that operate NSF’s major facilities, based on NSF’s Business R&D and Innovation Survey (BRDIS) and other efforts currently underway in Europe. The outcomes will help inform NSF how to develop communications tools like those used successfully by DOE, NASA and other NSF programs.

Link to Slides: https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/MajorFacilityInnovation1.pdf https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/MajorFacilityInnovation2.pdf

Actionable Recommendations:

• Steve Geiger (NRAO) volunteered to review draft Tech Transfer questionnaire discussed during the session.

Lessons Learned: Application to NHERI Experimental Facilities Speaker: Dan Zehner, Facility Scheduling and Operations Coordinator, NHERI Network Coordination

Office; Darren McKay, Site Operations Manager, UC San Diego; Pedro Lomonaco, Director, O.H. Hinsdale Wave Research Laboratory, Oregon State University; & Maryam Refan, Management Site Operations Manager, Florida International University Experimental Facility

Description: A panel discussion on the lessons learned program developed for the Natural Hazards Engineering Research Infrastructure (NHERI). The session shared outcomes from NHERI’s after-action reviews and demonstrate the benefits of an inter-facility lessons learned program.

Link to Slides: https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/NHERILessonsLearned1.pdf https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/NHERILessonsLearned2.pdf https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/NHERILessonsLearned3.pdf https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/NHERILessonsLearned4.pdf

Best Practices: • Ensure adequate monitoring of non-critical path activities along with critical path activities.

• Involve technicians in the review of the drawings (constructability) during the planning phase.

• Ensure all stakeholders have a voice during the various phases of a project. • Verify assumptions with pre-tests, especially when there are many significant

assumptions. • Encourage principal investigators (PIs) to have pre-proposal discussions with the Facility

Operations team. Involvement of the Facility with the PI before proposal submission can help improve cost estimates and reduce changes during the planning phase.

6B377BNSF 2018 Large Facilities Workshop

Page 11

Facilities Condition Assessment Speaker: Robert Reid, Facility Management & SAS Director, NCAR

Description: Robert Reid explained the use of quantitative strategic planning tools for infrastructure lifecycle maintenance at the National Center for Atmospheric Research (NCAR).

Link to Slides: https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/04/FacilitiesConditionAssessment.pdf

Idea Exchange Open Forum for Recipients Speaker: Tim Cockerill, Director of Center Programs, Texas Advanced Computing Center

Description: This open forum provided award recipients an opportunity to share ideas on common areas of interest. Critical feedback for NSF was provided in the form of reasonable, actionable, and documented recommendations. Actionable recommendations were provided to NSF during the open Workshop Debrief in the afternoon, and will be addressed by NSF after the workshop.

Link to Slides: No slides were used during the session.

Activity-Based Budgeting and Cost Estimating for Operations Speaker: Hannah Hansen, Laboratory Business Manager, Laser Interferometer Gravitational

Observatory (LIGO) & Giselle Hepker, Financial Analyst, LIGO Description: Using LIGO’s case study as an example, this session followed the process of interpreting

and applying guidelines from the NSF Large Facilities Manual and GAO Cost Estimating and Assessment Guide, to develop a cost estimating plan for an operations proposal. LIGO explained why it changed from a functional Work Breakdown Structure (WBS) to an activities-based WBS. The LIGO team also explained developing BOE templates and tools for easy reporting.

Link to Slides: https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/Activity-Based-Budgeting.pdf

Cost Estimating for NSF Facility Operations – Reasonable Application of GAO Good Practices

Speaker: Kevin Porter, Large Facilities Advisor, LFO, NSF Office of Budget, Finance, & Award Management; Rob Hengst, Large Facilities Advisor, NSF; Charlie Zeigler, Branch Chief, Cost Analysis and Pre-award Review (CAP) Branch, NSF DIAS; & Tim Kashmer, Grant & Agreement Specialist, NSF

Description: The GAO Cost Estimating and Assessment Guide includes estimating good practices for the entire facility life cycle (Development through Divestment). Using community input, this session explored which of the twelve steps (or practices) of a high-quality cost estimate are most relevant to the Operations Stage, and how NSF can further clarify expectations for proposal submission in Section 4.2 of the LFM.

Link to Slides: https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/CostEstimatingFacility-Operations.pdf

Best Practices: • Differentiate clearly which of the 12 steps apply as instructions, and which items are

criteria. • Have a cost estimating plan. • Use definitive “drop downs” for good justifications of individual activity estimates.

Workshop Debrief – Open Forum & Actionable Recommendations from Idea Exchange Description: This informal open forum enabled workshop attendees to provide direct feedback to NSF’s

LFO and NSF’s Committee on Strategy and Budget (CSB), and provide recommendations

6B377BNSF 2018 Large Facilities Workshop

Page 12

for future improvements and topics. NSF also discussed actionable recommendations developed during the Idea Exchange Open Forum for Award Recipients.

Link to Slides: No slides were used during the session.

6B377BNSF 2018 Large Facilities Workshop

Page 13

3. PARTICIPANT SUMMARY DATA Appendix B contains the list of 168 registered workshop participants (although not everyone participated each day). NSF’s Large Facilities were well represented. A cross section of different professionals were represented. Overall attendance exceeded expectations.

Professions by Organization

Recipients NSF Other TOTAL

Business Professionals 25 4 7 36

Operations Managers 10 0 0 10

Project Managers 18 1 1 20

Executives 18 3 3 24

Scientists, Engineers 18 1 3 22

Other / No Response 17 30 10 56

TOTAL 105 39 24 168

NSF Large Facility Award Recipient Participation

Facility Acronym # of Participants

Arecibo Observatory AO 5

Academic Research Fleet ARF 3

--Regional Class Research Vessel RCRV 2

Cornell High Energy Synchrotron Source CHESS 2

Gemini Observatory Gemini 1

Green Bank Observatory GBO 1

IceCube Neutrino Observatory ICNO 1

International Ocean Discovery Program (JOIDES Resolution) IODP 1

International Ocean Discovery Program (Science Support Office) IODP

Long Baseline Observatory LBO

Large Hadron Collider LHC 4

--A Toroidal LHC Apparatus (ATLAS) Detector Operations and High Luminosity Upgrade Design

ATLAS 1

--A Toroidal LHC Apparatus (ATLAS) Detector Phase I Upgrade ATLAS 3

--Compact Muon Solenoid (CMS) Detector Operations and High Luminosity Upgrade Design

CMS

--Compact Muon Solenoid (CMS) Detector Phase I Upgrade CMS

Laser Interferometer Gravitational-wave Observatory LIGO 5

Large Synoptic Survey Telescope LSST 7

Mid-Scale Facilities and Infrastructure - Advanced Modular Incoherent Scatter Radar

Mid-Scale Facilities and Infrastructure - National Nanotechnology Coordinated Infrastructure

AMIS

National Center for Atmospheric Research NCAR 5

National Ecological Observatory Network NEON 9

6B377BNSF 2018 Large Facilities Workshop

Page 14

Facility Acronym # of Participants

National Geophysical Observatory for Geoscience NGEO 4

--Geodesy Advancing Geosciences and EarthScope 2

--Seismological Facilities for the Advancement of Geoscience and EarthScope

1

National High Magnetic Field Laboratory NHMFL 5

National Optical Astronomy Observatory NOAO

National Radio Astronomy Observatory NRAO 14

--Atacama Large Millimeter/Submillimeter Array

--Very Large Array

National Solar Observatory NSO 5

--Daniel K. Inouye Solar Telescope DKIST 3

National Superconducting Cyclotron Laboratory NSCL 1

Natural Hazards Engineering Research Infrastructure NHERI 17

Ocean Observatories Initiative OOI 8

National Deep Submergence Facility

United States Antarctic Program USAP 2

--Antarctic Infrastructure Modernization for Science AIMS 1

Association of Universities for Research in Astronomy AURA 1

Associated Universities, Incorporated AUI 3

Supercomputing Centers 4

6B377BNSF 2018 Large Facilities Workshop

Page 15

4. SURVEY, IDEA EXCHANGE, AND WORKSHOP DEBRIEF – SUMMARY OF RESULTS & ACTIONABLE RECOMMENDATIONS

4.1. Key Survey Results Overall, how would you rate this entire workshop?

23 responses

How would you rate communication from meeting planners about workshop location, date, registration details and other information?

41 responses

Excellent52%

Very Good44%

Average4%

Excellent Very Good Average Fair Poor

Great75%

Ok23%

Needs Improvement

2%

Great Ok Needs Improvement

6B377BNSF 2018 Large Facilities Workshop

Page 16

How would you rate communication about the purpose and intended audience of the workshop?

41 responses

4.2. Consolidated Actionable Recommendations – Topical Sessions, Idea Exchange, Workshop Debrief, and Survey

Consolidated Actionable Recommendation List Type Source Number Actionable Recommendation Adopted ?

Large Facility Oversight and Initiatives

O&I TS Enhancing NSF Design and Construction Processes through Spiral Development: Provide more descriptions of different types or models of development (spiral, waterfall, etc.) but resist the temptation to implement more “guard rails” at lower levels of R&D.

O&I TS Enhancing NSF Design and Construction Processes through Spiral Development: Use mid-scale projects as a stepping stone, where spirals are relatively fast (on the scale of 4-5 years) per cycle.

O&I TS Enhancing NSF Design and Construction Processes through Spiral Development: Allow multiple spirals before convening a panel. (What is the appropriate oversight?)

O&I TS Enhancing NSF Design and Construction Processes through Spiral Development: Avoid BSRs on mid-scale recipients.

O&I TS Enhancing NSF Design and Construction Processes through Spiral Development: Be able to articulate goals to reviewers in a spiral development environment.

O&I TS Enhancing NSF Design and Construction Processes through Spiral Development: Enable software specialists and scientists to speak the same language.

O&I TS Enhancing NSF Design and Construction Processes through Spiral Development: Be able to move from R&D to MREFC while using spiral development.

O&I TS Core Competency Requirements for Recipients: Incorporate competencies in merit review criteria.

O&I TS Core Competency Requirements for Recipients: Scale PM competencies to the size of the project.

O&I TS Core Competency Requirements for Recipients: Define competency requirements/ recommendations for construction projects vs. operations.

O&I TS Knowledge Management Model for Improving Communication Among Facilities and with NSF: Make the LFO project list more prominent on the website.

O&I TS NSF Business Issues I: NSF should examine its scoping process for incurred costs audits and accounting system audits. Several awardees noted problems and expressed their

Very Clear52%

Clear44%

Unclear4%

Very Clear Clear Unclear

6B377BNSF 2018 Large Facilities Workshop

Page 17

Type Source Number Actionable Recommendation Adopted ? perception on the ineffective approaches to scoping which has resulted in: selection of lengthy award periods, non-grouping of related programs, partial year selection over stages (construction vs. operations), and cumbersome mapping against the NSF budget categories.

O&I TS Society for Science at User Research Facilities (SSURF, formerly NUFO): Conduct training reciprocity pilot program jointly with Office of Science facilities.

O&I TS Society for Science at User Research Facilities (SSURF, formerly NUFO): Document commonalities among major research facilities.

O&I TS User Engagement and Support: Highlight large facilities on the NSF home page.

O&I TS User Engagement and Support: NSF should produce information that facilities can provide to new faculty at orientation and to graduate students and post-docs.

O&I TS User Engagement and Support: NSF should produce a highly accessible slide deck on large facilities. Facilities can tailor it for local needs.

O&I TS NSF Business Issues II: In the proposed modification, NSF should clarify language on branding in various places (captured in detailed Q&A notes). Attendees provided a helpful read on draft text.

O&I TS NSF Business Issues II: NSF’s Cooperative Support Branch (CSB) will take a turn on related terms and circulate via the LF workshop mailing list in the next two weeks.

O&I TS NSF Business Issues II: NSF’s Cooperative Support Branch (CSB) to follow-up on cooperative R&D agreements (CRADA) question and NSF process for awardees who are interested in applicability to NSF major facility awards.

O&I TS Major Facility Innovations and Technology Transfer: Steve Geiger (NRAO) volunteered to review the draft Tech Transfer questionnaire discussed during the session.

General Workshop Planning & Agenda

P&A SY If NSF "branding" of the facilities is important, then education and public outreach leadership from the facilities should be encouraged to attend the workshop.

P&A SY Send the “Final Information Email” about the workshop at least one week in advance.

P&A SY Provide the list of attendees ahead of the workshop.

P&A SY Provide lunch at the workshop.

P&A SY Have more sessions that are attended by all participants.

P&A SY Include more operations relevant topics at the workshop.

P&A SY Include one or two talks about the science that is being conducted at large facilities.

P&A SY Include more female speakers at the workshop.

P&A SY Make the more technical scientific topics workshop breakout sessions.

P&A SY Introduce the large facilities that are in NSF portfolio and have those who was represented at the meeting stand up or something.

P&A SY Cover more subject matter relevant to budget and finance attendees.

P&A SY Provide breakfast at the workshop.

P&A SY Find a more effective way for speakers to end their sessions on time.

P&A SY Include more sessions on NSF audits, policies and procedures.

P&A SY Ensure more NSF Staff attend the workshop reception.

P&A SY Make sure there are more options to power laptops, phones and other devices.

P&A SY Find less expensive group lodging.

P&A SY Provide catered meals if there are limited food options nearby.

P&A SY Would like to have the presentation slides in advance of the workshop.

P&A SY Provide a list of restaurant options in the folder.

P&A SY Consider holding tracks at the workshop based on attendee categories: Directors and Deputy Directors; User Programs; and Budget and Finance.

6B377BNSF 2018 Large Facilities Workshop

Page 18

Type Source Number Actionable Recommendation Adopted ?

P&A SY Enable the app to share ideas/ ask questions in real time with the speaker(s), and audience.

P&A SY Have different colored ribbons for NSF Staff and Presenters, and non-NSF presenters.

Specific Workshop Topics & Speakers

T&S SY Provide more details on LF budget information at the workshop.

T&S SY Include more time for questions during the Policy Update.

T&S SY Include more time for questions during the Business Issues I session.

T&S SY Hold the federal budget outlook presentation in alternative years.

T&S TS Knowledge Management Model for Improving Communication Among Facilities and with NSF: For future LFO workshop, move up “Ideas” sessions to Day 1.

Legend Type Full Name

O&I Large Facility Oversight & Initiatives

P&A General Workshop Planning & Agenda

T&S Specific Workshop Topics & Speakers

Source Full Name

TS Topical Sessions

IE Idea Exchange

WD Workshop Debrief

SY Survey

6B377BNSF 2018 Large Facilities Workshop

Page 19

5. OVERALL CONCLUSIONS AND ACTIONABLE RECOMMENDATIONS

Actionable Recommendation Tracking Table Type Number Actionable Recommendation Owner Organization Action Due Date

Legend Type Full Name

O&I Large Facility Oversight & Initiatives

P&A General Workshop Planning & Agenda

T&S Specific Workshop Topics & Speakers

6B377BNSF 2018 Large Facilities Workshop

Page 20

APPENDIX A: AGENDA

LFO Opening Remarks – Status, Evolving Oversight & Other Developments 8:00am Speaker: Matthew Hawkins Welcome and update on significant happenings over the past year related to NSF Research Infrastructure.

Lessons for the Future 8:30am Speaker: Adam Cohen Experience in projects and facilities is always a balance sheet of successes and failures. My personal experience balance sheet has taught me that the success of a project depends on many factors, some of which can be predicted and planned. The cost, scope, schedule, and planned missions are not the whole story; how you pitch a project, how you ensure continued support, how you execute the management of a project, and how you use the facility all change over time. We are living in times of tremendous budget uncertainty, and there are indications that we will continue to be in a constrained budget environment for some time. Has the era of several “billion dollar” class R&D facilities progressing together ended? Experiences from the past might provide valuable lessons for the future planning and eventual construction and operations of the next generation of facilities.

Fast Radio Bursts 9:45am Speaker: Nipuni Palliyaguru This session will discuss fast radio burst observations across several NSF facilities, with particular focus on the observations of FRB 121102 using the Arecibo Observatory.

Impacts of Hurricane Maria on the Arecibo Observatory 10:15am Speaker: Francisco Cordova This session will focus on describing the impact hurricane Maria had on the Arecibo Observatory. It will also discuss best practices for hurricane preparation and response to major weather events.

Woodrow Wilson Bridge Replacement Project: A Mega-Project Success Story 11:00am Speaker: Jim Ruddell Overview of the successful Washington, DC–area Woodrow Wilson Bridge Replacement Project. Will highlight challenges and lessons learned, including application of scheduling best practices from the GAO Schedule Guide.

Session I: Introduction – Forming, Learning, and Operating in High Functioning Teams: Lessons from the Navy SEALs 11:00am Speaker: Larry Yatch How do organizations mistake-proof planning and execution? The Navy SEALS plan for success by identifying possible risks of failure and mitigating the possibilities before they occur. Larry Yatch will provide an introduction to this valuable tool that can be used to reduce risk in anything from your summer vacation trip to large complex projects. Participants will learn how the SEALs use a feedback cycle for systemic planning, after-action assessment, and learning from challenges to empower their teams and maximize performance. Session is a motivational introduction to the afternoon hands-on breakout sessions and the fundamentals for organizational success.

International Science and G7 Group of Senior Officials (GSO) Framework 1:30pm Speakers: Jim Ulvestad and Matthew Hawkins This International Topic will introduce the latest Group of Senior Officials (GSO) Global Research Infrastructure “Framework”: its history, purpose, and potential benefits for NSF major facilities. Potential case studies around NSF- funded facilities will be discussed.

Session II: Overview – Forming, Learning, and Operating in High Functioning Teams: Lessons from the Navy SEALs 1:30pm Speaker: Larry Yatch Spend an hour going into more detail on the steps in the Navy SEALS Contingency Planning Cycle. (Not required to attend Session III)

Enhancing NSF Design and Construction Processes through Spiral Development 2:40pm Speakers: Tony Beasley, Lou Lussier, and Tim Cockerill Although they may appear onerous, NSF procedures are inherently flexible. A panel of practitioners will take the audience on an exploration of the pros and cons of using a “spiral model” to facilitate successful design and construction of major and mid-scale facilities while working within NSF policies and procedures; including the No Cost Overrun Policy. The discussion will also include appropriate scaling of Design Stage oversight for mid-scale projects (Section 2.7 of the Large Facilities Manual) as well as major acquisitions.

Monday, April 30, 2018

6B377BNSF 2018 Large Facilities Workshop

Page 21

Session III: Hands-on Exercise – Forming, Learning, and Operating in High Functioning Teams: Lessons from the Navy SEALs 2:40pm Facilitator: Larry Yatch Spend a couple of hours applying the planning phase of the Contingency Planning Cycle to your own real-world problem. Employ a series of easy-to-use worksheets to plan for the worst and guarantee success. Leave the workshop with a great plan ready to set to action. (Not required to attend Session II).

Core Competency Requirements for Recipients 4:00pm Speaker: Jeff Zivick The Federal government is paying greater attention to the core competencies that will be required for Federal Program and Project Managers. The Program Management Improvement Accountability Act of 2015 (PMIAA), was signed into law in December 2017. In addition, the 2015

National Academy of Public Administration report on NSF recommended NSF establish minimum project management competencies for its staff and for award recipient staff. In addition, the Project Management Institute is currently revising its Project Manager Competency Development Framework. This talk will highlight some of the important aspects of these initiatives and how they may impact NSF award recipients. Session III continued: Hands-on Exercise – Forming, Learning, and Operating in High Functioning Teams: Lessons from the Navy SEALs 4:00pm Facilitator: Larry Yatch Spend a couple of hours applying the planning phase of the Contingency Planning Cycle to your own real-world problem. Employ a series of easy-to-use worksheets to plan for the worst and guarantee success. Leave the workshop with a great plan ready to set to action. (Not required to attend Session II).

Federal Budget Outlook 8:00am Speaker: Beth Blue Follow the NSF Budget Division through the federal budget process, and gain insights on the current budget and an outlook for FY 2019 and FY 2020.

NSF Office of Inspector General – Audit Results and Other Topics 8:30am Speakers: Elizabeth Goebels, Phil Emswiler, Melissa Prunchak and Keith Nackerud Join the OIG to discuss past audit results, identification of risk trends and emerging issues, audit coordination, and the resolution of past issues. Includes interactive question and answer session.

Knowledge Management Model for Improving Communication Among Facilities and with NSF 10:00am Facilitator: Rebecca Yasky This session will discuss a proposed model and enhanced tools for sharing best practices and lessons learned. Through group discussion, it will solicit feedback on additional collaborative techniques for knowledge integration such as peer assist network, knowledge map, idea exchange, and webpage resources.

NSF Business Issues I 10:00am Speakers: Jeff Lupis and Bill Kinser A discussion on business related issues including fee, incurred cost and accounting audits, and the Data Collection Tool.

Society for Science at User Research Facilities (SSURF, formerly NUFO) 11:10am Facilitator: Rich Leonard Get to know the Society for Science at User Research Facilities and exchange ideas with its leadership on how the society supports the research community.

NSF Proposal & Award Policy Update 11:10am Speaker: Jean Feldman An update on Federal and NSF policies related to grants management, oversight, procurement, subawards and contracts. Learn about recent changes to the Proposal and Awards Policies and Procedures Guide, new and revised Terms & Conditions and proposal system modernization/ FastLane/Grant.gov.

User Engagement and Support 11:40am Facilitator: Forrest Masters Join a community-led discussion sharing best ways to recruit, engage, and support users.

Monday, April 30, 2018

Tuesday, May 1, 2018

6B377BNSF 2018 Large Facilities Workshop

Page 22

Facility Performance Metrics 1:40pm Speakers: Rich Leonard, John LaFemina, Virginia Taberski, and Jamie Allan Performance metrics are more than a scorecard. Properly implemented, they can inform decisions on how best to keep research infrastructure on the forefront of the scientific endeavor. The session will include discussions on examples of operational performance metrics that can be used to describe facility outputs and outcomes, track operational performance (uptime), track operations and maintenance costs, and support budget requests for facility sustainment and improvement.

NSF Business Issues II 1:40pm Speakers: Eddie Whitehurst and Matthew Hawkins A discussion on revised Terms & Conditions and NSF “Branding.”

Energy Improvements and Alternative Funding Options 3:00pm Panelists: Scott Wolf, Tom Hattery, Richard Oram, Rob Reid, and Kevin Porter Learn how DOE’s successful Federal Energy Management Program can finance energy saving initiatives without up-front capital costs. Benefits to NSF major facilities include up-front funding without additional appropriations, guaranteed cost savings and performance, and freeing up money for science. NSF major facility case studies will be discussed, including examples funded via NSF Research and Related Activity funding for facility Operations and Maintenance versus DOE’s Energy Savings Performance Contracts (ESPCs).

Cyberinfrastructure for Facilities 3:00pm Speakers: Manish Parashar, Bill Miller, and Ivan Rodero This interactive session will cover (1) findings and reports from the 2017 Workshop on Cyberinfrastructure (CI) for Large Facilities, (2) updates on the NSF Office of Advanced Cyberinfrastructure, and (3) defining expectations/best practices for CI in large facility projects.

Major Facility Innovations & Technology Transfer 8:00am Facilitators: Tony Beasley, Rich Leonard, Drew Weisenberger, and Matthew Hawkins This community-led discussion will identify a path forward to develop good practices for conveying the socio- economic benefit from NSF Research Infrastructure investments on the broader U.S. innovation ecosystem and the economy. This will include follow-up discussion from 2017 Large Facilities Workshop on a tailored, volunteer questionnaire for the academic institutions and non-profits that operate NSF’s major facilities. The questionnaire will be based on NSF’s Business R&D and Innovation Survey (BRDIS) and other efforts currently underway in Europe. The outcomes will help inform NSF how to develop communications tools like those used successfully by DOE, NASA and other NSF Programs.

Lessons Learned: Application to NHERI Experimental Facilities 8:00am Speakers: Dan Zehner, Darren McKay, Pedro Lomonaco, and Maryam Refan Attend a panel discussion that leads to a hands-on exercise facilitated by representatives of the Large Facilities community on the formal lessons learned program developed for the Natural

Hazards Engineering Research Infrastructure (NHERI).

Facilities Condition Assessment 9:10am Speaker: Robert Reid Discussion of quantitative strategic planning tools for infrastructure lifecycle maintenance at National Center for Atmospheric Research.

Idea Exchange Open Forum for Recipients 10:30am Speaker: Tim Cockerill This session is intended for award recipients and offers an opportunity for the large facilities to share ideas amongst themselves. The goal of the session is to provide critical feedback to NSF in the form of reasonable, actionable, documented recommendations. Actionable recommendations will be provided to NSF during the open Workshop Debrief in the afternoon and addressed by NSF after the workshop.

Activity-Based Budgeting and Cost Estimating for Operations 10:30am Speakers: Hannah Hansen and Giselle Hepker Using LIGO’s case study as an example, follow the process of interpreting and applying guidelines from the NSF Large Facilities Manual and GAO

Wednesday, May 2, 2018

Tuesday, May 1, 2018

6B377BNSF 2018 Large Facilities Workshop

Page 23

Cost Guide to the development of a cost estimating plan for an operations proposal, including updating from a functional Work Breakdown Structure (WBS) to an activities-based WBS, developing BOE templates, and developing tools for easy reporting.

Cost Estimating for NSF Facility Operations – Reasonable Application of GAO Good Practices 11:30am Facilitators: Kevin Porter, Rob Hengst, Charlie Zeigler, and Tim Kashmer The GAO Cost Estimating and Assessment Guide includes estimating good practices for the entire facility life cycle (Development through Divestment). Using community input, this session will explore which of the twelve steps (or practices) of a high-quality cost estimate are most relevant to the Operations Stage and how NSF can further clarify expectations for proposal submission in Section 4.2 of the LFM.

Workshop Debrief – Open Forum & Actionable Recommendations from Idea Exchange 12:15pm An informal open forum for workshop attendees to provide direct feedback to NSF-LFO and NSF-CSB and recommendations for future improvements and topics. Actionable recommendations developed during the Idea Exchange Open Forum for Award Recipients will be discussed with NSF. Refreshments will be served.

7

Wednesday, May 2, 2018

6B377BNSF 2018 Large Facilities Workshop

Page 24

Monday, April 30, 2018

Time Topics

8:00am LFO Opening Remarks – Status, Evolving Oversight & Other Developments Speaker: Matthew Hawkins Location: Edison ABCD

8:30am Lessons for the Future Speaker: Adam Cohen Location: Edison ABCD

9:30am Refreshment Break: Ballroom Foyer 9:45am Fast Radio Bursts

Speaker: Nipuni Palliyaguru Location: Edison ABCD

10:15am Impacts of Hurricane Maria on the Arecibo Observatory Speaker: Francisco Cordova Location: Edison ABCD

10:45am Break 11:00am Woodrow Wilson Bridge Replacement Project:

A Mega-Project Success Story Speaker: Jim Ruddell Location: Edison ABCD

Session I: Introduction – Forming, Learning, and Operating in High Functioning Teams: Lessons from the Navy SEALs Speaker: Larry Yatch Location: Edison EFG

12:00pm Lunch On-Your-Own 1:30pm International Science and G7 Group of Senior Officials

(GSO) Framework Speaker: Jim Ulvestad & Matthew Hawkins Location: Edison ABCD

Session II: Overview – Forming, Learning, and Operating in High Functioning Teams: Lessons from the Navy SEALs Speaker: Larry Yatch Location: Edison EFG

2:30pm Break 2:40pm Enhancing NSF Design and Construction Processes

through Spiral Development Speakers: Tony Beasley, Lou Lussier & Tim Cockerill Location: Edison ABCD

Session III: Hands On Exercise – Forming, Learning, and Operating in High Functioning Teams: Lessons from the Navy SEALs Facilitator: Larry Yatch Location: Edison EFG

3:40pm Refreshment Break: Ballroom Foyer 4:00 – 5:00pm Core Competency Requirements for Recipients

Speaker: Jeff Zivick Location: Edison ABCD

Session III: Continued

5:30 - 6:30pm Networking Reception: Ballroom Foyer

Tuesday, May 1, 2018

Time Topics

8:00am Federal Budget Outlook Speaker: Beth Blue Location: Edison ABCD

8:25am Announcements: Edison ABCD 8:30am NSF Office of Inspector General – Audit Results & Other Topics

Speakers: Elizabeth Goebels, Phil Emswiler, Melissa Prunchak & Keith Nackerud Location: Edison ABCD

6B377BNSF 2018 Large Facilities Workshop

Page 25

Tuesday, May 1, 2018 (continued)

Time Topics

9:45am Refreshment Break: Ballroom Foyer 10:00am Knowledge Management Model for Improving

Communication Among Facilities and with NSF Facilitator: Rebecca Yasky Location: Edison ABCD

NSF Business Issues I Speakers: Jeff Lupis and Bill Kinser Location: Edison EFG

11:00am Break 11:10am Society for Science at User Research Facilities

(SSURF, formerly NUFO) Facilitator: Rich Leonard Location: Edison ABCD

NSF Proposal & Award Policy Update Speaker: Jean Feldman Location: Edison EFG

11:40am User Engagement and Support Facilitator: Forrest Masters Location: Edison ABCD

12:10pm Lunch On-Your-Own 1:40pm Facility Performance Metrics

Panelists: Rich Leonard, John LaFemina, Virginia Taberski & Jamie Allan Location: Edison ABCD

NSF Business Issues II Speakers: Eddie Whitehurst and Matthew Hawkins Location: Edison EFG

2:40pm Refreshment Break: Ballroom Foyer 3:00 – 4:30pm Energy Improvements and Alternative Funding Options

Panelists: Scott Wolf, Tom Hattery, Richard Oram, Rob Reid & Kevin Porter Location: Edison ABCD

Cyberinfrastructure for Facilities Speakers: Manish Parashar, Bill Miller & Ivan Rodero Location: Edison EFG

Wednesday, May 2, 2018

Time Topics

8:00am Major Facility Innovations and Technology Transfer Facilitators: Tony Beasley, Rich Leonard, Drew Weisenberger & Matthew Hawkins Location: Edison D

Lessons Learned: Application to NHERI Experimental Facilities Speakers: Dan Zehner, Darren McKay, Pedro Lomonaco & Maryam Refan Location: Edison ABC

9:00am Break 9:10am Facilities Condition Assessment

Speaker: Robert Reid Location: Edison D

Lessons Learned: Continued

10:10am Refreshment Break: Ballroom Foyer 10:30am Activity-Based Budgeting and Cost Estimating for

Operations Speakers: Hannah Hansen & Giselle Hepker Location: Edison D

Idea Exchange Open Forum for Recipients Speaker: Tim Cockerill Location: Edison ABC

11:30am Cost Estimating for NSF Facility Operations – Reasonable Application of GAO Good Practices Facilitators: Kevin Porter, Rob Hengst, Charlie Zeigler & Tim Kashmer Location: Edison D

12:10pm Break 12:15pm Workshop Debrief – Open Forum & Actionable Recommendations from Idea Exchange

Refreshments will be served. Location: Edison D

6B377BNSF 2018 Large Facilities Workshop

Page 26

APPENDIX B: PARTICIPANT INFORMATION

NATIONAL SCIENCE FOUNDATION 2018 Large Facilities Workshop Alexandria, VA Monday, April 30 – Wednesday, May 2, 2018 Participant List

Muzzammil Ahmed Battelle/Battelle Ecology, Inc.

Peter Bryan Lehigh University

Sohel Ahmed National Science Foundation (NSF)

Daniel Calabrese Large Synoptic Survey Telescope (LSST)/Association of Universities for Research in Astronomy (AURA)

Edward Ajhar National Science Foundation (NSF)

Tim Cockerill Texas Advanced Computing Center (TACC) University of Texas (UT)

Jamie Allan National Science Foundation (NSF)

Adam Cohen Associated Universities, Inc. (AUI)

Giacomo Apadula BDO

Katie Cohen Texas Advanced Computing Center (TACC)/ UT Austin

Heidi Bandulet Canada Foundation for Innovation

Francisco Cordova Arecibo Observatory

Ellen Baptiste Carpenter Battelle Memorial Institute

David Curren Associated Universities, Inc. (AUI)

Anthony Beasley National Radio Astronomy Observatory (NRAO)

Christopher Davis National Science Foundation

Margaret Benoit National Science Foundation

Kim Dawidowski Princeton University

Michael Binford National Science Foundation (NSF)

Kathleen Dedrick Cornell High Energy Synchrotron Source (CHESS) Cornell University

George Blaisdell Merrick & Company

Robert Detrick Incorporated Research Institutions for Seismology (IRIS)

Beth Blue National Science Foundation (NSF)

Deanna DiGiovanna National Science Foundation (NSF)

Debra Booth National High Magnetic Field Laboratory

Patrick Donahue Consortium for Ocean Leadership

Jennifer Bridge University of Florida

Steven Ellis National Science Foundation (NSF)

Gustaaf Brooijmans Columbia University

Philip Emswiler National Science Foundation (NSF)

Thaddeus Brown Associated Universities, Inc. (AUI)

Stephen Ettinger UNAVCO, Inc.

6B377BNSF 2018 Large Facilities Workshop

Page 27

NATIONAL SCIENCE FOUNDATION 2018 Large Facilities Workshop Alexandria, VA Monday, April 30 – Wednesday, May 2, 2018 Participant List

Richard Farnsworth National Ecological Observatory Network (NEON)/Battelle

Matt Hawkins National Science Foundation (NSF)

Jean Feldman National Science Foundation (NSF)

Rob Hengst National Science Foundation (NSF)

Montona Futrell-Griggs National Science Foundation (NSF)

Natalie Henriques Texas Advanced Computing Center (TACC) UT Austin

Arindam Gan Chowdhury Natural Hazards Engineering Research Infrastructure/Florida International University

Giselle Hepker Laser Interferometer Gravitational-wave Observatory (LIGO)

Amy Garwood Columbia University

Amanda Hoffman Lux Consulting Group, Inc.

Ralph Gaume National Science Foundation

Bauke Houtman National Science Foundation

Steven Geiger National Radio Astronomy Observatory (NRAO)

Rex Hunter National Solar Observatory/Association of Universities for Research in Astronomy (AURA)

Margaret Gelman Johnson National Center for Supercomputing Applications

Michael Janus Battelle

John Gillaspy National Science Foundation (NSF)

Laura Jensen Green Bank Observatory

Elizabeth Gobels National Science Foundation (NSF)

Tim Kashmer National Science Foundation (NSF)

Deborah Gronet Association of Universities for Research in Astronomy (AURA)

Ken Kawasaki Coho Consulting Services

Veronica Kinnison Association of Universities for Research in Astronomy (AURA)/Large Synoptic Survey Telescope (LSST)

Xiaofeng Guo National Science Foundation (NSF)

Patrick Haggerty National Science Foundation (NSF)

William Kinser National Science Foundation (NSF)

Hannah Hansen California Institute of Technology

Anne Kornahrens National Science Foundation (NSF)

Mark Hanson Woods Hole Oceanographic Institution

Victor Krabbendam Large Synoptic Survey Telescope (LSST)

Thomas Hattery US Department of Energy

John Kynoch National High Magnetic Field Laboratory

6B377BNSF 2018 Large Facilities Workshop

Page 28

NATIONAL SCIENCE FOUNDATION 2018 Large Facilities Workshop Alexandria, VA Monday, April 30 – Wednesday, May 2, 2018 Participant List

Michele Lacey Texas A&M Research Foundation/Integrated Ocean Drilling Program (IODP)

Forrest Masters Herbert Wertheim College of Engineering University of Florida

John LaFemina Pacific Northwest National Laboratory (PNNL)

Paul Matthias Woods Hole Oceanographic Institution

Mark Lagace Canada Foundation for Innovation

Luther Lau Associated Universities, Inc. (AUI)

Richard McCarthy Laser Interferometer Gravitational-wave Observatory (LIGO)/California Institute of Technology

Janet McCord Texas Advanced Computing Center (TACC)/ UT Austin

Jeff Leithead National Science Foundation (NSF)

Judy McEachern National High Magnetic Field Laboratory Florida State University

Richard Leonard National Ecological Observatory Network (NEON)/Battelle

Peter McEvoy Associated Universities, Inc. (AUI)

Laura Lockledge National Radio Astronomy Observatory (NRAO)

Darren McKay Englekirk Structural Engineering Center (ESEC) University of California (UC) San Diego

Pedro Lomonaco Oregon State University

Christopher McLaughlin National Radio Astronomy Observatory (NRAO)

Ramon Lugo Florida Space Institute

Bill McVeigh National Solar Observatory/AURA

Dave Lunger National High Magnetic Field Laboratory Florida State University

William Miller National Science Foundation (NSF)

Jeffery Lupis National Science Foundation (NSF)

Shree Mishra National Science Foundation (NSF)

Christine Lurtz Merrick & Co.

Anna-Lee Misiano National Science Foundation (NSF)

Louis Lussier National Center for Atmospheric Research (NCAR)

R Scott Moore Michael Baker International

Ruairi MacDonald Lux Consulting Group, Inc.

David Morrissey National Superconducting Cyclotron Laboratory Michigan State University

Jim Marsteller Pittsburgh Supercomputing Center (MSC)

Gilberto Mosqueda UC San Diego

Russell Moy Southeastern Universities Research Association

Kevin Porter National Science Foundation (NSF)

6B377BNSF 2018 Large Facilities Workshop

Page 29

NATIONAL SCIENCE FOUNDATION 2018 Large Facilities Workshop Alexandria, VA Monday, April 30 – Wednesday, May 2, 2018 Participant List

Mateo Munoz National Science Foundation (NSF)

Scott Powell University of Florida

Taina Munoz-Mulero National Science Foundation (NSF)

Melissa Prunchak National Science Foundation (NSF)

Brenda Murphy Associated Universities, Inc. (AUI)

Florence Rabanal National Science Foundation (NSF)

Keith Nackerud National Science Foundation (NSF)

Tim Reeme UNAVCO Inc

Deanne Nahaie-Noll Oregon State University (OSU)

Maryam Refan Natural Hazards Engineering Research Infrastructure Florida International University

Donna O'Malley National Science Foundation (NSF)

Robert Reid University Corporation for Atmospheric Research (UCAR)

Richard Oram Laser Interferometer Gravitational-wave Observatory (LIGO)

Clare Reimers Oregon State University (OSU)

Nipuni Palliyaguru Arecibo Observatory

Thomas Rimmele National Solar Observatory/AURA

Eric Palm National High Magnetic Field Laboratory

Roland Roberts National Science Foundation (NSF)

Manish Parashar National Science Foundation (NSF)

Crystal Roberts Scripps Institution of Oceanography

Joy Pauschke National Science Foundation (NSF)

Carolyn Robertson Natural Hazards Engineering Research Infrastructure Florida International University

Ron Payne National Center for Supercomputing Applications University of Illinois Urbana-Champaign

Ivan Rodero Rutgers University

Elizabeth Pentecost National Science Foundation (NSF)

Kaylyn Rogers National Center for Supercomputing Applications University of Illinois Urbana-Champaign

Joe Pesce National Science Foundation (NSF)

Evan Rolfe Avascent

Trisha Pinckney Incorporated Research Institutions for Seismology

Jay Roloff National Center for Supercomputing Applications

James Ruddell WSP USA

Jack Tade Associated Universities, Inc. (AUI)

6B377BNSF 2018 Large Facilities Workshop

Page 30

NATIONAL SCIENCE FOUNDATION 2018 Large Facilities Workshop Alexandria, VA Monday, April 30 – Wednesday, May 2, 2018 Participant List

Stephanie Saltzman National Science Foundation (NSF)

John Trowbridge Woods Hole Oceanographic Institution

Kim Sargent Woods Hole Oceanographic Institution

Michael Tuts Columbia University

Matthew Schoettler UC Berkeley

James Ulvestad National Science Foundation (NSF)

David Scott BDO Industry Specialty Services

Catherine Vakhnina University of Wisconsin – Madison

Jamie Slater Cornell University

Alex Walter National Radio Astronomy Observatory (NRAO)

Amy Smith University Corporation for Atmospheric Research

Kate Walters National Science Foundation (NSF)

Chris Smith Association of Universities for Research in Astronomy (AURA)

Mark Warner National Solar Observatory Association of Universities for Research in Astronomy

James Sowinski US Department of Energy

Joseph Wartman RAPID Facility/University of Washington

Kristin Spencer National Science Foundation (NSF)

Carolyn Watkins National Solar Observatory Association of Universities for Research in Astronomy

Tim Spuck Associated Universities, Inc. (AUI)

Andrew Weisenberger Thomas Jefferson National Accelerator Facility

Erica Stein National Science Foundation (NSF)

Eddie Whitehurst National Science Foundation (NSF)

Kenneth Stokoe UT Austin

Christi Whittredge National Science Foundation (NSF)

Andreas Stolz National Superconducting Cyclotron Laboratory Michigan State University

Jemal Williams National Science Foundation (NSF)

Virginia (Gina) Taberski University Corporation for Atmospheric Research

Dan Wilson UC Davis

Nazanin Tavassoli BCF Solutions

Andrea Wilson BDO

Verna Tomanek Battelle

Scott Wolf DOE/Oak Ridge National Laboratory

John Towns Extreme Science and Engineering Discovery Environment (XSEDE) National Center for Supercomputing Applications

Yan Xing Consortium for Ocean Leadership

6B377BNSF 2018 Large Facilities Workshop

Page 31

NATIONAL SCIENCE FOUNDATION 2018 Large Facilities Workshop Alexandria, VA Monday, April 30 – Wednesday, May 2, 2018 Participant List

Rebecca Yasky National Science Foundation (NSF)

Daniel Zehner Natural Hazards Engineering Research Infrastructure Purdue University

Lawrence Yatch Sealed Mindset Leaders

Kenneth Zeller Battelle Memorial Institute

Ashley Zauderer National Science Foundation (NSF)

Jeff Zivick National Science Foundation (NSF)

6B377BNSF 2018 Large Facilities Workshop

Page 32

APPENDIX C: PRESENTATION SLIDES The following table provides links to the slides used for each workshop presentation. The presentations are organized in chronological order, by date of the presentation.

Monday, April 30, 2018

Presentation Link to Slides

LFO Opening Remarks – Status, Evolving Oversight & Other Developments

https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/04/OpeningPresentation.pdf

Lessons for the Future No slides were used during the session.

Fast Radio Bursts https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/04/FastRadioBursts.pdf

Impacts of Hurricane Maria on the Arecibo Observatory

https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/ImpactsHurricaneMaria.pdf

Woodrow Wilson Bridge Replacement Project: A Mega-Project Success Story

https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/04/WoodrowWilsonBridge.pdf

Session I: Introduction – Forming, Learning, and Operating in High Functioning Teams: Lessons from the Navy SEALs

https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/NavySEALSSession1.pdf

Session II: Overview – Forming, Learning, and Operating in High Functioning Teams: Lessons from the Navy SEALs

https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/NavySEALSSession2.pdf

Session III: Hands On Exercise – Forming, Learning, and Operating in High Functioning Teams: Lessons from the Navy SEALs

https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/NavySEALSSession3.pdf

International Science and G7 Group of Senior Officials (GSO) Framework

https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/InternationalScience.pdf

Enhancing NSF Design and Construction Processes through Spiral Development

https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/EnhancingDesign.pdf

Core Competency Requirements for Recipients https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/CoreCompetencyRequirements.pdf

6B377BNSF 2018 Large Facilities Workshop

Page 33

Tuesday, May 1, 2018

Presentation Link to Slides

Federal Budget Outlook https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/04/FederalBudgetOutlook.pdf

NSF Office of Inspector General – Audit Results & Other Topics

https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/04/NSFOfficeInspectorGeneral.pdf

Knowledge Management Model for Improving Communication Among Facilities and with NSF

https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/04/KnowledgeManagement.pdf

NSF Business Issues I https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/CostEstimating.pdf

https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/NSFBuisnessIssues1a.pdf

Society for Science at User Research Facilities (SSURF, formerly NUFO)

https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/SocietyScience.pdf

NSF Proposal & Award Policy Update https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/NSFGrantsPolicyUpdate.pdf

User Engagement and Support https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/UserEngagement.pdf

Facility Performance Metrics https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/FacilitiesPerformanceMetrics.pdf

https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/04/FacilityPerformanceMetrics.pdf

NSF Business Issues II https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/NSFBusinessIssuesII.pdf

Energy Improvements and Alternative Funding Options

https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/EnergyImprovements.pdf

Cyberinfrastructure for Facilities https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/CyberinfrastructureFacilities1.pdf

https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/CyberinfrastructureFacilities2.pdf

6B377BNSF 2018 Large Facilities Workshop

Page 34

Wednesday, May 2, 2018

Presentation Link to Slides

Major Facility Innovations & Technology Transfer https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/MajorFacilityInnovation1.pdf

https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/MajorFacilityInnovation2.pdf

Lessons Learned: Application to NHERI Experimental Facilities

https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/NHERILessonsLearned1.pdf

https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/NHERILessonsLearned2.pdf

https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/NHERILessonsLearned3.pdf

https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/NHERILessonsLearned4.pdf

Facilities Condition Assessment https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/04/FacilitiesConditionAssessment.pdf

Idea Exchange Open Forum for Recipients No slides were used during the session.

Activity-Based Budgeting and Cost Estimating for Operations

https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/Activity-Based-Budgeting.pdf

Cost Estimating for NSF Facility Operations – Reasonable Application of GAO Good Practices

https://www.largefacilitiesworkshop.com/wp-content/uploads/2018/05/CostEstimatingFacility-Operations.pdf

Workshop Debrief – Open Forum & Actionable Recommendations from Idea Exchange

No slides were used during the session.