Software Overview and LCG Project Status & Plans Torre Wenaus BNL/CERN DOE/NSF Review of US LHC...

Software Overviewand

LCG Project Status & Plans

Torre WenausTorre Wenaus

BNL/CERNBNL/CERN

DOE/NSF Review of US LHC Software and ComputingDOE/NSF Review of US LHC Software and Computing

NSF, ArlingtonNSF, ArlingtonJune 20, 2002 June 20, 2002

June 20, 2002June 20, 2002Torre Wenaus, BNL/CERN DOE/NSF ReviewTorre Wenaus, BNL/CERN DOE/NSF Review 2

Outline

Overview, organization and planningOverview, organization and planning

Comments on activity areasComments on activity areas

PersonnelPersonnel

ConclusionsConclusions

LCG Project Status and PlansLCG Project Status and Plans


U.S. ATLAS Software Project Overview

Control framework and architectureControl framework and architecture Chief Architect, principal development role. Software agreement in place

Databases and data management Database Leader, primary ATLAS expertise on ROOT/relational baseline

Software support for development and analysisSoftware support for development and analysis Software librarian, quality control, software development tools, training… Automated build/testing system adopted by and (partly) transferred to

Int’l ATLAS

Subsystem software roles complementing hardware responsibilitiesSubsystem software roles complementing hardware responsibilities Muon system software coordinator

Scope commensurate with U.S. in ATLAS: ~20% of overall effortScope commensurate with U.S. in ATLAS: ~20% of overall effort

Commensurate representation on steering group

Strong role and participation in LCG common effortStrong role and participation in LCG common effort


U.S. ATLAS Software Organization

William WillisProject Manager

John HuthAssociate Project Manager,

Computing and Physics WBS 2

James ShankDeputy

External Advisory Group

Ian HinchliffeManager, Physics

WBS 2.1

Torre WenausManager, Software

WBS 2.2

Bruce GibbardManager, Facilities

WBS 2.3

C. TullControl/Framework

2.2.1.1,2.2.1.2

David MalonData Management

2.2.1.3

S. RajagopalanEvent Model

2.2.1.4

J. ShankDetector Specific

2.2.2, 2.2.2.1

F. MerrittTraining

2.2.5L. Vacavant

Pixel/SCT2.2.2.2

F. LuehringTRT

2.2.2.3

S. RajagopalanLiquid Argon Calorimeter

2.2.2.4

T. LeCompteTilecal2.2.2.5

S. GoldfarbMuons2.2.2.6

S. GonzalezTrigger/DAQ

2.2.2.7

Subsystems

Core Software

R. BakerTier 1 Facility

R. GardnerDistributed ITInfrastructure

Facilities

TBNCollaborative

Tools2.2.3

R. BakerDeputy

T. WenausSoftware Support

Coordinator2.2.4

A. UndrusSoftware Librarian

2.2.4.1

Computing CoordinationBoard

Physics Manager, IB Convener, co-chairs


U.S. ATLAS - ATLAS Coordination

William WillisU.S. ATLAS Project

Manager

Peter JenniATLAS

Spokesperson

John HuthAssociate PM

NormanMcCubbinSoftware

Coordiinator

I. HinchliffePhysics SM

F. GianottiPhysics

Coordinator

T. WenausSoftware SM

B. GibbardFacilities SM

C. TullFramework

D. MalonDatabase

SubsystemSoftware

T. WenausPlanning Officer

D. QuarrieChief Archectect

D.MalonDatabase

A. PutzerNCB

L. PeriniATLAS GRID

R. GardnerDistributed Computing

SubsystemSoftware

US roles in Int’l ATLAS software:US roles in Int’l ATLAS software:

D. Quarrie (LBNL), Chief ArchitectD. Quarrie (LBNL), Chief Architect

D. Malon (ANL), Database CoordinatorD. Malon (ANL), Database Coordinator

P. Nevski (BNL), Geant3 Simulation P. Nevski (BNL), Geant3 Simulation CoordinatorCoordinator

H. Ma (BNL), Raw Data CoordinatorH. Ma (BNL), Raw Data Coordinator

C. Tull (LBNL), Eurogrid WP8 LiaisonC. Tull (LBNL), Eurogrid WP8 Liaison

T. Wenaus (BNL), Planning OfficerT. Wenaus (BNL), Planning Officer

US International


ATLAS Subsystem/Task Matrix

Offline Offline

CoordinatorCoordinator

ReconstructionReconstruction SimulationSimulation DatabaseDatabase

ChairChair N. McCubbinN. McCubbin D. RousseauD. Rousseau A. Dell’AcquaA. Dell’Acqua D. MalonD. Malon

Inner DetectorInner Detector D. BarberisD. Barberis D. RousseauD. Rousseau F. LuehringF. Luehring S. Bentvelsen /S. Bentvelsen /

D. CalvetD. Calvet

Liquid ArgonLiquid Argon J. CollotJ. Collot S. RajagopalanS. Rajagopalan M. LeltchoukM. Leltchouk H. MaH. Ma

Tile CalorimeterTile Calorimeter A. SolodkovA. Solodkov F. MerrittF. Merritt V.TsulayaV.Tsulaya T. LeCompteT. LeCompte

MuonMuon J.ShankJ.Shank J.F. LaporteJ.F. Laporte A. RimoldiA. Rimoldi S. GoldfarbS. Goldfarb

LVL 2 Trigger/ LVL 2 Trigger/

Trigger DAQTrigger DAQ

S. GeorgeS. George S. TapproggeS. Tapprogge M. WeilersM. Weilers A. Amorim /A. Amorim /

F. TouchardF. Touchard

Event FilterEvent Filter V. VercesiV. Vercesi F. TouchardF. Touchard

Computing Steering Group members/attendees: 4 of 19 from USComputing Steering Group members/attendees: 4 of 19 from US

(Malon, Quarrie, Shank, Wenaus)(Malon, Quarrie, Shank, Wenaus)


Project Planning Status

U.S./Int’l ATLAS WBS/PBS and schedule fully unifiedU.S./Int’l ATLAS WBS/PBS and schedule fully unified ‘Projected’ out of common sources (XProject); mostly the same

US/Int’l software planning covered by the same person US/Int’l software planning covered by the same person Synergies outweigh the added burden of the ATLAS Planning Officer role

No ‘coordination layer’ between US and Int’l ATLAS planning: direct ‘official’ interaction with Int’l ATLAS computing managers. Much more efficent

No more ‘out of the loop’ problems on planning (CSG attendance)

True because of how the ATLAS Planning Officer role is currently scoped As pointed out by an informal ATLAS computing review in March, ATLAS would

benefit from a full FTE devoted to the Planning Officer function I have a standing offer to the Computing Coordinator: to willingly step aside

if/when a capable person with more time is found Until then, I scope the job to what I have time for and what is highest priority ATLAS management sought to impose a different planning regime on computing

(PPT) which would have destroyed US/Int’l planning commonality; we reached an accommodation which will make my time more rather than less effective, so I remained in the job


ATLAS Computing Planning

US led a comprehensive review and update of ATLAS computing schedule in US led a comprehensive review and update of ATLAS computing schedule in Jan-MarJan-Mar Milestone count increased by 50% to 600; many others updated Milestones and planning coordinated around DC schedule Reasonably comprehensive and detailed through 2002

Things are better, but still not greatThings are better, but still not great Schedule still lacks detail beyond end 2002 Data Challenge schedules and objectives unstable Weak decision making (still a major problem) translates to weak planning

Strong recommendation of the March review to fix this; no observable change

Use of the new reporting tool PPT (standard in ATLAS construction project) Use of the new reporting tool PPT (standard in ATLAS construction project) may help improve overall planningmay help improve overall planning Systematic, regular reporting coerced by automated nagging Being introduced so as to integrate with and complement XProject-based

planning materials. XProject adapted; waiting on PPT adaptations.


Short ATLAS planning horizon

Schedule items

0

50

100

150

200

250

1999

2000

2001

2002

2003

2004

2005

As of 3/02


Summary Software Milestones

0 1 2 3 4 5 6 1 Tbyte database prototype (Done)Release of Athena pre-alpha version (Done)Athena alpha release (Done)Geant3 digi data available (Done)Athena beta release (Done)Athena accepted (ARC concluded) (Done)Athena Lund release (Done)Event store architecture design document (Done)DC0 production releaseDecide on database product (Done)DC0 Completed - continuity testFull validation of Geant4 physics DelayDC1 CompletedComputing TDR Finished DelayDC2 Completed (followed by annual DCs)Physics readiness report completed Full software chain in real environmentFull DB infrastructure available

Green: Done Gray: Original date Blue: Current date


Data Challenge 1

DC1 phase 1 (simu production for HLT TDR) is ready to startDC1 phase 1 (simu production for HLT TDR) is ready to start

Software ready and tested, much developed in the USSoftware ready and tested, much developed in the US Baseline core software, VDC, Magda, production scripts

2M events generated and available for filtering and simulation2M events generated and available for filtering and simulation US is providing the first 50K of filtered, fully simulated events for QA

Results will be reviewed by QA group before the green light is given for full Results will be reviewed by QA group before the green light is given for full scale production in about a weekscale production in about a week

During the summer we expect to process 500k fully simulated events at the During the summer we expect to process 500k fully simulated events at the BNL Tier 1BNL Tier 1


Brief Comments on Activity Areas

Control Framework and ArchitectureControl Framework and Architecture

DatabaseDatabase

Software Support and Quality ControlSoftware Support and Quality Control

Grid SoftwareGrid Software


Control Framework and Architecture

US leadership and principal development rolesUS leadership and principal development roles

David Quarrie recently offered and accepted a further 2 year term as Chief ArchitectDavid Quarrie recently offered and accepted a further 2 year term as Chief Architect

Athena role in ATLAS appears well consolidatedAthena role in ATLAS appears well consolidated Basis of post-simulation Data Challenge processing Actively used by end users, with feedback commensurate with Athena’s stage of development LHCb collaboration working well FADS/Goofy (simulation framework) issue resolved satisfactorily Regular, well attended tutorials

Other areas still have to prove themselvesOther areas still have to prove themselves ATLAS data definition language

Being deployed now in a form capable of describing the ATLAS event model

Interactive scripting in Athena Strongly impacted by funding cutbacks New scripting service emerging now

Tremendous expansion in ATLAS attention to event modelTremendous expansion in ATLAS attention to event model About time! A very positive development Broad, (US) coordinated effort across the subsystems to develop a coherent ATLAS event model Built around the US-developed StoreGate infrastructure

Core infrastructure effort receiving useful feedback from the expanded activity


Database

US leadership and key technical expertiseUS leadership and key technical expertise

The ROOT/relational hybrid store in which the US has unique expertise in ATLAS is

now the baseline, and is in active development

The early US effort in ROOT and relational approaches (in the face of ‘dilution of

effort’ criticisms) was a good investment for the long term as well as the short term

Event data storage and management now fully aligned with the LCG effortEvent data storage and management now fully aligned with the LCG effort

~1 FTE each at ANL and BNL identified to participate and now becoming active

Work packages and US roles now being laid out

ATLAS and US ATLAS have to be proactive and assertive in the common project for the ATLAS and US ATLAS have to be proactive and assertive in the common project for the

interests of interests of ATLASATLAS (I don’t have my LCG hat on here!), and I am pushing this hard (I don’t have my LCG hat on here!), and I am pushing this hard

Delivering a data management infrastructure that meets the needs of (US) ATLAS

and effectively uses our expertise demand it


Software Support, Quality Control

New releases are available in the US ~1 day after CERN (with some exceptions New releases are available in the US ~1 day after CERN (with some exceptions when problems arise!)when problems arise!) Provided in AFS for use throughout the US

Librarian receives help requests and queries from ~25 people in the USLibrarian receives help requests and queries from ~25 people in the US US-developed nightly build facility used throughout ATLASUS-developed nightly build facility used throughout ATLAS

Central tool in the day to day work of developers and the release process Recently expanded as framework for progressively integrating more quality

control and testing Testing at component, package and application level Code checking to be integrated

CERN support functions being transferred to new ATLAS librarian Plan to resume BNL-based nightlies

Much more stable build environment than CERN at the moment Hope to use timely, robust nightlies to attract more usage to the Tier 1

pacmanpacman (Boston U) for remote software installation (Boston U) for remote software installation Adopted by grid projects for VDT, and a central tool in US grid testbed work


Grid Software

Software development within the ATLAS complements of the grid Software development within the ATLAS complements of the grid projects is being managed as an integral part of the software effortprojects is being managed as an integral part of the software effort Objective is to integrate grid software activities tightly into ongoing

core software program, for maximal relevance and return Grid project programs consistent with this have been developed

And has been successfulAnd has been successful e.g. Distributed data manager tool (Magda) we developed was

adopted ATLAS-wide for data management in the DCs

Grid goals, schedules integrated with ATLAS (particularly DC) programGrid goals, schedules integrated with ATLAS (particularly DC) program

However However we do suffer some program distortionwe do suffer some program distortion e.g. we have to limit effort on providing ATLAS with event storage

capability in order to do work on longer-range, higher-level distributed data management services


Effort Level Changes

ANL/Chicago – loss of .5 FTE in DBANL/Chicago – loss of .5 FTE in DB Ed Frank departure; no resources to replace

BNL – cancelled 1 FTE new hire in data managementBNL – cancelled 1 FTE new hire in data management Insufficient funding in the project and the base program to sustain the bare-

bones plan Results in transfer of DB effort to grid (PPDG) effort – because the latter

pays the bills, even if it distorts our program towards lesser priorities As funding looks now, >50% of the FY03 BNL sw development effort will be

on grid!! LBNL – stable FTE count in architecture/frameworkLBNL – stable FTE count in architecture/framework

One expensive/experienced person replaced by very good postdoc It is the DB effort that is most hard-hit, but ameliorated by common projectIt is the DB effort that is most hard-hit, but ameliorated by common project

Because the work is now in the context of a broad common project, US can still sustain our major role in ATLAS DB

This is a real, material example of common effort translating into savings (even if we wouldn’t have chosen to structure the savings this way!)


Personnel Priorities for FY02, FY03

Priorities are the same as presented last time, and this is how we are Priorities are the same as presented last time, and this is how we are doing…doing… Sustain LBNL (4.5FTE) and ANL (3FTE) support

This we are doing so far.

Add FY02, FY03 1FTE increments at BNL to reach 3FTEs Failed. FY02 hire cancelled.

Restore the .5FTE lost at UC to ANL No resources

Establish sustained presence at CERN. No resources

As stated last time… we rely on labs to continue base program and As stated last time… we rely on labs to continue base program and other lab support to sustain existing complement of developersother lab support to sustain existing complement of developers And they are either failing or predicting failure soon. Lab base

programs are being hammered as well.


SW Funding Profile Comparisons

0.00

1.00

2.00

3.00

4.00

5.00

6.00

FY01 FY02 FY03 FY04 FY05 FY06

M$

2000 agency guideline

January 2000 PMP

11/2001 guideline

Compromise profile

requested in 2000

Current bare bones


Conclusions

US has consolidated the leading roles in our targeted core software areasUS has consolidated the leading roles in our targeted core software areas

Architecture/framework effort level being sustained so farArchitecture/framework effort level being sustained so far

And is delivering the baseline core software of ATLAS

Database effort reduced but so far preserving our key technical expertiseDatabase effort reduced but so far preserving our key technical expertise

Leveraging that expertise for a strong role in common project

Any further reduction will cut into our expertise base and seriously weaken the US

ATLAS role and influence in LHC database work

US has made major contributions to an effective software development and release US has made major contributions to an effective software development and release

infrastructure in ATLASinfrastructure in ATLAS

Plan to give renewed emphasis to leveraging and expanding this work to make the

US development and production environment as effective as possible

Weakening support from the project and base programs while the emphasis on grids Weakening support from the project and base programs while the emphasis on grids

grows is beginning to distort our program in a troubling waygrows is beginning to distort our program in a troubling way

LCG Project Status & Plans(with an emphasis on applications software)

Torre WenausTorre Wenaus

BNL/CERNBNL/CERN

DOE/NSF Review of US LHC Software and ComputingDOE/NSF Review of US LHC Software and Computing

NSF, ArlingtonNSF, ArlingtonJune 20, 2002 June 20, 2002


The LHC Computing Grid (LCG) Project

Developed in light of the LHC Computing Review conclusionsDeveloped in light of the LHC Computing Review conclusions Approved (3 years) by CERN Council, September 2001 Injecting substantial new facilities and personnel resources

Activity areas:Activity areas: Common software for physics applications

Tools, frameworks, analysis environment

Computing for the LHC Computing facilities (fabrics) Grid middleware Grid deployment

Global analysis environment

Foster collaboration, coherence of LHC computing centers Foster collaboration, coherence of LHC computing centers

Goal – Prepare and deploy the LHC computing environment


OtherLabs

The LHC Computing Grid Project

LHCC

Project Overview Board

TechnicalStudy

Groups

Reports

Reviews

ResourcesBoard

Resource Issues

ComputingGrid

Projects

Project execution teams

HEPGrid

Projects

Project Manager

ProjectExecution

Board

Requirements,Monitoring

Software andComputingCommittee

LCG Project Structure


Current Status

High level workplan just written (linked from this review’s web page)High level workplan just written (linked from this review’s web page) Two main threads to the work:Two main threads to the work:

Testbed development (Fabrics, Grid Technology and Grid Deployment areas)

A combination of primarily in-house CERN facilities work and working with external centers and the grid projects

Developing a first distributed testbed for data challenges by mid 2003 Applications software (Applications area)

The most active and advanced part of the project Currently three active projects in applications:

Software process and infrastructure Mathematical libraries Persistency

Pressuring the SC2 to open additional project areas ASAP – not enough current scope to put available people to work effectively (new LCG and existing IT people)


LHC Manpower needs for Core Software

20002000

Have (Have (missmiss))

20012001 20022002 20032003 20042004 20052005

ALICEALICE 12(12(55)) 17.517.5 16.516.5 1717 17.517.5 16.516.5

ATLASATLAS 23(23(88)) 3636 3535 3030 2828 2929

CMSCMS 15(15(1010)) 2727 3131 3333 3333 3333

LHCbLHCb 14(14(55)) 2525 2424 2323 2222 2121

TotalTotal 64(64(2828)) 105.5105.5 106.5106.5 103103 100.5100.5 99.599.5

Only computing professionals counted

From LHC Computing Review (FTEs)

LCG common project activity in applications software: Expected number of new LCG-funded people in applications is 23 Number hired or identified to date: 9 experienced, 3 very junior Number working today: 8 LCG (3 in the last 2 weeks), plus ~3 existing IT, plus expts


Applications Area Scope

Application Software InfrastructureApplication Software Infrastructure Scientific libraries, foundation libraries, software development

tools and infrastructure, distribution infrastructure Physics Data ManagementPhysics Data Management

Storing and managing physics data: events, calibrations, analysis objects

Common FrameworksCommon Frameworks Common frameworks and toolkits in simulation, reconstruction

and analysis (e.g. ROOT, Geant4) Support for Physics ApplicationsSupport for Physics Applications

Grid ‘portals’ and interfaces to provide distributed functionality to physics applications

Integration of physics applications with common software


Typical LHC Experiment Software Architecture… a ‘Grid-Enabled’ View

Standard LibrariesH

igh level tr

iggers

One main framework, e.g. ROOT

Various specialized frameworks: persistency (I/O), visualization, interactivity, simulation, etc.

Grid integration

Widely used utility libraries (STL, CLHEP); distributed services

Applications built on top of frameworks

Analy

sis

Sim

ula

tion

Reco

nst

ruct

ion

FrameworksToolkits

FrameworksToolkits

= Common solutions being pursued or foreseen

GridServices

GridInterfaces


Current major project: Persistency

First major common software project begun in April: First major common software project begun in April: Persistency Framework (POOL)Persistency Framework (POOL) To manage locating, reading and writing physics data

Moving data around will be handled by the grid, as will the distributed cataloging Will support either event data or non-event (e.g. conditions) data

Selected approach: a Selected approach: a hybrid storehybrid store Data objects stored by writing them to ROOT files

The bulk of the data Metadata describing the files and enabling lookup are stored in relational

databases Small in volume, but with stringent access time and search requirements, well suited to

relational databases Successful approach in current experiments, e.g. STAR (RHIC) and CDF (Tevatron)Successful approach in current experiments, e.g. STAR (RHIC) and CDF (Tevatron)

LHC implementation needs to scale to much greater data volumes, provide distributed functionality, and serve the physics data object models of four different experiments

Early prototype is scheduled for September 02 (likely to be a bit late!)Early prototype is scheduled for September 02 (likely to be a bit late!) Prototype to serve a scale of 50TB, O(100k) files, O(10) sites Early milestone driven by CMS, but would have been invented anyway: we need

to move development work from abstract discussions to iterating on written software

Commitments from all four experiments to development participationCommitments from all four experiments to development participation ~3 FTEs each from ATLAS and CMS; in ATLAS, all the participation (~2 FTEs) so

far is from the US (ANL and BNL); another ~1-2 FTE from LHCb+ALICE


Hybrid Data Store – Schematic View

EventEvent

TrackListTrackList

TrackerTracker CalorimeterCalorimeter

TrackTrackTrackTrack

TrackTrackTrackTrackTrackTrack

HitListHitList

HitHitHitHitHitHitHitHitHitHit

Experiment event model

ObjectDictionary

Service

PersistencyManager

Experimentspecific object model

descriptions

Storage:

Pass object(s)

Get ID(s)

Retrieval:

Pass ID(s)

Get object(s)

Locator ServiceDistributed

Replica Manager(Grid)Locate Files

File(s)

Storage Manager

ID – FileDB

fopen etc.

ObjectStreaming

Service

File info

File records

Data objects

Objectinfo

Object descriptions

DatasetLocator

Process dataset DatasetDB

File(s)

Name

Data File

Human Interaction


Coming Applications RTAGs

After a very long dry spell (since Jan), the SC2 has initiated the first After a very long dry spell (since Jan), the SC2 has initiated the first stage of setting up additional projects – establishing requirements stage of setting up additional projects – establishing requirements and technical advisory groups (RTAGs) with 2-3 month durationsand technical advisory groups (RTAGs) with 2-3 month durations

Detector geometry and materials description To address high degree of redundant work in this area (in the case

of ATLAS, even within the same experiment)

Applications architectural blueprint High level architecture for LCG software

Pending RTAGs in applicationsPending RTAGs in applications Physics generators (launched yesterday) A fourth attempt at a simulation RTAG in the works (politics!) Analysis tools (will follow the blueprint RTAG)


Major LCG Milestones

June 2003 – LCG global grid service available (24x7 at 10 centers)June 2003 – LCG global grid service available (24x7 at 10 centers)

June 2003 – Hybrid event store releaseJune 2003 – Hybrid event store release

Nov 2003 – Fully operational LCG-1 service and distributed production Nov 2003 – Fully operational LCG-1 service and distributed production

environment (capacity, throughput, availability sustained for 30 days)environment (capacity, throughput, availability sustained for 30 days)

May 2004 – Distributed end-user interactive analysis from Tier 3May 2004 – Distributed end-user interactive analysis from Tier 3

Dec 2004 – Fully operational LCG-3 service (all essential functionality Dec 2004 – Fully operational LCG-3 service (all essential functionality

required for the initial LHC production service)required for the initial LHC production service)

Mar 2005 – Full function release of persistency frameworkMar 2005 – Full function release of persistency framework

Jun 2005 – Completion of computing service TDRJun 2005 – Completion of computing service TDR


LCG Assessment

In the computing fabrics (facilities) area, LCG is now the context (and In the computing fabrics (facilities) area, LCG is now the context (and funding/effort source) for CERN Tier0/1 developmentfunding/effort source) for CERN Tier0/1 development But countries have been slow to follow commitments with currency

In the grid middleware area, the project is still trying to sort out its role In the grid middleware area, the project is still trying to sort out its role as ‘not just another grid project’; not yet clear how it will achieve the as ‘not just another grid project’; not yet clear how it will achieve the principal mission of principal mission of ensuringensuring the needed middleware is available the needed middleware is available

In the deployment area (integrating the above two), testbed/DC plans In the deployment area (integrating the above two), testbed/DC plans are taking shape well with an aggressive mid 03 production deploymentare taking shape well with an aggressive mid 03 production deployment

In the applications area, the persistency project seems on track, but In the applications area, the persistency project seems on track, but politics etc. have delayed the initiation of new projectspolitics etc. have delayed the initiation of new projects The experiments do seem solidly committed to common projects

This will change rapidly if LCG hasn’t delivered in ~1 year CMS is most proactive in integrating the LCG in their plans; ATLAS

less so to date (this extends to the US program). I will continue to push (with my ‘ATLAS’ hat on!) to change this

Software Overview and LCG Project Status & Plans Torre Wenaus BNL/CERN DOE/NSF Review of US LHC...

Documents

Transcript of Software Overview and LCG Project Status & Plans Torre Wenaus BNL/CERN DOE/NSF Review of US LHC...