PHENIX Off-line Computing System PEP The PHENIX Event Processor The PHENIX Collaboration presented...
-
Upload
silvester-briggs -
Category
Documents
-
view
215 -
download
3
Transcript of PHENIX Off-line Computing System PEP The PHENIX Event Processor The PHENIX Collaboration presented...
PHENIX Off-line Computing System
PEPPEPThe PHENIX Event ProcessorThe PHENIX Event Processor
The PHENIX Collaborationpresented by
S. P. Sorensen
University of Tennessee
at CHEP97
The PEP Team:The PEP Team:Han Geng, Qihang Liu, Han Geng, Qihang Liu, David P. MorrisonDavid P. Morrison, , Li Qun, Li Qun, Victor PerevoztchikovVictor Perevoztchikov, Kyle Pope, , Kyle Pope,
Mark Pollark, Soren Sorensen and Mark Pollark, Soren Sorensen and Yan ZhaoYan Zhao
PHENIX Off-line Computing System
Outline
PHENIX’s Computing Model PHENIX Requirements Types of Analysis Frameworks Domains and Implementation Choices Interprocess Communication User Modules Data Structure Management RHIC Off-line Software Collaboration
PHENIX Off-line Computing System
PHENIX Computing Model
Level ? Trigger
Online Monitoring
Event Reconstruction
Data Mining
RHIC Data Analysis
Routing Layer
HDSS:Hierarchical Data Storage System [raw]
HDSS [particles]
HDSS: [Filtered
data]
HDSS: [Data Analysis
results]
Central Data Base
Local Data Analysis
Local Data Base
Local Data StorageSimulation
Local Data Base
HDSS
Data Mining
Local Data Analysis
Central Data Base
PHENIX Counting House
RHIC Computing Center
Remote Sites Remote Sites
Regional Computing Center
PHENIX Off-line Computing System
PHENIX’s CPU and Data Storage Requirements
CPU LocationGflops kSPECInt92
Event Reconstruction 70 175 RHICData Mining 10 30 RHICData Analysis (Data Scaning) 20 60 RHIC & JapanPhysics Models 25 75 JapanDetector Simulation 25 75 JapanTotal 150 415
Type of data TB/year Storage MediumDisk Robot
Raw Data 350 1 12Data Summary Level 175 3 8micro DST's 100 5 10Simulated Data 150 1 5Physics Model Data 10 1 5Off-line Database 0.1 0.1Total 785 11 40
CPU
Data Production and Storage
PHENIX Off-line Computing System
PHENIX Analysis Framework Requirement I
Environment Can operate in a distributed,
heterogeneous environment Can operate in a single node environment Can operate both on the highest
performance chips and the best price/performance chips
PHENIX Off-line Computing System
PHENIX Analysis Framework Requirement II
Configurability Can be configured to do all aspects of the
PHENIX Data analysis Will allow the creation of
• a small, ``tight'' executable for the single user
• a large comprehensive set of executables for the large-scale tasks like Event Reconstruction
Can operate in both batch and interactive mode with same functionality
Will allow already existing software (PISA/PISORP) to be incorporated
PHENIX Off-line Computing System
PHENIX Analysis Framework Requirement III
Data Access Will allow the user access to all of
PHENIX data through a uniform API Will allow the user to perform data queries Will allow full access to both a global,
central database and local data bases Will allow the user to graphical view the
data Will allow access specifically to all data in
the online/DAQ system injected into the routing layer
PHENIX Off-line Computing System
PHENIX Analysis Framework Requirement IV
User Interface GUI and batch interface shall provide
identical functionality GUI should be tailorable by the user GUI-batch mode switchable at run time GUI can run on a small, cheap computers
(PC, Mac, etc..) User Interface the same in all running
modes except for a few mode dependent menus. (a la Mac menus)
PHENIX Off-line Computing System
PHENIX Analysis Framework Requirement V
I/O Will provide I/O from/to disk, tape,
HSMS, memory and the PHENIX routing layer
Visualization Will provide interface to HEP visualization
tools: PAW, ROOT, .. Will allow all event data and as much
static data as possible to be viewed superimposed on the PHENIX geometry
PHENIX Off-line Computing System
Classical Type ofAnalysis Framework
MainInitializationBookkeepingetc.
EventLoop
Module
Module
Module
Co-located
ControlCore vs. User Core vs. User
Run time vs.Compile time
Run time vs.Compile time
ExamplesGEANT: UserPISA: CoreROOT: Core
PEP1: Core, CompilePEP2: User, CompileSTAF: User, RunROOT: User, Run
PEP: CompileSTAF: CompileROOT: Run
PHENIX Off-line Computing System
“Modern” Type of Analysis Framework
MainEventLoop
Module
Module
ModuleIndi
rect
ion
Indi
rect
ion
Indirection: Software Bus, Communication Layer
•Advantages of indirection:•Insulates Modules (Language …)•Allows distributed architecture•Insulates GUI from Modules
PHENIX Off-line Computing System
PEP’s LayeredArchitecture
Control Messages
Controller
Event Data Messages
DatabaseEvent Data Store
DataOpera-tor A
DataOpera-tor B
I/O Tool
Data-base Tool
(Graphical) User Interface
This is a typical example of an implementation ofthe “Toaster” Model.
UserLayer
StandardInterface
StandardInterface
PHENIX Off-line Computing System
SDC Toaster Model
PHENIX Off-line Computing System
PEP Domains(Shlaer-Mellor OOA)
Controller Data Operators
GUIDatabase
Tool
Interprocess Communication
Data Monitoring
Data Visualization
Tool
In/Out Tool
Error Handling
Architecture
GNU C++ Unix Cernlibs Fortran
Data Structure
Management
PVM OnX DSPACK
C++ STL
CORBA TCL/TK
APPLICATION DOMAINS
SERVICE DOMAINS
ARCHITECTURAL DOMAIN
IMPLEMENTATION DOMAINS
PHENIX Off-line Computing System
PEP Implementation Choices I
GUI: TCL/TK • Allows both GUI and Macro interface
• Available on many platforms Interprocess Communication: PVM
• Parallel Virtual Machine
• Between TCP/IP and CORBA in complexity, flexibility and overhead
Data Structure Management: DSPACK • Created by Ryszard Zybert et al. for NA49
Database: ORACLE • Robust and fast for simple data structures
• Does the job for off-line
• But an OODB might be better in the long run
PHENIX Off-line Computing System
PEP Functional Units II
Input & Output Tool • Provides all I/O of data structures (as
opposed to control data which flows through the GUI)
Data Base Tool • Provides local access to either the central
data base or a local, cache data base Central Data Base Server
• Provides access to the central data base at RHIC. Runs on the RHIC cluster.
Visualizer • Provides all graphics services like histogram
display, event display and graphical data interaction
• Provides monitoring of all PHENIX specific aspects of the data flow
PHENIX Off-line Computing System
PEP Control and Data Flow
User Interface
Process Controller
Main Event Data Flow
Data Operators Data Tools
DatabaseManager
Input
Output
DataMonitor
Calibration
LocalEv. Rec. I
LocalEv. Rec. II
GlobalEv. Rec.
Con
trol
Dat
a F
low
Con
trol
Dat
a F
low
SecondaryData Flow
PHENIX Off-line Computing System
PEP Functional Units I
Controller • The ``brain'' of PEP
• Starts and stops all other processes
• Controls all communication between processes
• Monitors the status of all processes
• Allocates resources Graphical User Interface (GUI)
• Provides for all interaction between PEP and the user
Data Operators • Accept data structures as input, perform
operations on them and provide data structures as output
• Contains 95\% of all user code
PHENIX Off-line Computing System
PEP Implementation Choices II
Visualization • Spectra: DSPACK & PAW --> DSS
• Event display: OnX. OS: UNIX & NT
• UNIX: SGI, HP, DEC, IBM, Sun
• NT: On Pentium Pro Languages: C, C++
• Most of core PEP implemented in C
• A bit of FORTRAN for interface to Event Generators and for GEANT
PHENIX Off-line Computing System
PEP V2 (PEPSI)Single Co-located Configuration
PHENIX Off-line Computing System
PEP V2Simple Configuration
PHENIX Off-line Computing System
PEP V2Production System Configuration
PHENIX Off-line Computing System
PEP Standard Message Passing “Handshake” Procedure
Controller
Input DataOperator
12
3 4
5
6
7 8
Message # From To Message Action by Receiver1 Dop Cont Idle Controller locates idle Input2 Contr Input Process Input read and prepares DS3 Input Contr Done & p Controller prepares message4 Contr Dop Process & p DOP state change & message prep5 Dop Input Deliver & p Insert DS in PVM buffer and submit6 Input Dop DS Buffer Unpack DS7 Input Contr Idle Update proxy8 Dop Contr Processing Update proxy
PHENIX Off-line Computing System
PEP Controller Internal Objects
Controller
Data Operator A
PVM
PVM
Proxies
Dop A
Dop B
Data Operator B
PVM
CheckIn
PostOffice
Broker
Mailman
ConfigurationFile
Pep Stub
User Code User Code
Pep Stub
PVM
PHENIX Off-line Computing System
PEP Controller Main Loop
do while ( ActiveOperators )
CheckIn !Checks for mail and !sends it to the postoffice
do while ( new mail in PostOffice )
MailMan gets mail with generic address (Dop) MailMan asks Broker for best specific address (Dop.A.1)
If ( Specific address exists ) MailMan delivers mail to Proxy Proxy Object sends mail
to real object Original mail in Postoffice
is destroyed else Original mail is kept end if
end do
Audit
end do
PHENIX Off-line Computing System
PEP Data OperatorSimple State Model
IDLE
RECEIVING EVENT
PROCESSING EVENT [event]
DELIVERING EVENT
INITIALIZING [initg]
FINISHING [finis]
PHENIX Off-line Computing System
PEP Data OperatorRealistic State Model
IDLE
RECEIVING EVENT
PROCESSING EVENT [event]
DELIVERING EVENT
INITIALIZING [initg]
FINISHING [finiG]
SPAWNING
PRE-NATAL
FINISH RUN [finiR]
DEAD
FINISH RUN [finiR]
INITIALIZE RUN [InitR]
PHENIX Off-line Computing System
Pep User Module “Hooks”
Initialize DOP
JobInit
Loop over Runs
RunInit
Loop over Events Event
RunFinis
JobFinis
End of DOP
Error
PEP Data Operator Core User Module Program Units
PHENIX Off-line Computing System
PEP WAN Access to the Central Database
Controller
Data Ope- rator
Database Tool
Local Data- base
Remote Database Access
Local Database
Remote Database Access Database
Tool ORACLE Interface [proC]
Central ORACLE Database
Off-site Location
RHIC Computing Facility
ORACLE table
DSPACK Object
DSPACK Object
Controller
Data Ope- rator Database
Tool
ORACLE Interface [proC] ORACLE
table
PHENIX Event Reconstruction Facility
DSPACK Object
PHENIX Off-line Computing System
PEP Physics Module Types
EVENT GENERATOR [Gen]
HIT GENERATOR [Hit]
SIGNAL GENERATOR [Sig]
FEE SIMULATOR [Fee]
LEVEL 1 TRIGGER [Tl1]
LEVEL 2 TRIGGER [Tl2]
CALIBRATOR [Cal]
LOCAL RECONST. [ReA]
GLOBAL RECONST. [ReC]
EVENT BUILDER [Evb]
DsGen
DsHit
DsSig
DsGenI
DsHitI
DsSigI
DsFeeI
DsFee
DsTl1
DsTl1I
DsTl2I
DsTl2
DsEvb
DsCal
DsReB
DsReC
DsPar
DsEvbI
DsCalI
DsReAI
DsReCI
DsParI
PAIR RECONSTRUCTION [Par]Input Data Structure
Module Type
Event Data Structure
ANALYSIS [Ana]
N Times
DsAnaI
SUBSYSTEM RECONST. [ReB]
DsReBIDsReA
N Times
N Times
PHENIX Off-line Computing System
PEP Module Types I(Simulation)
Event Generator [Gen]• Generates simulated events in the form of a
list of particles created at the vertex Hit Generator [Hit]
• Accepts the Gen output and generates hits in all sensitive volumes. (GEANT Application)
Signal Generator [Sig] • Accepts the Hit output, sums hits over all
readout volumes and calculates the final physics output signal in each read-out channel
FEE Simulator [Fee] • Simulates the response of the front-end
electronics to provide the digital signals in each channel. Typical effects: digitization, discriminator walk, pedestals etc.
PHENIX Off-line Computing System
PEP Module Types II(Trigger)
Level 1 Trigger Simulator [Tl1] • Simulates all aspects of the data handling in
the level 1 trigger Level 2 Trigger Simulator [Tl2]
• Simulates all aspects of the data handling in the level 2 trigger
Level 3 Trigger Simulator [Tl3] • Simulates all aspects of the data handling in
the level 3 trigger
PHENIX Off-line Computing System
PEP Module Types III(Initial Off-line)
Calibrator [Cal] • Accepts the raw data as input an either
calculates calibrations constants or performs all calibrations and corrections of the raw data.
Local Event Reconstruction [ReA] • Performs all event reconstruction at the
detector level which an be done in a stand-alone mode. Examples: Cluster finding in EMC, Hit finding in the Drift Chambers
Sub-system Event Reconstruction [ReB] • Performs all event reconstruction at the sub-
system level, which is done based on the input of information from another detector in the same sub-system. Example: Finding muon tracks in the muon tracking systems based on a list of identified muons from the muon identifier.
PHENIX Off-line Computing System
PEP Module Types IV(Late Off-line)
Global Event Reconstruction [ReC] • Performs all event reconstruction at the
global PHENIX level, where information from other sub-systems is necessary. Example: Particle identification
Data Mining [DaM] • Perform all function related to data mining:
preparation of data in storage system and search of data.
Analyzer [Ana] • Performs subsequent data analysis.
PHENIX Off-line Computing System
PEP Current Status
PEP has been frozen as of Mar 1997 RHIC Software Collaboration
PHENIX has decided to collaborate with STAR and the other RHIC experiments on Core Off-line Software
Straw man model• Analysis Framework: STAF
• Display: ROOT
• Database: ORACLE
• Event Data Storage: Outcome of Computational Grand Challenge Project on Data Organization