TBD - halldweb.jlab.orgFeb 22, 2013 · 75.40 ms (7539.8%) DAnalysisResults:PreKinFit 25.37 ms...

TBD Tantummodo Brevis Diffamatus

(Just to briefly make known)

David Lawrence -‐ JLab Feb. 22, 2013

2/22/13 TBD -‐ D. Lawrence -‐ JLab 1

Resources •  A “resource” here is a file needed by reconstrucKon that

doesn’t fit well into the CCDB model (generally because the file is too large) –  e.g. MagneKc Field Map

•  Resources should be available on the WWW

•  CCDB is used to manage locaKon (URL) of resource

•  Resource manager should minimize the number of copies on the local disk (hopefully only 1!)

2/22/13 2 local disk

CCDB h&p://site.org/file.dat

jana_program

File is installed into resource directory and corresponding

URL logged

Subsequent requests will use the local file without downloading

Full API is s7ll under development!

TBD -‐ D. Lawrence -‐ JLab

Call-‐graph of GlueX standard reconstrucKon

DAnalysisResults75.40 ms (7539.8%)

DAnalysisResults:PreKinFit25.37 ms (2537.2%)

10 calls12.08 s

1208457.3%

DParticleCombo199.00 us ( 19.9%)

10 calls448.00 us

44.8%

DAnalysisUtilities21.01 ms (2101.2%)

4 calls20.98 ms2101.8%

DBeamPhoton118.00 us ( 11.8%)

20 calls0.00 us 8.6%

DChargedTrack1.39 ms (138.9%)

20 calls0.00 us 5.3%

DMCThrown152.00 us ( 15.2%)

11 calls0.00 us 5.2%

DMCThrownMatching559.00 us ( 55.9%)

30 calls486.00 us

69.8%

DNeutralParticle145.00 us ( 14.5%)

20 calls114.00 us

20.0%DParticleCombo:PreKinFit

9.36 s (936423.3%)

10 calls12.04 s

1203717.9%

DParticleID63.00 us ( 6.3%)

1 calls0.00 us 1.0%

DChargedTrackHypothesis7.78 ms (777.5%)

10 calls2.62 s

262265.1%

10 calls0.00 us 2.3%

10 calls0.00 us 10.0%

10 calls0.00 us 0.1%

10 calls0.00 us 0.3%

DChargedTrackHypothesis:Reaction191.00 us ( 19.1%)

10 calls0.00 us 0.0%

DNeutralParticleHypothesis23.71 ms (2370.8%)

10 calls0.00 us 1.2%

10 calls0.00 us 5.6%

10 calls0.00 us 0.0%

DKinFitResults160.00 us ( 16.0%)

10 calls249.00 us

24.9%

10 calls0.00 us 3.2%

10 calls466.00 us

46.6%

DEventRFBunch23.02 ms (2301.8%)

10 calls0.00 us 1.0%

10 calls23.72 ms2372.2%

DParticleComboBlueprint24.61 ms (2460.7%)

10 calls2.65 s

264871.6%

DParticleID:PID10.15 s (14996.2%)

1 calls0.15 s

14996.2%

DBCALShower:KLOE658.00 us ( 65.8%)

DBCALGeometry163.00 us ( 16.3%)

11 calls156.00 us

16.3%

DBCALHit81.00 us ( 8.1%)

20 calls0.00 us 8.1%

DCDCTrackHit0.75 s (75293.0%)

DCDCHit67.00 us ( 6.7%)

10 calls0.00 us 6.7%

DMCTrackHit105.00 us ( 10.5%)

10 calls0.00 us 5.9%

1 calls0.15 s

14997.6%

10 calls0.00 us 0.0%

10 calls23.01 ms2300.8%

DFCALShower95.80 ms (9579.8%)

10 calls0.00 us 0.3%

DSCHit33.00 us ( 3.3%)

10 calls0.00 us 1.0%

DTOFPoint1.01 ms (100.7%)

10 calls0.00 us 3.2%

DTrackTimeBased0.17 s (17247.4%)

10 calls2.44 s

244192.5%

10 calls0.00 us 0.0%

10 calls0.00 us 0.2%

10 calls0.00 us 0.0%

10 calls0.00 us 0.8%

10 calls0.00 us 2.1%

DTrackTimeBased:Reaction171.00 us ( 17.1%)

10 calls244.00 us

24.4%

1 calls0.00 us 1.3%

DFCALCluster464.00 us ( 46.4%)

10 calls777.00 us

77.7%

DTOFHit1.13 ms (112.8%)

10 calls1.20 ms120.2%

1 calls0.00 us 0.3%

10 calls902.00 us

90.2%

54 calls0.00 us 4.7%

10 calls96.57 ms9657.0%

10 calls0.00 us 1.5%

10 calls2.16 ms215.6%

DFDCPseudo0.31 s (30750.2%)

54 calls0.00 us 3.4%

DTrackFitter96.00 us ( 9.6%)

1 calls0.37 s

36690.8%

DTrackHitSelector215.00 us ( 21.5%)

54 calls0.00 us 6.0%

DTrackWireBased0.40 s (39669.6%)

10 calls1.80 s

180275.6%

10 calls0.00 us 7.3%

DEventProcessor_b1pi_hists

10 calls12.16 s

1216041.9%

DFCALGeometry66.00 us ( 6.6%)

1 calls13.00 us 1.3%

DFCALHit247.00 us ( 24.7%)

10 calls0.00 us 30.0%

10 calls0.00 us 5.3%

DFDCCathodeCluster374.00 us ( 37.4%)

DFDCHit283.00 us ( 28.3%)

10 calls0.00 us 4.3%

10 calls0.00 us 4.6%

10 calls417.00 us

41.7%

10 calls0.00 us 24.0%

DFDCSegment25.56 ms (2556.0%)

10 calls0.31 s

30814.0%

10 calls0.00 us 8.5%

1 calls0.00 us 0.4%

10 calls0.00 us 1.1%

DNeutralShower212.00 us ( 21.2%)

10 calls0.00 us 7.1%

10 calls0.00 us 0.0%

10 calls0.00 us 7.5%

10 calls0.00 us 0.2%

10 calls2.62 s

262396.4%

10 calls218.00 us

21.8%

DTOFRawHit74.00 us ( 7.4%)

10 calls0.00 us 7.4%

DTrackCandidate61.83 ms (6183.3%)

10 calls0.00 us 1.1%

DTrackCandidate:CDC33.37 ms (3337.1%)

10 calls0.79 s

78627.5%

DTrackCandidate:FDCCathodes0.22 s (22380.4%)

10 calls0.56 s

55750.4%

10 calls0.75 s

75290.4%

10 calls0.33 s

33370.0%

DTrackFitter:KalmanSIMD_ALT10.37 s (36683.6%)

1 calls0.37 s

36683.6%

DTrackHitSelector:ALT1110.00 us ( 11.0%)

10 calls24.00 us 11.0%

1 calls0.00 us 2.3%

85 calls0.00 us 9.4%

85 calls0.00 us 3.1%

10 calls1.41 s

140562.3%

1 calls0.00 us 2.4%

85 calls171.00 us

26.5%

(Starts at DAnalysisResults)

Complexity of the system conKnues to grow!

2/22/13 TBD -‐ D. Lawrence -‐ JLab

3

Colorized based on Library DAnalysisResults

83.77 ms (8377.3%)


10 calls12.42 s

1242315.1%


10 calls461.00 us

46.1%


4 calls21.56 ms2159.7%


20 calls0.00 us 8.3%


20 calls0.00 us 5.6%


11 calls0.00 us 8.0%


30 calls514.00 us

73.9%


20 calls120.00 us

21.5%DParticleCombo:PreKinFit

9.57 s (956705.5%)

10 calls12.38 s

1237633.1%


1 calls0.00 us 0.9%


10 calls2.76 s

275658.0%

10 calls0.00 us 2.6%

10 calls0.00 us 10.1%

10 calls0.00 us 0.1%

10 calls0.00 us 0.2%


10 calls0.00 us 0.1%


10 calls0.00 us 1.1%

10 calls0.00 us 6.0%

10 calls0.00 us 0.0%


10 calls249.00 us

24.9%

10 calls0.00 us 3.8%

10 calls495.00 us

49.5%


10 calls0.00 us 1.1%

10 calls24.90 ms2490.1%


10 calls2.78 s

278383.1%


1 calls0.17 s

16678.5%

DBCALShower:KLOE679.00 us ( 67.9%)


11 calls155.00 us

16.2%

DBCALHit74.00 us ( 7.4%)

20 calls0.00 us 7.4%


DCDCHit69.00 us ( 6.9%)

10 calls0.00 us 6.9%


10 calls0.00 us 5.4%

1 calls0.17 s

16679.9%

10 calls0.00 us 0.0%

10 calls24.14 ms2414.5%

DFCALShower0.11 s (10571.5%)

10 calls0.00 us 0.1%

DSCHit37.00 us ( 3.7%)

10 calls0.00 us 1.0%


10 calls0.00 us 3.2%


10 calls2.56 s

255675.8%

10 calls0.00 us 0.1%

10 calls0.00 us 0.3%

10 calls0.00 us 0.1%

10 calls0.00 us 1.1%

10 calls0.00 us 2.3%


10 calls267.00 us

26.7%

1 calls0.00 us 1.2%


10 calls493.00 us

49.3%

DTOFHit1.18 ms (118.0%)

10 calls1.26 ms125.8%

1 calls0.00 us 0.3%

10 calls906.00 us

90.6%

54 calls0.00 us 4.6%

10 calls0.11 s

10620.5%

10 calls0.00 us 1.6%

10 calls2.22 ms221.7%


54 calls0.00 us 3.7%


1 calls0.41 s

41494.9%


54 calls0.00 us 6.3%


10 calls1.86 s

186184.7%

10 calls0.00 us 7.8%


10 calls12.51 s

1250738.5%


1 calls12.00 us 1.2%

DFCALHit67.00 us ( 6.7%)

10 calls0.00 us 10.9%

10 calls0.00 us 4.2%

DFDCCathodeCluster370.00 us ( 37.0%)

DFDCHit234.00 us ( 23.4%)

10 calls0.00 us 3.5%

10 calls0.00 us 4.6%

10 calls405.00 us

40.5%

10 calls0.00 us 19.9%


10 calls0.32 s

31973.3%

10 calls0.00 us 8.3%

1 calls0.00 us 0.4%

10 calls0.00 us 1.2%


10 calls0.00 us 8.5%

10 calls0.00 us 0.8%

10 calls0.00 us 7.8%

10 calls0.00 us 0.1%

10 calls2.76 s

275788.3%

10 calls220.00 us

22.0%


10 calls0.00 us 7.8%


10 calls0.00 us 1.1%


10 calls0.82 s

82052.7%


10 calls0.58 s

57831.9%

10 calls0.79 s

78658.3%

10 calls0.35 s

34540.2%


1 calls0.41 s

41488.5%


10 calls25.00 us 11.5%

1 calls0.00 us 2.3%

85 calls0.00 us 10.0%

85 calls0.00 us 2.4%

10 calls1.46 s

146419.3%

1 calls0.00 us 2.4%

85 calls180.00 us

27.1%

2/22/13 TBD -‐ D. Lawrence -‐ JLab

5

Colorized and grouped w/

labels

ANALYSIS

BCAL

CDC

FCALFDC

PID

TOF

TRACKING

DAnalysisResults76.20 ms (7620.1%)


10 calls12.17 s

1216769.7%


10 calls1.01 ms100.5%


4 calls25.49 ms2564.7%


20 calls0.00 us 9.4%


20 calls0.00 us 10.5%


11 calls0.00 us 18.9%


30 calls962.00 us116.5%


20 calls432.00 us

55.0%

DParticleCombo:PreKinFit9.42 s (942237.2%)

10 calls12.10 s

1210193.9%


1 calls0.00 us 8.1%


10 calls2.62 s

262463.0%

10 calls0.00 us 2.9%

10 calls0.00 us 14.3%

10 calls0.00 us 0.5%

10 calls0.00 us 0.4%


10 calls0.00 us 0.2%


10 calls0.00 us 1.6%

10 calls0.00 us 11.6%

10 calls0.00 us 0.6%


10 calls731.00 us

73.1%

10 calls0.00 us 10.4%

10 calls469.00 us

46.9%


10 calls0.00 us 1.1%

10 calls22.97 ms2296.8%


10 calls2.66 s

265601.5%


1 calls0.18 s

17865.8%

DBCALShower:KLOE2.05 ms (205.1%)


11 calls212.00 us

23.1%

DBCALHit187.00 us ( 18.7%)

20 calls0.00 us 18.7%


DCDCHit88.00 us ( 8.8%)

10 calls0.00 us 8.8%


10 calls0.00 us 27.9%

1 calls0.18 s

17889.5%

10 calls0.00 us 0.2%

10 calls22.01 ms2200.6%

DFCALShower98.44 ms (9843.5%)

10 calls0.00 us 0.2%

DSCHit84.00 us ( 8.4%)

10 calls0.00 us 1.0%


10 calls0.00 us 3.1%


10 calls2.42 s

241564.2%

10 calls0.00 us 0.8%

10 calls0.00 us 0.1%

10 calls0.00 us 0.1%

10 calls0.00 us 1.0%

10 calls0.00 us 2.3%


10 calls244.00 us

24.4%

1 calls0.00 us 1.3%


10 calls1.27 ms127.2%

DTOFHit1.24 ms (124.0%)

10 calls1.37 ms137.0%

1 calls0.00 us 0.4%

10 calls2.46 ms245.8%

54 calls0.00 us 5.1%

10 calls99.70 ms9969.5%

10 calls0.00 us 6.4%

10 calls2.63 ms262.9%


54 calls0.00 us 3.8%


1 calls0.37 s

36913.0%


54 calls0.00 us 6.7%


10 calls1.77 s

176653.6%

10 calls0.00 us 7.3%


10 calls12.24 s

1224490.3%


1 calls324.00 us

32.4%

DFCALHit203.00 us ( 20.3%)

10 calls0.00 us 26.5%

10 calls0.00 us 6.2%DFDCCathodeCluster

1.16 ms (116.4%)DFDCHit

694.00 us ( 69.4%)

10 calls0.00 us 6.8%

10 calls0.00 us 7.7%

10 calls1.23 ms123.2%

10 calls0.00 us 62.6%


10 calls0.30 s

29513.3%

10 calls0.00 us 11.8%

1 calls0.00 us 1.9%

10 calls0.00 us 1.1%


10 calls0.00 us 7.3%

10 calls0.00 us 0.1%

10 calls0.00 us 8.3%

10 calls0.00 us 0.9%

10 calls2.63 s

262605.2%

10 calls357.00 us

35.7%


10 calls0.00 us 13.0%


10 calls0.00 us 1.4%


10 calls0.75 s

74999.9%


10 calls0.54 s

53740.2%

10 calls0.71 s

71483.0%

10 calls0.32 s

31978.0%


1 calls0.37 s

36893.0%


10 calls114.00 us

20.3%

1 calls0.00 us 5.4%

85 calls0.00 us 9.7%

85 calls0.00 us 3.5%

10 calls1.35 s

135126.8%

1 calls0.00 us 12.4%

85 calls356.00 us

45.4%


Other JANA Features

•  Status word for each event –  64 bit number –  Globally defined bit definiKons –  Can be used for things like “Tracking Failed”

•  E.g. event was too busy as opposed to no hits in detector •  XML Node name caching used for xpath matching –  Address problem causing very slow startup when using more than just a few threads

•  Centralized mutex registraKon system for automaKc unlocking of mutexes in standard HUP signal handler –  (see next slide)


Data Challenge Lesson Learned Issue: •  jana-‐based program hangs occasionally unKl killed •  Problem not reproducible when run mulKple Kmes •  Seems to appear “in streaks”

Program hangs for > 30s while wri7ng event to disk. Possibly due to stalled network disk access.

…

JANA framework kills stalled thread and launches new one

New thread tries to lock mutex which is s7ll locked by now ex7nct thread. Waits forever…

1

2

3 A Solu=on: •  Change mutex type to ERROR_CHECKING •  Replace HUP signal handler •  Unlock mutex whenever HUP signal received •  ( need a formal JEventSink class?)

Comment: This was a consequence of having mul7-‐threaded support even though a single processing thread was used.


MIC system tesKng


Terminology

•  MIC = Many Integrated Cores •  Xeon Phi = Intel’s MIC system –  60 cores, 1GHz on a PCIe x16 card –  Other names: Knight’s Ferry, Knight’s Corner, Knight’s Landing

–  Original project: Larrabee •  Anempt to make GPU from older x86 design

•  SIMD = Single InstrucKon MulKple Data (a.k.a vector) – MMX 64-‐bits 1996 –  SSE1-‐SSE4 128-‐bits 1999, 2001, 2004, 2007 –  AVX 256-‐bits 2011 –  IMCI 512-‐bits 2012 <-‐-‐-‐-‐-‐ Intel Xeon

Comparable in price with equivalent tera-‐flop GPU, but cores are more flexible 2/22/13 TBD -‐ D. Lawrence -‐ JLab 10

Some System details •  Linux variant runs on MIC card independent of host OS

– MIC system is based on 2.4 Linux kernel –  File system not automaKcally shared

•  MIC cards can be configured to mount host’s filesystem via NFS •  Intel provides tools to run on host so logging into MIC system is not always necessary

•  Must use intel-‐provided cross-‐compiler to build executables* –  Could not build sim-‐recon because ROOT was needed –  Could not build ROOT because libX11-‐devel was needed –  ….

•  So, … JANA was built using cross-‐compiler and the included tesKng facility was used

* there may be a a way around this … 2/22/13 TBD -‐ D. Lawrence -‐ JLab 11

Performance using JANA TestSpeed plugin

MIC

HOST


Results

•  Lack of full Linux development plarorm is a real limitaKon for us –  Possible soluKons:

•  Use Intel cross-‐compiler to build libX11 and all other dependencies so ROOT can be compiled

•  Build a naKve compiler so compilaKon can be done on MIC •  Install a host OS that is binary compaKble with MIC architecture

•  VirtualizaKon? •  Single thread rate is 50 Kmes slower on MIC than Host (???) –  Compiler opKmizaKon?

May not use the 512bit SIMD vectors


MIC Summary

•  HPC group has invested some Kme to develop experKse with these devices. They think they will use them for next generaKon HPC farm

•  A single box with 240 real cores (1GHz) has a lot of computaKonal ability – Leveraging this will take some investment on our part (doesn’t come for free!)

– Univ. of Regina CS grad student (StamaKos) looking into this


TBD - halldweb.jlab.orgFeb 22, 2013 · 75.40 ms (7539.8%) DAnalysisResults:PreKinFit 25.37 ms...

Documents

Transcript of TBD - halldweb.jlab.orgFeb 22, 2013 · 75.40 ms (7539.8%) DAnalysisResults:PreKinFit 25.37 ms...