High Speed Signal/Data Transmission in BESIII Trigger and PANDA TDAQ Systems

I would like to thank Prof. Y. Sakai and Dr. Y. Iwasaki for their kind help in BESIII trigger design, and I am happy that we have this chance to share our experience of BESIII in KEKB Trigger and DAQ design.

LIU,Zhen'An, TriggerGroup,IHEP

High Speed Signal/Data Transmissionin BESIII Trigger and PANDA TDAQ SystemsZhenAn LIUTrigger Group, IHEP Beijing

2nd Open meeting for the KEKB proto-collaboration July 3-4th 2008


OutlineBackground infor for BESIII triggerStudy of High speed signal transmission in BESIII triggerBESIII trigger PANDA TDAQComputer NodeProof of Concept Application: HADES DAQ UpgradeComments and conclusion


Key points in BESIII trigger Design Optical isolation with FEE, to prevent from ground loop current interferenceMost latest FPGAs, Boards with simplicity, high reliabilityFPGA online downloadable via VMEGeneralized hardware, firmware for different function, for easy maintenanceScheme optimization with simulation


Difficulties for MDC trackingDifficulties Bad number of wires for both axial and stero layers (trigger point of view, Hard to define Sector/board border for signal input) Hard to input signals via 9U front and rare pannels Too many sharing signals for neighbor boards Solution: RocketIO for input signals (32bits/ch, 8b/10b)private VME J3 Backplane for sharing signals

1234SL 140444856/16SL 264728080/16SL 376768888/8SL 4100100112112/16SL 5128128128128/16SL10128/256128/256128/256128/256/16


Study of High speed signal transmission in BESIII trigger


HFBR-5921L

Transmitter1

Transmitter2

Fiber

HFBR-5921L

Fiber

Transmitter16

HFBR-5921L

RocketIO

HFBR-5921L

HFBR-5921L

RocketIO

RocketIO

RocketIO

Fiber

XC2VP50

RocketIO

HFBR-5921L

RocketIO

XC2VP2

Receiver

XC2VP2

XC2VP2

MFT (MDC Fiber Transmitter)2796 hits signals from MDC QT boards are collected in MFT, 32 channels per MFTVirtex-II Pro FPGA: XC2VP2Fuctions:Stretching to 500nsSynchronization + alignment signals(private Protocol)Serialization(8b/10b)


TKF (TracK Finder)XC2VP40 : FF1152, 804 user IOs, 43,632 logic cells, 3,456Kbit BRAM, 12 RocketIOs, 2 PowerPCs, 192 multiplier blocks10 layers 9UVME PCBXC2VP40HFBR5921LFunctions: Deserialization Channel alignment TSF TF Track information


Clock correctionK codesCOMMAIDLERXRECCLKRXUSRCLK


, 0111011010 , 1101001010 , 0100111010 , 11000011110010111001110010011

COMMA

10bit

10bit

10bit

Data alignmentContinuousParallel bits transmitted in same clock => recovered also in same ClockSpecial private protocol


RocketIO

FIFO-TX

Protocol-TX

MUX

PRBS

SYSCLK

LocalCLK

TriggerSignals

MUX

RocketIO

Protocol-RX

FIFO-RX

SYSCLK

LocalCLK

TriggerSignals

PRBS Check

Frst

Transmitter

Receiver

Frst

RocketIO

FIFO-TX

RocketIO

TriggerSignals

FIFO-RX

TriggerSignals

RocketIO

FIFO-TX

RocketIO

TriggerSignals

TriggerSignals

FIFO-RX

Delay1

Delay2

Channel1

Channel2

m

m+n

ClocksCrystalMust follow the recommendationUse built-in DCMClocksREFCLKUSRCLKUSRCLK2


Example of RocketIO InstantiationRocketIO_1 : GT_CUSTOM generic map( CRC_END_OF_PKT => "K29_7", CRC_FORMAT => "USER_MODE", CRC_START_OF_PKT => "K27_7", TX_CRC_USE => TRUE, TX_DATA_WIDTH => 4, );port map (REFCLK=>REFCLK_IN, TXUSRCLK=>TXUSRCLK_IN, TXUSRCLK2=>TXUSRCLK2_IN, TXCHARISK(3 downto 0)=>TXCHARISK_IN(3 downto 0), TXDATA(31 downto 0)=>TXDATA_IN(31 downto 0), TXFORCECRCERR=>TXFORCECRC_IN, TXN=>TXN_OUT, TXP=>TXP_OUT,);


Transmission protocolTransmitter sideReceiver side


CommaReset FIFO

Frst

Reset

AlignmentEnable FIFO Writing

Enable FIFO Reading

CRC SOP

ILDE

Disable FIFO Reading

CRC EOP

Empty

Not Empty

1

2

3

4

5

6

7

8

Empty

Not Empty

Good reliability with RocketIOBER 13.010e13 40errorsperiods 40

MDC Sub-Trigger


BESIII trigger Block diagramTCBAGlobal Trigger Logic6.4 sTOF FEEMDC FEEEMC FEEMU FEETOFPRMFTMu trackFastORTrack CounterEtotal EnergyHit/Seg CountTSF + TFBEPCII RFTTCTC SumL1P41.65 MHz Track MatchEnergy BalanceCluster CountingFast ControlFC DaughtersNear DetectorsCounting Room499.8 MHz High Lights: Optical Isolation-no ground loop currentTKFTKFTKF


InstallationJumper BoxMFTs in FEE crate


TOFPR


InstallationTCBA: EMC preprocessor and transmitter


Installation contdTDCEMC/TOF/GTLTriggerMDCTriggerOptical FibersOpticalCablesCLK + FCOpt-Cable under


Status of BESIII triggerSuccessful in Cosmic-ray test Commissioning with BEPCII


PANDA: DAQ Requirements Interactions: 10**7 HzData: 200GB/sContinuously Sampling ADCNo hardware triggerHi Speed DevicesLarge BuffersLarge Bandwidth


PANDA DAQ & TriggerDetectors Front-end2 Alternative DAQ Concepts Still Under Discussion


The Compute Node(CN)5 Virtex4 FX60 FPGALarge Computer Power10 GB DDR2 RAM (2GB per FPGA)Buffering capabilities2 Embedded PowerPC in each FPGASlow control32Gbit/s Bandwidth13x RocketIO to backplane5x Gbit Ethernet Front Pannel1x Gbit Ethernet Backplane8x Optical LinksATCA CompliantManageability


The Compute Node


Prototype of the CN by IHEP BeijingBackplaneFPGA #OFPGA #1-4Front PanelOptical LinksEthernet Plugs


Proof of Concept Application:HADES DAQ Upgrade Est.: 2009 Read Out TriggerTRBFaster ReadoutTrigger and DataOptical LinksInclude Tracking in Trigger12 Compute Node1 Full ATCA shelfCOMPUTE NODEOnline TrackingRICH & TOFMatching with TrackingEvent Building on FPGAOthersRemote UpgradeIPMI


Motivation/AimProduce a Configurable and Scalable Hardware Platform for Multiple Applications & Experiments

Capable of High Performance ComputingLarge BandwidthReal Time processingTriggerFlexibility: ReusableHADES - BESIII - PANDAScalabilityFlexible network topology


Comments and ConclusionRocketIO can be used for HS transmission at KEKBAwareness to KEKB TDAQ design: System clockLarger trigger latencyLatency by RocketIO (SEDES takes time )Needs larger pipeline buffer in FEESynch. + alignmentUncertainty in recovered clockOpt-cable length differenceOpt-transceiver difference



A Picture of HADES here. -> Just flash it before BESIII*Mention Krystoff, Igor, Alexander Mann.*Mention Johannes talk*Bigger PhotoNew Photo from ZhenAn?

*1 Full ATCA.12 Compute Node.Replace the entire Trigger and Event Building in HADES. *

High Speed Signal/Data Transmission in BESIII Trigger and PANDA TDAQ Systems

Documents

Transcript of High Speed Signal/Data Transmission in BESIII Trigger and PANDA TDAQ Systems