The CMS Modular Track Finder (MTF7) Trigger Board

D. Acosta, G. Brown, A. Carnes, M. Carver, D. Curry, G.P. Di Giovanni, I. Furic, A. Kropivnitskaya, A. Madorsky, D. Rank, C. Reeves,

B. Scurlock, S. Wang 

University of Florida/Physics, Gainesville, FL, USA

M. Matveev, P. Padley

Rice University, Houston, TX, USA

CMS Endcap Muon System

TWEPP 2013 A. Madorsky 2

φ

θ, η

η coverage: 1.2 to 2.4

Muon Trigger structure rework

TWEPP 2013 A. Madorsky 3

Endcap TF

Overlap TFBarrel TF

- Overlap TF is now separated from Endcap and Barrel

CMS Endcap Muon Trigger

Each of two Endcaps is split into 6 sectors, 60° each

Each sector is served by one Sector Processor (SP)

Total 12 SPs in the entire system

CMS trigger requires us to identify distinct muons

Each SP can build up to 3 muon tracks per BX

4

Trigger sector 60˚

TWEPP 2013 A. Madorsky

Endcap Muon Trigger upgrade

5

Trigger information Wiregroup

patterns Strip hits

Muon EndcapTrigger sector (60°)

UpgradedPort Cards One per station

1/6 filtering

Trigger primitives 2 per chamber18 per station90 total

18 primitives per station90 total

Fibers(~100 m)

UpgradedSector Processor Complete 3-D tracks

assembled from primitives Up to 3 tracks per BX

TWEPP 2013 A. Madorsky

Motivations for upgrade Current Endcap Trigger is adequate for LHC luminosity before the upgrade The following improvements are needed for the upgrade:

Transverse Momentum (Pt) assignment Final Pt assignment is currently done with 2MB LUT Address space is already over-saturated Need bigger Pt assignment memory

Trigger primitives bandwidth With luminosity upgrade, we expect ~7 Trigger primitives per Sector

per BX (average). Current system selects only 3 best primitives in each sector

Inadequate for the upgrade Need to import preferably all primitives on each BX

18 per sector Also need to import other data (Resistive Plate Chambers) The above means:

Higher input bandwidth Bigger FPGA

TWEPP 2013 A. Madorsky 6

7

Block diagram

Optical plant(fanouts and splitters)

From MPCs60 12-core fibers, 8 cores used in each.90 trig. primitives per 60° sector3.2 Gbps

From RPCUp to 216 fibers at 1.6 Gbps (may be concentrated to higher bandwidth and fewer fibers)

Sector Processors12 units60° sector each

Muon Sorter 8 Best Muons

to Global Muon Trigger

To Overlap Track Finder

Best 3 Muons in each sector

TWEPP 2013 A. Madorsky

8

uTCA chassis

SPSP SP SP SP

SPSP SP SP SP

SP SPMS

Sector Processor (SP) occupies 2 uTCA slots12 units in system

Muon Sorter (MS) hardware identical to SP1 unit in system

All chassis use AMC13 (designed by Boston University)

clocking and DAQ3 units

Plan to control boards via PCI-express Will make compatible with IPbus as well

Chassis #1

Chassis #2

Chassis #3

TWEPP 2013 A. Madorsky

Hardware prototype

2012 prototype based on Virtex-6 FPGA Modular design

Makes future partial upgrades easier

TWEPP 2013 A. Madorsky 9

Optical module

Custom backplaneCore logic module

Optical module

Core logic module

Pt LUT module

Custom backplane

Virtex-6 Core logic module

TWEPP 2013 A. Madorsky 10

Custom backplane connector

Core logic FPGA

Control FPGA

uTCA connector

Control FPGA JTAG

PT LUT module connector

FMM connector

SD card connector

MMC USB console

MMC JTAG

MMC CPU

Estimated power consumption: ~50 W (assuming FPGAs nearly full)PT LUT mezzanine not included

1Gb FLASHMain FPGA firmware storage

MMC = Module Management Controller

Virtex-6 Core logic module:Features

Serial I/O:53 GTX receivers (up to 4.8 Gbps)8 GTH receivers (10 Gbps)12 GTX transmitters (up to 4.8 Gbps)2 GTH transmitters (10 Gbps)

MMC – Wisconsin designSee this link for details

Configuration memory for Core FPGA:PC28F00AP30EFA

1 Gb parallel FLASH Can be used to store any other information (in addition to firmware)

Permanent configuration settings Multiple firmware versions

SD card slotCan also be used to store Core FPGA firmware and settings

Fast Monitoring (FMM) connectorCompatible with the current FMM system

Control interfaces:PCI express IPbus

TWEPP 2013 A. Madorsky 11

Optical module

TWEPP 2013 A. Madorsky 12

uTCA connector

Custom backplane connector

Backplane redrivers

Optical receivers(2 out of 7 installed)

MMC

Optical transmitters (2 out of 3 installed)

Optical module Receivers:

7 12-channel RX Avago’s AFBR-820BEZ 84 RX channels

Transmitters: 3 12-channel TX

Avago’s AFBR-810BEZ 28 TX channels (12+12+4)

All of them 10 Gbps parts Not enough space on front panel to accommodate all TX parts located inside

connect with short fibers to MPO fiber couplers on front panel Tight but enough space to fit couplers on top of AFBR-820 parts.

Receivers are on front panel to minimize count of fiber-to-fiber transitions for inputs

Control: Wisconsin MMC design, no FPGA

Compatible with future Virtex-7 design of Core logic board

TWEPP 2013 A. Madorsky 13

Glue logic FPGA(Spartan-6)

Base boardconnector

RLDRAM3 memory16 chips, 8 on each side(clamshell topology)Total size: 512M x 18 bits ≈ 1GBUpgrade possible to2 GB with bigger RLDRAM3 chips(no board redesign)

DC-DC converters

Clock synthesisand distribution

PT LUT module

14TWEPP 2013 A. Madorsky

Optical communication test@3.2 Gbps

47 input channels Transmission from:

LoopbackMuon Port CardEarlier VME prototype (2010)

For MPC and VME prototype clock was synchronized with VME crate

Twisted pair LVDS connection to uTCA backplane@10 Gbps

6 input channels Asynchronous clock Transmission from:

LoopbackEarlier 10Gbps

prototype (2006)Results

Zero errors for hours

TWEPP 2013 A. Madorsky 15

Eye pattern @10 Gbps.GTH receiver input.

PT LUT testsParameters:

RLDRAM clock : 200 MHz Address & control: 200 Mbps each bit Data: 400 Mbps each bit RLDRAM can tolerate up to ~1GHz clock. However:

Hard to implement in FPGANeeded for burst-oriented applications mostlyDoes not change latency for random address accessLower clk F lower power consumption

Tests performed (random data, full 1GB space): Writing into consecutive addresses Reading from consecutive addresses Reading from random addresses

No errors detected Except soldering defects in one RLDRAM chip

TWEPP 2013 A. Madorsky 16

PCI Express tests: Setup

TWEPP 2013 A. Madorsky 17

PC adapter

Motherboard AMC113 (uTCA PCIE adapter)

2012Prototype

Fiber (up to 50 m)

Multiple PC adapters testedBest results: HIB35-x4Also least expensive

Vendor CardVadatech PCI113

Samtec PCIEA

OneStopSystems HIB35-x4

PCI express performance Theoretical PCIe performance (generation 2, single lane):

Link bitrate: 5 Gbps

After 8b/10b encoding: 4 GbpsAfter PCIe packet overhead: ~3.56 Gbps

On top of that: SoftwareHardware overhead of a particular chipset on host systemHardware overhead of your device

Our test design has very small overhead Results of performance tests at UF:

Sustained performance, all overheads included5 meter fiber:

Reading: 2.4 Gbps Writing: 2.88 Gbps

50 meter fiber: Reading: 2.3 Gbps Writing: 2.88 Gbps

TWEPP 2013 A. Madorsky 18

Plans for Virtex-7 designCore logic moduleFPGAs:

Core logic: XC7VX690T-FFG1927 or XC7VX550T-FFG1927 Control: XC7K70-FB676

Inputs: 80 Virtex-7 GTH links (10 Gbps) directly to Core FPGA

Minimal latency All available receivers are designated for trigger data

Maximum flexibility 4 additional Kintex GTX links (6.4 Gbps) to Control FPGA

Delivered to Core FPGA via parallel channel Longer latency

Outputs: 28 Virtex-7 GTH links (10 Gbps)

Control: PCI express Gen 2, 2 lanes Ipbus

Status: PCB layout nearly doneTWEPP 2013 A. Madorsky 19

Core logic module FPGA interconnections

TWEPP 2013 A. Madorsky 20

XC7VX690T orXC7VX550T

XC7K70

PT L

UT

conn

ecto

r

28 outputs10 Gbps

80 inputs10 Gbps

4 inputs6.4 Gbps

IPbus

PCI express2 lanes

Parallel data exchange channel(control, DAQ control, 4 extra links)

Cus

tom

bac

kpla

ne c

onne

ctor

uTC

A b

ackp

lane

con

nect

or

DAQ control (RX)

4 TX to

AMC13

DAQ TX

Conclusions

Virtex-6 based prototype built and tested53 inputs up to 4.8 Gbps8 inputs at 10 Gbps RLDRAM3-based Pt LUT (1GB)PCI expressIPbus

Virtex-7 base board prototype is in its final design phase

TWEPP 2013 A. Madorsky 21

Backup

TWEPP 2013 A. Madorsky 22

1 2 3

4 5 6

7 8 9

1 2 3

4 5 6

78 9

TWEPP 2013 A. Madorsky 23

CSC trigger data sharing Endcap and Overlap processorsME1

ME2,3,4

1 2 3

45 6 7 8 9

Subsector 1 Subsector 2

Overlap TF

Endcap TFMultiple chambers have to beshared between Endcap and Overlap TFs (shown in yellow)

TWEPP 2013 A. Madorsky 24

CSC trigger data sharingNeighbor sector

ME1ME2,3,4

1 2 3

4 5 6

7 8 9

1 2 3

45 6 7 8 9

1 23

4 5 6

7 8 9

Subsector 1 Subsector 2

Neighborsector TF

Neighborsector TF

Several chambers shared with neighbor Sector Processor for better sector overlap coverage(shown in pink)Most of the chambers shared between 2 SPsSome chambers shared between 4 SPs

Optical components

TWEPP 2013 A. Madorsky 25

MPO Fanout

2-way splitter

4-way splitter

Components from Fibertronics

Optical plant (one sector)

TWEPP 2013 A. Madorsky 26

MTP connectors (inputs)

LC-LC adapters

MTP connectors (outputs)

Splitters

MTP fanouts

Slack spool

Slack spool 19” 1U

Rack-mountEnclosure

* Componentsshown not in full quantities