Post on 01-Apr-2015
Tracker Strip and Pixel FEDs
John Coughlan
Tracker Readout Upgrade Meeting
September 12th 2007
Strip FED
9U VME64x
96 optical fibres
Analogue ORx
96 ADC channels
Limited by I/O Component density
1 nsec timing
Board mounted
Digital FPGAs Xilinx
Cluster Algorithms
Raw input data rate >3 GB/s.
Processed Output rate < 200 MB/s
VME FPGA
Front-End data processing FPGA
Power DC-DCs on board
Output to DAQ
Event Builder FPGA
ModularFE Unit
96 ADC channels
Double Sided Board
Opto Receivers
TTC rx
450 x boards + Slinks
Synch APV
Cost Driven
Strip FED Board Manufacture
Board parameters:- 9U x 440 mm VME64x form factor- Optical/Analogue/Digital logic ; 96 ADC channels- Double-sided (secondary side with half of analogue channels)- 6,000 components (majority of passives 0402) (finest pitch < 20 thou)- 25,000 tracks- 37 BGAs (typical FPGA 676 pins on 1mm pitch). All BGAs located on primary side.- 14 layer PCB (incl. 6 power & gnd)- controlled impedance
High density components.Close up of analogue section on primary sideAlmost all components on board are Surface Mount.
VALUE is in the COMPONENTS
Design for TEST
Must have HIGH YIELD
Strip FED Quality Control
1. Custom CMS TestsAt Assembly Plant
Boundary ScanAnalogue
3. Tests at CERNPrevessin 904
B186 Tracker Integration
2. Tests at RAL
Optical, SLINK,Full crate
4. Installation at CMSUSC55
0. Quality Controlsduring Assembly
processAOI, X-ray
RAL System Rig
Assembly Plant FED
Optical Test SystemFrom Imperial College
RAL Test Rig
CERN
Pixel FED9U VME64x
36 optical fibres
Analogue ORx
36 ADC channels
Limited by SLINK output rate
1 nsec timing
Digital FPGAs Altera
Inputs are ZS
Unsynchronised Events on fibres
ROC 16 events
Build DAQ Events
Timestamp TTC
Encoded header
Pixel FED
9U VME64x boards
+ Slink
Mezzanine cards 9 Analogue,5 FPGA Altera cards(not ORx)
Common Electrical SLINK
P3 64 bits @ 80 MHz
+ TTS throttle FMMs
40 x 9U boards + Slinks
Reduces Motherboard complexity 10 layer(Micro vias for Alteras on mezz)Less risk. Testing, repairing easier. Spares
Takes more spaceConnector reliabilitySignal integrity. Terminations
Power
Strips and Pixel FEDs–Same Form Factor 9U VME . 450 vs 32 boards
–Both Analogue & Digital designs
–Same Interfaces ORx, ADCs
–Different Constraints on Channel Count ; I/O vs Output rate
–Same TTC, SLINK
–Same fine clock skew 1 nsec
–One board type in each system.
–No trigger functions.
Minor differences, power supply, slink connectors, fpga config, vme monitor bus
Different PCB implementations
- Motherboard vs Motherboard + Mezzanines
Main Difference Digital Processing. (Altera vs Xilinx FPGAs)
- Data Inputs. Raw & Synch vs ZS & Unsynch
FED Lessons“Good Design and Robust Manufacturing”.
–Design evolves in response to technology (ASICs in TDR) –Invested a lot to get design right first time (2 board versions.)
–Protoype to production was a long process.
–Started at cutting edge finished with v. mature technology FPGAs (support)
–Cost estimates evolve ADCs, FPGAs, Manufacture. Got FPGAs right.
–Support of detector development v. underestimated , PMCs cf 9U FEDs
–Prototype support underestimated ~ 50 proto + pre-prodn 9U FEDs
–Tender process takes a long time. (not needed for pxFED)
–Both ended up with “local” manufacturers. Good relationship essential.
–Quality Control critical. Early BGA failures. V. High production yield. (also pxFED) No problems ORx assembly
–Doing board testing at Assembly Plant took effort but paid off
- Need for Custom Test equipment Optical Testers
FED Lessons–System interfaces came late. Electrical SLINK cards, FMMs. More elegant solutions.
–Choice of industry standard VME for mechanics, power was good one
–VME bus as monitoring is poor , better Ethernet (late design change?)
–VME64x features not all used. (plug & play cost pxFED one unnecessary board iteration). Keep it simple.
–Think about practical issues cf prototypes with final system , FPGA configuration.
–Design issues with cooling. Tests in final configurations necessary.
–(Un)expected effects (temperature variation of optical inputs for pxFED encoding)
-Boards are built, debugged and delivered. But Firmware is never finished.
–Custom protocols , SLINK, TTC, -> custom Firmware. Duplication of Firmware blocks (across CMS), SLINK, TTC, I2C, VME.
– Firmware Libraries? Commercial/Vendor
–Duplication of test software? Software effort ? (costings)
Future FED Possible Implementation
FPGASwitch
Off Detector sFED
SDRAMBuffer
SNAP 12
Rear Transition Module
STTC
10G SerialBackplaneDAQCrate Event Builder
12 x 3 Gbps
FE
ATCA Crate 8U
Cntrl/Mon
PowerORx
FPGAsMGBTsGBT PHYMACSparsification1 per ORx
ORx and FPGA on Mezzanine?
ORx
ORx
ORx
ORx
ORx
ORx
ORx
Mezzanine Prototype
GBT
Industry StandardsATCA cratesIndustry Protocols Data TransmissionSerial PCIe …More use of Commercial Firmware cores. Data protocols, memory
COTS Carrier Motherboards+ CMS Mezzanines,Transition cards
Future FED-FEC
Integrate FEC functions on FED
FPGASwitch
Off Detector sFED
SDRAMBuffer
MPT 12
Rear Transition Module
STTCTriggerThrottle
10G SerialBackplaneDAQCrate Event Builder
12 x 3 Gbps
sTTC
FPGAsMGBTsGBT RecvMACZS1 per ORx
sAPV
ORx
ORx
ORx
ORx
ORx
OTx
ATCA Crate 8U
Cntrl/MonEthernet
PowerORx
ORx and FPGA on Mezzanine?
Digital Inputs
Zero Suppressed FE Data Inputs?Constraint Data Volumes on output
Final System Ideas New sDAQ (sFEDs connected direct to Filter via Super Event
Builder Network) New sTTC (Broadcasting Filter Addresses to FEDs) Crates
Just Mechanics, Power, Cooling. -> Control/Monitoring via Ethernet.
Serial Backplane based crates (Telecom ATCA , VME46?). Less Slots (but wider) Better Power & Cooling ? Better control & monitoring ? FED Event Builder Crate
Future FEDs–Common Tracker FED h/w probably technically feasible. Practical?
–Common CMS FED ? Common SLINK, Common FEC
–Go to Digital Input Data
–Zero Suppressed at FE tbd
–Large systems channel counts. Large form factors (cost driven).
–Constraints on Channel count?
–Not considered Tracker Trigger
–Use of Emerging Industry Crate Standards e.g. Telecomms ATCA (VME)
–Exploit Industry Data Protocols e.g. Serial PCIe (SLINK)
–More use COTS Vendor Firmware cores, Industry standards
–Use common COTS ATCA Carrier boards
– with custom CMS Mezzanines (cost effective)
Spare Slides