Post on 04-May-2022
12-Sep-2011 Dietrich Beck, GSI
White Rabbit @ GSI (BEL, EE, …)
• Motivation • Who is White Rabbit? • Associated: BuTiS • White Rabbit for Experiments • Status
Acknowledgements: WR Team @ CERN, BEL Timing Team @ GSI, EE (JH, NK,…), …
12-Sep-2011 Dietrich Beck, GSI
Motivation
• Timing system of GSI accelerator based on 1MBit/s MIL 1553-B bus (+ 3 extra lines) è not suitable for FAIR
• Development/application of a new timing system is required. • The timing control / time stamping of FAIR experiments could
benefit from a new accelerator timing system. • For me (db): Timing systems for control systems of ion-traps:
Timing is an interesting topic. • è Project for/with the BEL group
12-Sep-2011 Dietrich Beck, GSI
Who is the White Rabbit (WR)?
“ “The White Rabbit (WR) project is a multi-laboratory and multi-company effort to bring together the best of the data
transfer and timing worlds …”
"Oh dear! Oh dear! I shall be too late!“ (Proc. ICALEPCS 2009) (Alice's Adventures in Wonderland)
12-Sep-2011 Dietrich Beck, GSI
Requirements to WR at GSI
• ∼ 2000 timing receiver nodes on the GSI/FAIR campus • ∼ 100 µs reaction time “granularity window” (e.g. interlocks) • ∼ 10 µs propagation time of signals (∼ 2km) • ∼ 1-2 ns precision (kicker control) • ∼ 10-12(-15) robustness (1 event per year may be lost) • 1 GBit/s throughput • Complements other components (FESA, …) • WR is not intended for protection of machines or personnel • Collaborative effort (CERN, GSI…)
– avoiding isolated solutions – involving industry
12-Sep-2011 Dietrich Beck, GSI
Solution Path for WR • 1 GigE, extended by • IEEE 1588 (PTP protocol, time stamps, ∼ 1 µs precision), extended by • WR-PTP
– syntonization: adjust clock frequency of WR node to master (transceiver, FPGA) – Asymmetric model for link delays (up/down) – ∼ 8 ns jitter due to 125 MHz carrier of GigE
• extended by SyncE – synchronization: adjust phase of WR node to WR master – sub-ns precision
• Dedicated WR switches – clock propagation (frequency, phase) – real-time control messages + some low priority traffic – guaranteed maximum latency for real-time control messages
• Guaranteed maximum propagation time by network layout • Timing Master: Broadcasts real-time control messages • Message encoding using Forward Error Correction (FEC) • WR initiated at CERN: Collaborative effort – GSI joins • Open Hardware Repository (www.ohwr.org): Sharing effort and solutions
12-Sep-2011 Dietrich Beck, GSI
Timing Master
Control System (application layer)
LSA (settings management)
AP
I
Equipment 1 Equipment N
FES
A
WR
Equipment 2 Equipment 3
Beam Interlock System Post Mortem System
AP
I
BuTiS another link
Equipment 4
a link
Equipment ..
Common Systems
Alarm, Archive, Logging, Oscilloscope
AP
I
BuTiS
Disclaimer: This is my personal view and may be incorrect and not up-to-date
12-Sep-2011 Dietrich Beck, GSI
Timing Concept
Problem: Direct triggering of devices via a timing network is imprecise due to differences in propagation time (cable length) and “unknown” latency of WR switches (only upper bound is known), Δt ∼ 10 µs (not good enough).
Solution: Timing based on UTC timestamps: Jitter defined by precision of
distributed clocks, Δt < 1 ns (Intrinsic property of WR). • Timing Master sorts/calculates data ∼ 100 µs ahead of requested
execution time: Timing messages (node x: do action y at UTC z) • Timing messages are broadcasted considering max. propagation time. • Timing Receiver (node x) receives message: do action y at UTC z • If actual time (UTC) equals requested time UTC z: WR node x executes
action y autonomously.
12-Sep-2011 Dietrich Beck, GSI
WR versus BuTiS
WR: • UTC time stamp distribution (sub ns precision) • Clock distribution (8 ns granularity, sub ns jitter) • Cheap (< 1 kEuro per node) BuTiS: • No time stamp distribution • Best clock distribution (14 ps granularity, accuracy 100 ps/km,
low fs jitter) • Expensive (∼ 10 kEuro per receiver station)
12-Sep-2011 Dietrich Beck, GSI
BuTiS and WR
Timing Master
BuTiS Center Rb Normal 10 MHz
P0: 100 kH
z S
1: 10 MH
z
DWDM Ref. Synth. LP0: 100 kHz LS1: 10 MHz
Clock Master DDS
WR Master
WR Receiver DDS:T0
DWDM Ref. Synth. LP0: 100 kHz LS1: 10 MHz
(T0: 100 kHz) C2: 200MHz
UTC
delay correction T0/C
2 phase adjust
timing events
UTC
via PTP
UTC
/T0 phase …
T0: 100 kHz C2: 200MHz
Equipment
GPS GPS
WR Receiver DDS:T0
Equipment
WR Switch
backplane DDS: AFG
12-Sep-2011 Dietrich Beck, GSI
WR Hardware
• WR multi-purpose carrier boards for PCIe, VME, (µTCA, PXI) • SPEC (Simple Pci Express Carrier) board
• I/O defined by FMC mezzanine boards • At GSI: ∼ 1600 SCU (Standard Control Units) for FAIR
Combined with FPGA:
• WR VHDL code
• Soft-CPU (versatility, re-use existing open source C-libraries, fast reconfig.)
• User VHDL code
12-Sep-2011 Dietrich Beck, GSI
Use Cases
• Timing System for GSI/FAIR accelerator • DAQ for GSI/FAIR accelerator – “Distributed Oscilloscope” • … • Time Stamping for triggered DAQ systems
– Latch WR node via cable – Read timestamp via host bus
• Timestamp distribution for free running DAQ systems – WR node spits out 100 kHz epoch markers plus encoded
timestamps on a serial line – Fan-out from WR node to DAQ boards
12-Sep-2011 Dietrich Beck, GSI
Status
• Presented on IEEE Real Time in Stockholm, 2005 • … • Establishment of WR link between two SPEC boards
demonstrated. Synchronization to about 100 ps. • Functionality of WR switch demonstrated • 10 SPEC boards arrived at GSI • 2 WR switches arrived at GSI • … • 8 FMC mezzanine boards developed and produced (JH et al.) • 10 PEXARIA2A boards – “development board for SCU” (JH),
converts to WR node, by adding two VCOs • … • Specification (Draft) for interface to experiments • …
12-Sep-2011 Dietrich Beck, GSI
Further Reading…
• www.ohwr.org • https://www-acc.gsi.de/wiki/Timing/WebHome • Talks by Tibor Fleck and Peter Moritz @ FAIR Technikforum
http://www-bd.gsi.de/dokuwiki/doku.php?id=meetings:fair-technikforum