Post on 10-Mar-2020
EPICS Collaboration Meeting Fall 2012 @ PAL
October 22 ~ 26, 2012
LCLS-I/LCLS-II Timing System
Low Level Kukhee Kim for LCLS Timing Team
ICD Software, SLAC National Accelerator Laboratory
October 22, 2012
EPICS Collaboration Meeting Fall 2012 @ PAL
October 22 ~ 26, 2012 2
Event System is a most complicated system!
EPICS Collaboration Meeting Fall 2012 @ PAL
October 22 ~ 26, 2012 3
Accelerator complex for current and future
(LCLS-I, LCLS-II and …)
LCLS
LCLS-II
End of
linac
LCLSLCLS-II
Bypass line
|
Sector 0
|
Sector 10
|
Sector 20|
Sector 30
FEL Injector Test Facility
3 km SLAC LINAC
LTU
Undulators
Interleaved Accelerator Operation
LCLS-III ? LCLS-I
One gun, one destination
120 Hz
LCLS-II
One gun, two destinations,
plus shared tunnel,
plus multibunch?
120 Hz
Separate but synchronous timing systems
360 Hz = 6 x 60 Hz power line timeslots
Independent rate control to each destination
MPS-aware timing
TS4, and TS1
TS5 and TS2
TS6, and TS3
EPICS Collaboration Meeting Fall 2012 @ PAL
October 22 ~ 26, 2012 4
360Hz – constraint: What is the timeslots?
LCLS-I LCLS-I LCLS-II LCLS-II FACET FACET
EPICS Collaboration Meeting Fall 2012 @ PAL
October 22 ~ 26, 2012 5
Legacy Timing System
• SLC timing system
- The Linac is a Pulsed Machine (get a packet of beam per pulse) runs
at a max of 360Hz (120Hz)
- Three Main Timing Signals:
• Hardware
- 476MHz Master Accelerator Clock (runs down 2mile Heliax Main Drive Line cable)
- 360Hz Fiducial Trigger / encoded onto the 476MHz master clock
• Software – MPG (Master Pattern Generator)
- 128-Bit PNET (Pattern Network) Digital Broadcast (contains time slot, trigger setup,
beam type & rate information, beam destination)
- Provide the timing pattern 3 fiducial prior
• Advantage of the Legacy system
- Physicists & Operator are used to the old system
- Provide event patterns 3 fiducial prior
• give a time margin to slow device, or pre-trigger up to 2 fiducial ahead
EPICS Collaboration Meeting Fall 2012 @ PAL
October 22 ~ 26, 2012 6
Old architecture for the LCLS-I Timing System
D
E
V
~ Linac main drive line
Sync/Div
SLC
MPG
P
N
E
T
119 MHz 360 Hz
SLC
events
LCLS
events
P
N
E
T
m
P
P
D
U
E
V
R
I
O
C
TTL-NIM
convert.
Digitizer
LLRF
BPMs
Toroids
Cameras
Wire Scanner
SLC klystrons
TTL
SLC Trigs
F
A
N
I
O
C
Low Level RF
EPICS Network
Precision<10 ps
fiber
distribution
LCLS Timing System
components are in RED
*MicroResearch
*
*
E
V
G
LCLS Master
Oscillator
476
MHz Linac
Master Osc
System is based around the
EVent Generator and EVent
Receiver
FIDO PDU
Raw 360 Hz LCLS Timeslot Trigger
EPICS Collaboration Meeting Fall 2012 @ PAL
October 22 ~ 26, 2012 7
LCLS-I Timing System
• LCLS Timing System
• Use MRF EVG and EVRs
• Generate event code at 360 Hz based on:
- PNET pattern input (beam code + 128 bits pattern that define beam
path and other conditions, scheduled information)
- Receive MPS*) information (real-time information), and queue it up
into the pattern pipeline (pipeline bypass for quasi real-time)
- Add LCLS information (MPS, BPM calibration, diagnostics and ….)
• Extend the pattern to 192 bits
- Hardwired BCS fault
• Send out timing pattern, including EPICS timestamp with encoded
pulse ID on the timing fiber
• Manage user-defined Beam Synchronous Acquisition (BSA)
MPS*) Please, see the talk for LCLS Machine Protection System (M. Boyes)
in hardware solution session (16:30 – 16:50 Oct 22, 2012)
EPICS Collaboration Meeting Fall 2012 @ PAL
October 22 ~ 26, 2012 8
How to work the EVG
• PNET message broadcast from the old timing system (following a
fiducial, 360 Hz rate)
• PNET receiver in EVG IOC generates IRQ
• EVG software takes the PNET data and uses it to assign the proper
event codes (the PNET data pipelined 3 fiducial ahead)
• Event code setup in EVG one cycle head
• Next fiducial is sent: 360 Hz fiducial signal received by EVG
• The EVG begins to send out the event codes in its Sequence RAM
• The event codes get sent across the timing fiber links to the EVRs
• Inside the EVR, its mapping RAM is set to map a specific HW trigger to
an event code
• when the Event Code matches the same value in the mapping RAM, a
hit is generated and after a programmed delay, a HW trigger is output
form the EVR to the device
EPICS Collaboration Meeting Fall 2012 @ PAL
October 22 ~ 26, 2012 9
How to work the EVR
• Set trigger delays, pulse width, and enable/disable via
user requests
• Set event code per trigger
• Receive timing pattern ~8.3 msec (3 fiducials prior) before
corresponding pulse. Provides EPICS timestamp to record
processing
• Perform BSA based on tags which are set by EVG in the
timing pattern
• Process pre-defined records when specific event codes
are received
EPICS Collaboration Meeting Fall 2012 @ PAL
October 22 ~ 26, 2012 10
Timeline for the EVR
F0
0
Fiducial
Time (usec)
B0
Acq
Trigger
1023
Beam Kly Standby
Record processing (event, interrupt)
Fiducial Event
Received
Event Timestamp,
pattern records,
and BSA ready
Receive pattern for
3 pulses ahead
Hardware Triggers
~500 0.3
~40 100
Triggering
Event Codes
Kly Accel
F1
Fiducial
… 2778
110
Start End
EPICS Collaboration Meeting Fall 2012 @ PAL
October 22 ~ 26, 2012 11
Upgrade Timing system for both LCLS-I and LCLS-II
• Main Goals
• Remove the dependency with the SLC timing system
• Independent EVG for each beam program - LCLS-I, LCLS-II, FACET, XTA and future beam programs
- Promise for independent operation
- Relieves the pain from limited event number (255 limitation)
• Keep compatibility with old system, keep SLAC specific features
(advantage of the legacy system) - Beam code and timing pattern pipeline:
• For pattern aware operation
- BSA
- Event Code Sequence for photon group
• Covers various platforms for EVR: RTEMS, linux, and linuxRT - LCLS-I, and LCLS-II both are going to use various hardware platforms
• VME, PMC, Standard PC, uTCA, and COM-X
• Need to support linux/linuxRT for new hardware platform
EPICS Collaboration Meeting Fall 2012 @ PAL
October 22 ~ 26, 2012 12
Timing System Upgrade Cont’d
• Upgrade
• Develop a new hardware called VMTG: VME based Master
Trigger Generator which provides 360 Hz IRQ to drive EVG
software instead of the PNET receiver (VMTG in EVG side)
• Develop new software module PABIG: Pattern Bits Generator
which mimics the PNET
• Develop a new software tool to design the event/timing
pattern visually: evGUI1)
• Work on the EVR software to support linux/linuxRT2) for the
new hardware platform: uTCA, and COM-X
evGUI1) Please, see the talk for LCLS timing system for pattern design , evGUI and high level (M. Zelazny)
in the timing system session (15:00 – 15:20 Oct 22, 2012)
Support for linux/linuxRT 2) Please, see the talk for Real-Time Performance Issues on linux/linuxRT (K. Kim)
in the low level application session (14:10 – 14:30 Oct 23, 2012)
EPICS Collaboration Meeting Fall 2012 @ PAL
October 22 ~ 26, 2012 13
New architecture for the timing system
Device
Analog
Front End
Device
Analog
Front End
EVR
DAQ
IOC
EVR
DAQ
IOC
Linac LLRF Phase Distribution~
LCLS
Local
Oscillator
Synchronizer
Divider
Fiducial
Detector
EV
G
Fa
no
ut
IOC
Timing
Reference
Distribution
360 Hz
119 MHz 360 Hz
119 MHz 119 MHz Fiber Distribution
Timing Event System Fiber Distribution
EPICS CA Network
476 MHz
Device
Analog
Front End
Hardware trigger
signal
Beam
Synchronous
Acquisition
BSA
Data Buffers
To clocked devices
To LCLS-I
EVG
EVR
DAQ
IOC
Master
Oscillator
MPS Link Node MPS Network
LCLS-II CD-2 DOE Review / Aug 21 - 23, 2012
EPICS Collaboration Meeting Fall 2012 @ PAL
October 22 ~ 26, 2012 14
VMTG
Sector 0 (LI00) VMTG – Master Trigger Generator
• Provides the synchronized 360Hz trigger to the fiducial generator
• Input: 360Hz AC trigger train + 476 MHz MDL RF
• Ouput: 360Hz synchronized trigger
• Then, the fiducial generator makes synchronization with the Master
Oscillator
Sector 20 (IN20) VMTG with the EVG – Master Trigger
Generator
• Provide 360 Hz interrupts with the Timeslot counter
• Provide Timeslot Synchronization
• Input: 60 Hz AC trigger (AC phase C) + Synchronized 360 Hz trigger
• Output: 360 Hz interrupts + timeslot counters
EPICS Collaboration Meeting Fall 2012 @ PAL
October 22 ~ 26, 2012 15
Disabling PNET allows VMTG control
• PNET task needs to be delayed ~ 150 usec after the PNET IRQ to wait the MPS
message.
• PNET IRQ is using HW_TIMER to make the delay
• The PNET ISR implement 150 usec delay on the HW_TIMER
• The callback for the HW_TIMER releases signal to execute the PNET Task.
• VMTG has configurable IRQ delay
• Configure the VMTG IRQ delay as ~570 usec
• The VMTG IRQ occurs 150 usec after the PNET IRQ
Fiducial
PNET
VMTG
Unknown delay
PNET IRQ
ISR implements the timer
Timer callback releases the signal
to the PNET task
PNET task
VMTG IRQ
The ISR releases the signal to the PNET task
~150 usec
~ 570 usec
MPS message arrives somewhere before the signal.
EPICS Collaboration Meeting Fall 2012 @ PAL
October 22 ~ 26, 2012 16
Software Stacks for the new EVG application and the
PABIG
PNET VMTG Acromag DIO EVG
BCS SBI Permit BCS Beam Permit 119MHz Clock 360Hz Fiducial Trigger PNET broadcasting
Asyn module VMTG
module
EVG
Drv/Dev
PABIG GenVar Master
Beam
Control
Bits 0 to 8
via CA
HW input bits in PABIG EVG Application
EPICS
DB
EDEF reservation for the BSA
PNBN PVs from SoftIOC
SNL program
476MHz Clock
360Hz Trigger
360Hz Fiducial Trigger
2nd NIC
udpComm
mpsComm PNET Task
MPS message
(beam destination &
Permit)
EPICS Collaboration Meeting Fall 2012 @ PAL
October 22 ~ 26, 2012 17
EVR Software Stacks/ Form factor & OS dependency
VME EVR Hardware PMC EVR Hardware
mrfCommon/mrfVme64
drvErMrf
devErMrf
erRecord BSA
EVR Processing Logic
Event Module for RTEMS/vxWorks
erapi
drvLinuxEvr
devErMrf
erRecord BSA
EVR Processing Logic
Event Module for linux/linuxRT
PMC EVR Hardware
Works with old register map Works with modular register map (new)
EPICS Collaboration Meeting Fall 2012 @ PAL
October 22 ~ 26, 2012 18
Summary
•Successfully upgraded timing system
• Standalone from the SLC system
• Same code base for LCLS-I, LCLS-II and future projects
• Backward compatibility, keep the advantages from the legacy
system
•Our Team has done the following work:
• Develop new VMTG hardware: John Dusatko
• Develop PABIG software: Till Straumann
• Develop evGUI: Mike Zelazny and Partha Natampalli
• EVG software integration,
EVR software support for linux/linuxRT and real-time performance
issues: Kukhee Kim, Till Straumann, Stephanie Allison
EPICS Collaboration Meeting Fall 2012 @ PAL
October 22 ~ 26, 2012 19
Related Talks
• LCLS Machine Protection System (M. Boyes)
in hardware solution session (16:30 – 16:50 Oct 22, 2012)
• LCLS timing system for pattern design , evGUI and high level (M. Zelazny)
in the timing system session (15:00 – 15:20 Oct 22, 2012)
• Real-Time Performance Issues on linux/linuxRT (K. Kim)
in the low level application session (14:10 – 14:30 Oct 23, 2012)
EPICS Collaboration Meeting Fall 2012 @ PAL
October 22 ~ 26, 2012
Thank You!