John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 20111

The LCLS Timing & Event System- An Introduction –

John Dusatko / Instrumentation & Controls - EIE

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 20112

Outline

1. Introduction to LCLS2. Some background on SLAC Timing3. The SLAC Linac Timing System4. The LCLS Timing System5. Comments / Future Directions

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 20113

Introduction

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 20114

LCLS Introduction

The Linac Coherent Light Source is a tunable X-ray FEL based on the SLAC Linac:

1.0nC, 14-4.3 GeV e- are passed thru an undulator, a Self Amplifying Stimulated Emission process produces 1.5-15 Angstrom X-Rays.

LCLS is an addition to the existing SLAC Linac: it uses the last 1/3 of the machine►This is important to note because we have to integrate

the New LCLS Timing System with the Existing Linac (SLC) Timing System.

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 20115

The LCLS – Schematic View The LCLS – Schematic View (ignoring photon beamline)(ignoring photon beamline)

Single bunch, 1-nC charge, 1.2-Single bunch, 1-nC charge, 1.2-m m sliceslice emittance, 120-Hz repetition rate… emittance, 120-Hz repetition rate…

(RF phase: (RF phase: rfrf = 0 is at accelerating crest)= 0 is at accelerating crest)

SLAC linac tunnelSLAC linac tunnel research yardresearch yard

Linac-0Linac-0L L =6 m=6 m

Linac-1Linac-1L L 9 m9 m

rf rf 25°25°

Linac-2Linac-2L L 330 m330 mrf rf 41°41°

Linac-3Linac-3L L 550 m550 mrf rf 10°10°

BC-1BC-1L L 6 m6 m

RR5656 39 mm39 mm

BC-2BC-2L L 22 m22 m

RR5656 25 mm25 mm LTULTUL L =275 m=275 mRR56 56 0 0

DL-1DL-1L L 12 m12 mRR56 56 0 0

undulatorundulatorL L =130 m=130 m

6 MeV6 MeVz z 0.83 mm 0.83 mm 0.05 %0.05 %

135 MeV135 MeVz z 0.83 mm 0.83 mm 0.10 %0.10 %

250 MeV250 MeVz z 0.19 mm 0.19 mm 1.6 %1.6 %

4.54 GeV4.54 GeVz z 0.022 mm 0.022 mm 0.71 %0.71 %

14.1 GeV14.1 GeVz z 0.022 mm 0.022 mm 0.01 %0.01 %

...existing linac...existing linac

newnew

rfrfgungun

21-1b21-1b21-1d21-1d XX

Linac-Linac-XXL L =0.6 m=0.6 mrfrf= =

21-3b21-3b24-6d24-6d

25-1a25-1a30-8c30-8c

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 20116

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 20117

LCLS Timing – Some Definitions

The LCLS Timing System can be viewed as consisting of three parts:

Part 1: ‘Standard’ Accelerator Timing10ps Triggers for Acceleration and Diagnostics

Part 2: S-Band Timing2856MHz LCLS RF Phase Reference Distribution

Part 3: Ultra-Precise Timing10fs Synchronization for Experiments (LBL System)

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 20118

LCLS Timing – Some Definitions

The LCLS Timing System can be viewed as consisting of three levels:

Part 1: ‘Standard’ Accelerator Timing10ps Triggers for Acceleration and Diagnostics

‘Triggers’ are signals from the timing system used by HW to accelerate & measure the beam

Part 2: S-Band Timing2865MHz RF Phase Reference Distribution

Part 3: Ultra-Precise Timing10fs Synchronization for Experiments (LBL System)

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 20119

LCLS Timing – Performance RequirementsLCLS Timing System Requirements

Maximum trigger rate: 360 Hz (120Hz)

Clock frequency: 119 MHz

Clock precision: 20 ps

Delay Coarse step size: 8.4 ns ± 20 ps

Delay range; >1 sec

Fine step size: 20 ps

Max timing jitter w.r.t. clock; 10ps rms

Differential error (skew), location to location:

8 ns

Long term stability: 20 ps

Signal Level: TTL

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 201110

Requirements Comparison

Timing Reqmts for earlier SLAC systems:Original Linac:

(~1968)

- Resolution: 50 nSec

- Jitter: 15 nSec

- Main Trigger Line

Waveform:

+ / - 400 Volts

PEP – II:

(~1998)

- Resolution: 2.1 nSec

- Jitter: 20 pSec

- NIM Level

Waveform:

0 to –0.7 V into 50 Ohms

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 201111

Background

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 201112

SLAC’s Timing Systems

In order to explain the New LCLS timing system, we first need to understand how the old SLAC timing system works – i.e. how we got from there to hereThe SLAC Accelerator complex consists of several machines: Linac, Damping Rings, Stanford Linear Collider, PEP-II, FFTB, NLCTA / each with its own timing sub-system►The overall timing system consists of incremental add-

ons to the original system►Design Challenge for LCLS Timing System was that it

had to know about and work with the existing system

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 201113

(pre-LCLS) SLAC Accelerator Complex(Lots of Pieces)

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 201114

SLAC Linac Timing System

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 201115

Old SLAC Timing System

We’ll talk a little about the existing SLAC Linac Timing System (PEP-II Timing was very similar):The Linac is a Pulsed Machine (get a packet of beam per pulse) runs at a max of 360HzThree Main Timing Signals:

476MHz Master Accelerator Clock (runs down 2mile Heliax Main Drive Line cable)360Hz Fiducial Trigger (used to ‘tell’ devices when the beam bunch is present) / encoded onto the 476MHz master clock128-Bit PNET (Pattern Network) Digital Broadcast (contains trigger setup, beam type & rate information)

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 201116

Some More Details

Why 360Hz?Original design rate of the Linac / derived from the 3-phase, 60Hz AC power line frequency: want to trigger devices (Klystrons, etc.) consistently so as to not create huge transients on the Power Line Sync’d to 476MHz

PNET BroadcastA special computer called the Master Pattern Generator (triggered by the 360Hz fiducial) broadcasts a 128-bit digital message containing information (conditions, rate, charge, etc.) about the beam This is used by the Trigger Generator computers to set up triggers and their delaysSent over SLAC coax cable TV network

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 201117

How The 360Hz is GeneratedThe Sequence Generator

creates a 360Hz signal as well as 6 Timeslot pulses (used for

further synchronization)

Source: SLAC Blue Book

c.1962

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 201118

Timing HW at Head-End of Linac

For Phase Stabilization

Source: D. Thompson

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 201119

The 360Hz Signal is Amplitude Modulated onto the 476MHz Accelerator Clock and propagated down the Main Drive Line.

The AM process is not ideal and some FM occurs; in addition, the signal gets more dispersed as it heads down the 2 mile MDL

Generation of The Linac Timing Signal

(AM Modulator)

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 201120

Linac RF Phase Reference DistributionThis slide is to give you an idea of how the Linac Phase Reference is used

Each sector (30 total) taps off the MDL, to extract the RF clock

This is just the Phase Ref, trigger generation is accomplished by a different set of HW

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 201121

Old Timing System (CAMAC based) generates triggers by combining the RF Clock, 360Hz Fiducial and PNET Data (processed by local micro)

Timing CAMAC Crate

PNET Data on Serial Link

476MHz is divided/4 to get 119Mhz + fiducial by another system / This is because the older technology HW could not run at 476MHz

The Programmable Delay Unit (PDU) Module generates the triggers. It contains digital counters that get set with delay values from PNET and started when the Fiducial Pulse comes along

CAMAC Trigger Generation

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 201122

The LCLS Timing System

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 201123

Finally – The LCLS Timing System

Old CAMAC System is no longer viable for new Systems (performance limited, obsolete)Seek to implement a new Timing System that has similar functionality, better performance, and can be laid atop the old system, working alongside itIn addition, LCLS has have its own master oscillator (PLL sync’d with Linac MO) and local phase reference distribution system at S20

►LCLS System is VME based, using High-Speed digital serial links to send Clock, Trigger and Data all on one optical Fiber to timing clients

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 201124

The LCLS Timing System

New Timing HW:Event Generator (EVG): Similar to the MPGEvent Receiver (EVR): Similar to the PDU

Timing is more closely integrated into each sub-system:

Each Subsystem has its own EVR (living in VME crate with an IOC)Compared to SLC timing which had multiple types of devices served by one PDU typically

LCLS Timing is architected as a Star network with one master broadcasting to many clients

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 201125

Timing Compare/ContrastSLC Timing:

CAMAC Based

Copper data cable @ 5Mb/s

PDU Triggers:

NIM-Level

Width: Fixed @ 67ns

Polarity: Fixed NIM level

Delay Range: 0…2.7ms (5.4ms)

Step size: 8.4ns (100ps w/VDU)

16 outputs per PDU

One PDU per CAMAC crate

SCP-Based

LCLS Timing:V

ME Based

Fiber data & clk cable @ 2.38Gb/s

EVR Triggers:

TTL-Level (can have others)

Width: Adjustable: 8.4ns…550us

Polarity: Selectable Pos/Neg

Delay Range: 0…36 seconds

Step size: 8.4ns (420ps w/EVR-RF)

14 outputs per EVR

Can have multiple EVRs per VME Crate

EPICS-Based

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 201126

LCLS RF Front End LCLS Master Osc – slaved to Linac MO / Lower Phase noise req’d by LCLS

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 201127

DEV

LCLS Timing/Event System Architecture~ Linac main drive line

Sync/Div

SLCMPG

PNET

119 MHz 360 Hz

SLCevents

LCLSevents

PNET

P

PDU

EVR

OC

TTL-NIMconvert.

DigitizerLLRFBPMsToroidsCamerasWire ScannerSLC klystrons

TTL

SLC Trigs

FAN

OC

Low Level RF

EPICS Network

Precision<10 ps

fiber distribution

*MicroResearch

*

*

EVG

LCLS MasterOscillator

476MHz

Linac Master Osc

System is based around the EVent

Generator and EVent Receiver

FIDO PDURaw 360 Hz LCLS Timeslot Trigger

TRD Tx

TRD Rx

TRD Rx

119Mhz + FID

Sq Wave on Coax 119Mhz + FIDSq Wave on Coax

Fiber Cable

TO:- Cav BPM- MPS BLM- MPS PIC- BCS

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 201128

LCLS Timing/Event System Architecture

DEV

~ Linac main drive line

Sync/Div

VMTG

119 MHz 360 Hz

IRQTimeslot

LCLSevents

EVR

CPU

TTL-NIMconvert.

DigitizerLLRFBPMsToroidsCamerasWire ScannerKlystrons

TTL

FAN

CPU

EPICS Network

Fiber distribution

*

*

EVG

LCLS MasterOscillator

476MHz

Linac Master Osc

FIDO PDURaw 360 Hz LCLS Timeslot Trigger

119Mhz + FID

Sq Wave on Coax

Fiber Cable

TRD Tx

TRD Rx

TRD Rx

119Mhz + FIDSq Wave on Coax

TO:- Cav BPM- MPS BLM- MPS PIC- BCS

360Hz IRQ (Fiducial)

60Hz Timeslot 1 Marker

System Upgraded:

Removed Dependency on old MPS using new HW (VMTG) and SW

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 201129

The Event System

Based on Commercial Hardware (MicroResearch Finland)Which was based on a design from ANL-APS Timing System

Designed Around Xilinx Virtex-II FPGAsUses FPGA’s internal High-Speed 2.38 Gb/s Serial xcvr, which connects to a fiber optical transceiverFPGA logic implements all of the timing functionsUses 119MHz clock from Linac which get multiplied up to by FPGA internal PLL to 2.38GHzEVG sends out 8-bit event code to EVRs along with clock and trigger information over one fiberEVR Receives event code & with its associative memory, generates a trigger with a delay set by digital counters

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 201130

The Event System

EVR

EVG EVR

Data Frame:2 Bytes @ 119MHz

8-bitDistributed Databus /

Data Buffer Bus

119MHz RF Clock

360Hz Fiducial

128-bit PNET Pattern

MPS Signals

From SLC Timing System

HWTriggers

PNETEPICS TSMPS Data

EVR

EVR

Event System Architecture

One EVG Per System

[Note: there can be multiple EVGs in one system, arranged in a daisy-chain fashion with priority encoded data tranmission]

Fiber Fanout(s)

8-bit

Eve

nt C

od

eP

NE

TE

PIC

S T

ime

stam

p

Timing Event Code Byte►Upon RX’ing a 360Hz Fid, the EVG

sends out a stream of serial Data to the EVRs over a fiber link. The serial stream consists of 16-bit words sent every 8.4ns.

►Each word contains an Event Code byte and some trigger setup data

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 201131

Event Generator ►EVG contains a RAM that gets loaded with event codes, based on PNET data. 360Hz Fiducial causes the RAM to get sent out over the Serial fiber link to the EVRs.

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 201132

Event Receiver

►EVR contains another RAM that looks for matches of event codes to its contents. If match, it starts a counter running that generates a trigger

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 201133

EVG Hardware

Event Generator

VME-64x Module: Sits in Master Timing Crate with VME PNET receiver and Master Timing CPU.

Receives 119MHz reference and 360Hz master timing fiducial from SLC timing system.

Receives PNET pattern from SLC system.

Broadcasts timing system data in the form of a high-speed 2.38Gb/s serial data stream to the EVRs over an optical fiber.

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 201134

Event ReceiverComes in two flavors: VME and PMC.

Receives 2.5Gb/s serial datastream from EVR and generates triggers based on values of the event codes.

Also receives and stores PNET timing pattern, EPICS timestamp and other data and stores them in an internal data buffer.

Can output 14 total pulsed-output triggers and several more level-type

Triggers are output via a rear transition module (not shown)

Trigger delay, width, duration and polarity are fully programmable

Trigger signal level format is TTL VME Version has 10ps jitter PMC Version has 25ps jitter

EVR Hardware

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 201135

Fiber Fanout Module

Basic function is to receive one optical signal and generate copies of the same signal for transmission.

1:12 way fanout (1 In 12 Out) Contains Clock-Data Recovery (CDR)

circuitry to re-generate & clean-up the signal

Fits into a VME crate, but ONLY uses power

Uses commercial SFP (Small Form-factor Pulggable) optical xcvrs

SFP units are hot-swappable Front Panel LEDs indicate link health No other diagnostics than this There is one of these modules at each

critical distribution point in the system / if one module fails, it takes timing out for a large group of clients

Fanout Hardware

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 201136

System SW

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 201137

What Happens during one 2.8mS machine interval:

F3

0

Fiducial

B0

Acq Trigger

1023

BeamKly Standby

Record processing (event, interrupt)

Fiducial Event Received

Event Timestamp, pattern records, and BSA ready

Receive pattern for 3 pulses ahead

Hardware Triggers

5000.3

18100

Triggering Event Codes

StartKly Accel

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 201138

LCLS Systems – Master Timing Rack

119MHz Synchronizer Chassis

Located in LI20 RF Hut (Rack LKG-21)

Contains:

• VME CPU

• VME PNET Rx

• EVG

• Master Fanouts

Master Timing Crate

Master FODU

Connects fibers to Long-Haul Trunks for entire machine

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 201139

LCLS Timing System – BPM Client What you’d see in a typical service building (e.g. B105) …

Rx FODU & Fanout CrateBPM Crate w/VME-EVR

Rear of BPM Crate / Showing Trigger Rear

Transition Module

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 201140

LCLS Timing System – Other ClientsToroid Crate w/PMC-EVR

Rear of Toroid Crate / Showing Trigger Rear

Transition Module

Profile Monitor Crate w/ (4)

CPUs & PMC-EVRs

MCOR Magnet Crate

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 201141

Performance

9 ps rms jitter

EVR jitter w.r.t. fiducial

The Event System Trigger Jitter was measured using an Agilent Infinium 54845A Digital Oscilloscope in Jitter Histogram Mode / data was collected for 30 minutes

The EVR output (shown) was measured against the system input trigger (360Hz fiducial)

The Actual jitter performance is much better after subtracting off the intrinsic jitter of the scope:

Actual Jitter:JITTERsystem = [ (JITTERsys_meas)2 – (JITTERscope)2 ]1/2

Event System Jitter:JITTEREVR =[ (9.7472ps)2 – (6.3717ps)2 ]1/2

= 7.3763 ps

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 201142

CommentsCurrent LCLS Timing System has growing pains (integration w/old system, SW, HW)Use of commercial HW doesn’t quite fit our needs / having to modify EVG & EVRProblems Seen Thus Far:

“Trigger Storms”: Due to LCLS Master Osc unlocking / Fix: New MO / De-Couple LCLS Timing Sys from it (connect direct to MDL) Dead Links: Fibers / Xcvrs / Still Troubleshooting

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 201143

LCLS Timing – Future DirectionsMPG Replacement:

Planning on Moving the EVG Master Crate to MCC and Replacing the MPG Micro With it COMPLETED Summer 2011

Additional Diagnostics:Designed & Built New Syncer Chassis with added diagnostics: Fiduciual Loss, Rate Monitors, RF Input Power, Clock Lock Detect, Clock Rate Monitors, etc.

Status: Chassis Installed / Need to wire up diagnostic readout HW and finish the SW

Would like to re-design the Fanout modules to take advantage to the built-in diagnostics (Tx & Rx optical power, temp, voltage, etc.) of the Fiber SFP modules

Additional Timing System Upgrades:FIDO Replacement / New Fid Generation Scheme (future…)Linac Sector 0 Timing Upgrade (future…)

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 201144

LCLS Timing – Future DirectionsEventual Goal:

Completely replace SLC Timing System in the Linac while retaining all of the legacy functionality and providing new features Develop a timing system that is expandable & adaptable to new machines (e.g. PEP-X) Challenges:

Temperature-induced phase drifts Handling PEP Injection Damping Ring Timing

Coming Soon….LCLS-II (First Light 2019)

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 201145

LCLS-II:Uses middle 1/3rd of Linac

-New Injector- Re-purpose PEP-II Inj Line- 2 Undulators:

- Hard X-Ray (2-13KeV)-Soft X-Ray (0.25-2KeV)

First Light: Early 2018

John Dusatko

Introduction to the LCLS Timing System jedu@slac.stanford.edu

December 7, 201146

End of Talk