Overview of timing systems (TS) Joze Dedic (joze.dedic@cosylab.com)

… Head of Hardware

t h e b e s t p e o p l e m a k e c o s y l a b

ESS, Lund, 19 Apr., 2011

This presentation

  what is the role of the timing system – TS   functions/purpose   TS-clients   concepts

  three dimensions of TSs   TS domain knowledge   build HW (i.e. transfer layer)   customize = build accelerator-specific RT application on top of

existing TS transfer-layer

  past/present projects & experiences

  how to proceed to TS development

Cosylab 2011

Different machines, different TSs

Cosylab 2011

CERN

SNS

ESRF, ELETTRA…

GSI / FAIR MedAustron

UNDERSTANDING ITS ROLE IS OF CRUCIAL IMPORTANCE.

Timing system is a service.

Cosylab 2011

service  reqs.

services

Interconnection of systems

Cosylab 2011

...cooling  system magnet  power  supplies

magnets

Device  n...Device  2

Device  1

Control  System

BLMs

BPMsRF  amplifiers

real  time  actions:configure  real-­‐time  response,

time  stamping...

machine  protection;  operation-­‐mode  dependent  

status  outputs,  post  mortem...

control  functionality;  configuration  settings,  status  

readout...

...

electron

ic  re

qs.

equipm

ent/mechanics  re

qs.

physics  req

uiremen

ts

control  req

s.

...

DB

MPS

Timing  System;master  dispatcher

Timing  System  event  receiver

Timing  System;master  orchestrating  

hard-­‐Real-­‐Time  application

Application  specific,  real-­‐time  response

...

Timing  System  Transfer  layer

Timing  System  service

WHAT IS THE ROLE OF TS?

Cosylab 2011

Orchestrator

  accelerator = n x 100’s devices that must   perform pre-agreed actions at the right moment (beam: ~c)   respond to real-time status of the machine

  some operational parameters are known just prior taking actions

  technical challenges   execution: must be HARD Real-Time   simultaneousness: us, ns, ps, fs… (depends on type)   but: 100m fiber ~ .5us

Cosylab 2011

Understanding TS reqs. = understanding TS clients

  To understand what is needed from TS, we need to understand, which TS clients need to fulfill which machine operation needs.

  gather a list of TS clients and

  figure out how they work (i.e. interact with others devices & rest of the machine)

  coarsely 3 types of devices that require TS services

Cosylab 2011

TS clients – trigger only (1/3)

Cosylab 2011

TS e.g. Power Supply

SW/CS

trigger

settings

crate

TS

SW/CS

crate

e.g. ADC

trigger = do action here

TS

SW/CS

crate

e.g. logger

> ~10 us

ps…ns…

ps…ns…

TS clients – trigger + real-time data (2/3)   requiring RT data

  delivered just prior action (e.g. machine settings)

  normally, after reception of the trigger (+data), they do “one-off“ action

Cosylab 2011

TS e.g. Power Supply

SW/CS

trigger

settings

crate

TS

SW/CS

crate

e.g. ADC TS

SW/CS

crate

e.g. logger

RT data

e.g. automatic time stamping

e.g. time synchronization

e.g. adjusting current with machine operation

TS clients – trigger + real-time data + clock sync. (3/3)   requiring RT data   requiring trigger   after reception of the trigger do series of (complex) actions,

which have to be tightly linked with the operation of the rest of the machine (”breathing” with master clock)

TS

SW/CS crate

e.g. beam

chopper FPGA!

SNS beam chopper example: resolution is 1/64 of the accumulator ring frequency (~15ns) … which is function of energy

trigger data

clk

multiple actions carried out synchronously

(using reconstructed clock that matches

master oscillator)

Even more demanding devices

  in some cases RF amplifiers for RF cavities require fs clock synchronization   clock only   technically demanding = expensive

  …different approaches exist for meeting this demands   e.g. FAIR; TS (WR) + BuTiS

  … normally this isn’t part of the TS

  clock distribution system vs. timing system   clock distr. system

  clock only (no data, no triggers)   fs

  timing system   clock + triggers + data   ps

Cosylab 2011

TS transfer layer concepts

  keep the traffic over the network fully deterministic (e.g. MRF)   if you know when you send it out, you know when you’ll receive

it, process it, & make action   pros:

  simple, raw, robust (one way), ~15ps jitter   cons:

  tricky automatic delay compensation   slave to master communication complicates things

  agree on absolute time using mechanisms like PTP (e.g. WR)   figure out the time of travel for the packet (down & up)   determine local-receiver time, send out request for actions

taking time at specified time in the future   pros:

  multiple transmitters + automatic delay compensation   cons:

  no implementation available /tested/in-operation… yet   inherently slower master to slave response

Cosylab 2011

A

B

WR

  design goals   Scalability

  Up to 2000 nodes.   Range

  10 km fiber links.   Accuracy and precision

  1 ns time synchronization accuracy, 20 ps jitter.

  Ethernet compliance   Eth. standard ++   synchronous Eth.   PTP; special switches!

Cosylab 2011

  Darmstadt, April, demo   node + switch (V2) demonstarted

  plans   WR project “end” – end of 2011   Full system commercially available

mid-2012   start of CERN (partial, not LHC)

commissioning end of 2012   white-box for sys. integrators

Javier Serrano, GSI, Darmstadt, April 2011

http://www.ohwr.org/attachments/545/oh_darmstadt_apr2011.pdf

Synchronous vs. asynchronous

  asynchronous = more than one! clock   no matter how expensive, clocks jitter & drift

  synchronous = one! clock

  however, partial (=good enough) synchronization can be done by synchronizing clocks often enough

  be aware of multiple (master) clock sources

Cosylab 2011

receiver 1, oscillator 1

incoming trigger do action after 15…

do action after 15…

mismatch of actions on two different receivers receiver 2,

oscillator 2

receiver 1, reconstructed clk

incoming trigger

receiver 2, reconstructed clk

actions on two different receivers match!

Synchronize (or swim…)

  timing and synchronization system from 1917   twin Vickers machine guns, synchronized to fire at 500 rounds/min

between the blades of the propeller rotating in front of them   In the event of a timing error, a few hits on the wooden blades were

sufficient for the plane’s own propeller to be shot away.

Cosylab 2011

Sopwith Camel

master oscillator

synchronously operating device

THREE DIMENSIONS OF TS

Cosylab 2011

Let’s talk dimensions

Cosylab 2011

 

term

inol

ogy

 

timin

g ev

ent,

trans

fer r

ate,

 

cloc

k, ti

me,

reso

lutio

n,

jitte

r, ac

cura

cy,

 

time-

stam

ping

, res

pons

e ra

te,

 

dela

y pr

opag

atio

n co

mpe

nsat

ion…

 

even

t, da

ta a

nd p

aylo

ad

 

dela

y co

mpe

nsat

ion

stra

tegy

 

PTP

(IE

EE

1588

), eq

ual

fiber

leng

th, .

..  

auto

mat

ic d

elay

co

mpe

nsat

ion

 

data

dis

tribu

tion

 

prot

ocol

and

prio

ritie

s  

RT

data

bus

timing domain knowledge

huh?

timing master

GPS

clock (=signal)

output

RF

clock (=signal)

splitter (1 to n)splitter (1 to n)

receiver m

receiver n

splitter (1 to n)Resolution

Timetime (=data)

Clock

Jitter

10010011

10010011

Timing event

10010011

10010011

10010011

timeline

Event transmission rate

output

10010011

Response to event

Propagation delay

timeline

timeline

the two outputs should

change at the very same

moment (at mark o), however

they differ due to accuracy

and precision

Accuracy

Precision

time of actual output

 

Cosylab, Control System Laboratory

Teslova ulica 30

SI-1000 Ljubljana

Slovenia Accuracy  (thesaurus:  The  quality  of  being  near  to  the  true  value)  is  the  mean  of  the  time  error  between  

the  time  under  test  and  a  perfect  reference  time,  over  an  ensemble  of  measurements.

Precision  (thesaurus:  The  quality  of  being  reproducible  in  amount  or  performance)  is  a  measure  of  the  

deviation  of  the  time  error  between  the  time  under  test  and  a  perfect  reference  time  from  the  mean  

time  error.Jitter  is  unwanted  variation    of  a  periodic  signal's  property  (e.g.  period).

Latency  is  time  delay  between  the  moment  something  is  triggered  on  the  source's  end  and  the  moment  it  is  detected  on  the  receiving  end..

Signal  propagation  delay  is  the  time  that  is  required  for  a  signal  to  travel  along  the  transmission  layer.  For  accelerator  timing  systems  this  cannot  be  

neglected;    e.g.  it  takes  ~1 s  to  transmit  a  pulse  over  200m  of  fiber  (speed  of  signal  delivery  in  optical  fiber  is  ~c/1.5).  If  required,  timing  system  design  

enables  compensation  of  different  propagation  delays  (due  to  different  cable  lengths).

Resolution  is  defined  by  a  minimum  discrete  step  in  which  a)  outputs  can  be  set  or  b)  inputs  can  be  time-­‐stamped.  I.e.,  it  is  the  granularity  by  which  

actions  can  be  made  or  stimuli  perceived.

Cosylab timing team cheat-sheet; for further explanation contact - joze.dedic@cosylab.com

2009-12-07

inputResolution

00

0?

11

11

Time-stamping

the process of

determining time when

signal changed (with the

perception of local time)

Accuracy

Timing event is a sequence of bits (corresponding to event encoding and, if applicable, also to event

payload), sent over the real-time dedicated timing system network. It originates from master timing generator

and must be decoded in hard real-time by all timing receiver components when received.

Let’s talk dimensions

Cosylab 2011

timing domain knowledge

  …require dedicated FPGA + VHDL timing-receiver code

  special devices, tightly integrated in timing system…

  SW drivers   add IOs for HW triggers (dummy devices)

  basic FPGA firmware code gets extended   PCB layout; board design becomes complex

and costly   …VME, PXI, etc.

  target platform…   FPGA firmware code (initially not complex)

  prototype (Xilinx dev board for <$500)

  8b/10b encoding, fiber link sync.   FPGA + SFP

Let’s talk dimensions

Cosylab 2011

timing domain knowledge

integration know-how

  tests, documentation, support, training…   (straightforward) implementation   slaves; RT part; FPGA,SW part; integrate

with “simple” devices, tightly embed with complex devices

  master; RT part; FPGA, SW part   configure hardware (access to the

implementation source code, FPGA expert, low-level SW expert…)

  virtual accelerators, execution slots, deliver user data

  sync with (patient) breathing, linac grid…   sync with mains (but… not too fast for

choppers, etc…)   machine physics, accelerator/machine

specific requirements

timing domain knowledge

integration know-how

Let’s talk dimensions

Cosylab 2011

Three dimensions of TS

Cosylab 2011

timing domain

knowledge

integration know-how

“hardware”

Transfer layer (WR, MRF, etc.) is only part of the solution

Cosylab 2011

  orchestrating the behavior of complete machine, knowing every possible mode of operation, RT data transfer (hard RT logic)

machine specific logic

and TS integration

  delivering globally synchronized clock and data (in hard RT);   MRF, WR, etc…

  supplying TS clients in hard RT with clock, triggers, and data to fulfill machine mission

TS customization in one slide

  figure out what all TS clients (devices) require

  figure out devices’ operational concepts (=mapping to physics law of the machine)

  determine physics-law requirements on timing precision

  implement the above (transfer layer + machine specific behavior) to be run in hard RT

  …even in one sentence: inform: the right devices, in the right moment, with the right information

Cosylab 2011

EXPERIENCES

Cosylab 2011

ORNL SNS Spallation neutron source Timing master renovation

Cosylab 2011

SNS

  in the period 2008-2011 we cooperated on SNS timing system renovation

  old HW (several VME crates) moved to single Virtex 5 LX-50

  requirements, architecture, development, testing

  we did only the master FPGA (transfer layer remained)

  EventLink (sync with acc RF)   RealTimeDataLink (async)

  effort: ~1.5 MY Cosylab 2011

Fo as function of tuning word T, expressed as offset from T(60Hz)

59.800

59.840

59.880

59.920

59.960

60.000

60.040

60.080

60.120

60.160

60.200

-­‐4,500 -­‐4,000 -­‐3,500 -­‐3,000 -­‐2,500 -­‐2,000 -­‐1,500 -­‐1,000 -­‐500 0 500 1,000 1,500 2,000 2,500 3,000 3,500 4,000 4,500

5x20 MHz

0-crossing@ 5050

0-crossing@ 5050

reset EL counter

reset EL counter

nominally:16,666 ± 500 µs

1/fRF »911-973 ns

0-crossing@ 5050

local 20 MHz counter:Ø  32-bitØ  latch @ line 0-crossingØ  no reset

+

desired output frequency Fo 60 [Hz]adder freqency Fs 100 [MHz]

desired Fo resolution Fores 50 [µHz]

number of bits required N.round 41 [bit]actual Fo resolution Fores.calc 45.5 [µHz]

ideal output-period step ΔTo 12.6 [ns]Fo jitter 1/Fs 10 [ns]

1 x 32, RLinePeriod_reg(50 ns resolution)

1 x 32, RWDDS_tuning_word_reg

preset with 60 Hz value

POWER_LINE_REF_IN

EL_ResetTurnCounter

LineZeroCrossing

overflow detection

ELCLK

distribution CLK_20M

Line frequency counter and the line frequency generator must be derived from the same clock source.

Line frequency counter and the line frequency generator must be derived from the same clock source.

0.033333333 911.2 973.4

Fdesired Tuning word T-T(60) Factual16 x turncounter

16 x turncounter

59.80 1,315,016 -4398 59.80000 293,633 274,87059.83 1,315,749 -3665 59.83334 293,470 274,71759.87 1,316,482 -2932 59.86667 293,306 274,56459.90 1,317,215 -2199 59.90000 293,143 274,41159.93 1,317,948 -1466 59.93334 292,980 274,25959.97 1,318,681 -733 59.96667 292,817 274,10660.00 1,319,414 0 60.00000 292,654 273,95460.03 1,320,147 733 60.03334 292,492 273,80260.07 1,320,880 1466 60.06667 292,330 273,65060.10 1,321,613 2199 60.10000 292,167 273,49860.13 1,322,346 2932 60.13333 292,005 273,34660.17 1,323,079 3665 60.16667 291,844 273,19560.20 1,323,812 4398 60.20000 291,682 273,044

0b1010000 & word(13:0)

Tuning word Factual0b101000000000000000000 1,310,720 59.604640b101000011111111111111 1,327,103 60.34966

LINE_STS_LEDecho POWER_LINE_REF / stretch 50%

CLK_20Mx5

phase measurement register is in EL module

Period MeasureØ   Mains Frequency Ø   TC Reset Frequency

(DDS) 1 x 32, RLineZeroCrossingFreq

1 x 32, RTurnCounterResetFreq MainsTracking

complete timing operation breathes with mains, minding choppers (rotation inertia)

MedAustron MedAustron centre for ion-therapy and research Complete timing system

Cosylab 2011

MedAustron timing

Cosylab 2011

1x MRF EVG

n-x MRF EVR

  REDNET = real-time event and data network

  requirement analysis,   architecture design   development, testing…

  effort: ~2+1 MY

Furthermore

  FAIR; system level integration support in progress

  CERN; collaboration on timing system upgrade, WR

  US medical company, Optivus, starting collaboration

  tight collaboration with MRF; insight into all internals, full R&D support   however, not exclusively linked to MRF and can also use any

other transfer layer

  machine protection system (MPS) as a similar concepts of technology use   n – n hard-real time communication + time synchronization

  …

Cosylab 2011

NEXT STEPS?

Cosylab 2011

Next steps

  figure out what all devices require   define interfaces, specify TS service

  figure out devices’ operational concepts (=mapping to physics law of the machine)

  determine physics/diagnostics/electronics requirements on timing precision (accuracy jitter)

  implement the above (transfer layer + machine specific behavior) to be run in hard RT

  integrate the above with control box   proof of concept for technology choice on transfer layer

Cosylab 2011

FINALLY…

Cosylab 2011

Design a TS with complete accelerator in mind

  TS, MPS, or any other services… are very tightly integrated in the operation of the machine   understanding the role of the service is of crucial importance

  Quick win for the proof of concept (=proof of technology), does not yet automatically imply the design will satisfy later needs from the integration point of view.

  Architecture and design has to be done “smart and future proof”,

being aware of all possible caveats that multiple labs already addressed.

Cosylab 2011

Cosylab 2011 …and now let’s get down into it and make it work

Backup slides

MPS

Cosylab 2011

Machine protection system

  if energy of the beam can damaged the machine   the beam and machine status must be monitored   and mitigation actions must be taken

  concepts similar to TS   in depth understanding of the operation of the complete

machine   real-time & very fast data communication among multiple

spatially distributed nodes   reuse the knowledge of the TS transfer layer

  work in progress together with   SNS, FRIB   proof of concept prototype

  also, next project steps very similar to TS

Cosylab 2011

Slo

w M

PS

(B

eam

Per

mit

/ Run

P

erm

it)

Pos

t-m

orte

m

(Offl

ine

app.

)

Fast

M

PS

Prevent, observe & act, analyze   Prevent starting beam before the machine is ready

  Check that the facility is configured and secured appropriately for the operator-selected experiment

  Check that all relevant equipment is operating and within the desired setpoint ranges for the selected operating mode

  Prevent changing operating mode if the machine is not ready   Check that all involved devices and their settings are ready for

new mode   Observe the machine (systems, devices, sensors…) during run-time

  Stop the machine as soon as something is wrong   Make post-mortem analysis to determine a cause of a trip

  Remove/resolve the cause before you restart the machine

Cosylab 2011

  MPS-relevant devices MPS inputs   beam status (BLM, BPM, Faraday Cup, Wire scanners),

devices’ statuses…   MPS outputs mitigation devices

  Chopper, Extraction Kicker, RFQ, Ion-source, Mechanical shutter…

  decide on mitigation scenario   log everything to be able to reconstruct

what was happening

Cosylab 2011

MPS supervises the machine and switches it off if something goes wrong

  devices need to be designed to work in conjunction with MPS   devices need to be aware what means OK

and what is fault (even if operating mode dependent)

  fail-safe (digital) outputs   post mortem buffer within devices!

MPS Switch

MPS Master

MPS Node

MPS NodeI/O I/O

I/O

MPS Switch

MPS Node

MPS NodeI/O I/O

MPS Switch

MPS Node

MPS NodeI/O I/O

MPS Switch

MPS response time

  device detection time   MPS processing logic + distribution   mitigation device to execute mitigation action

  …mind that beam already in the pipe will hit something   t1+t2+t3

Cosylab 2011

Fast Devices

t1

MPS

t2

Mitigation Devices

t3

Flexible input matrix

  different types of input   beam-abort capable   post mortem diagnostics & warnings

  (dynamically) adjustable   operating mode (energy levels, faraday cup in, etc.)

  requires dynamic masking + dynamic response matrix   step-by-step commissioning

  requires masking (certain sections not yet available)

  however, all inputs:   digital!   fail safe

  this puts complexity to   configuration tools(see above + permissions levels)

Cosylab 2011

Post-mortem analysis - reconstructing the fault

  MPS logs all state changes of connected devices   As log buffers are limited, buffers are overwritten circularly   Logging must be stopped after a fault occurs

  Timeline of events can be reconstructed and first fault can be identified   Logs from all MPS devices can be merged into one, thanks to

fact that:   MPS equips all logs with precise timestamp   Time of all MPS devices is synchronized

  Other devices can also perform their own logging   All such devices should be time synchronized (using Timing

System for time reference) in order that logs from all MPS and non-MPS devices can be merged into the same timeline

Cosylab 2011

MPS A&D + implementation

  learn from SNS experiences / they are upgrading to add:   robust link   reliable operation   standalone (not integrated in “hosts”)   global time-stamping for effective post mortem

  either use TS (not the case for SNS)   or built-in MPS node-sync

Cosylab 2011

Backup slides

tech. details, team info, etc.

Cosylab 2011

Team

  13 people   architecture design, FPGA development, analogue & digital

design (high voltage, low-current, high speed, etc.), product development, board design, prototyping,

  try a lot of people, hire the best   + handful of external experts

  investment into future; keeping pace with advancements   CAD tools; FPGA, PCB   hot development boards (FPGA, DSP, processors…)   ongoing education & training

  ongoing FPGA knowledge improvements   FPGA academy   VHDL best coding practices cooperation with CERN et al.   LEGO FPGA

Cosylab 2011

Cosylab 2011

MA; EVG++

  VAcc (ExecSlots), multiple tables; 256 entries, relative delay   all tables synced to internal time grid (VAcc specific offset)   command / data / XML real time data distro to receiver clients (all

per ES)   asynchronous events   priorities   automatic heart beat generation

Cosylab 2011

ES 3

ES 2

Global Scheduling

ES 0

Δt Event Type Identifier

Event Table

CommandsAsynchronous Timing Events

Heartbeatevent

generator

214

ES 1

Δt Event Type Identifier

Event Table

CommandsAsynchronous Timing Events

3 3

12

5 8

MRF EVG

VA 4

VA 1

VA 0

Δt Event Type Identifier

Event Table

Commands

Asynchronous Timing Events

Seqencer

Internal Interfaces

5 Virtual Accelerators

Seqencer

HeartbeatEvent

generator

Internal Interfaces

PXI crate controller

ReceivercomponentsReceiver

componentsReceiver

components

Internal HW interface (cPCI)

Fast Equipment Interface - FEI (real-time fiber optic link)

Timestamp

Heartbeat offset

MA; Receiver side

  digital/optical signals on the MRF EVR outputs   distribute events to clients (neighbor PXI cards)   trigger clients (neighbor PXI cards)   software IRQ (LV Vis)   time stamping support   distribute reference clock (10 MHz GPS)

Cosylab 2011

MTR  PXI

PXI  -­‐  System  Controller

Device specificFECOS

Component

hard  drive

RAM

MTRFECOS

Component

PXI  backplane

MRF  EVR

Internal HW interface (cPCI)

FlexRIOFlexRIO

AuxiliaryInterface

AI1

4 TTL outputs

4 optic outputsOR

EVR TTL/opticAuxiliary Interface

FlexRIO

PXI Shared Trigger Lines

PXI Shared Trigger Bus

(ETID distribution)

Notifications to MTR FECOS component

MA; EVR++ (flex. outputs)

Cosylab 2011

Event response configuration table*TIMING CONFIG. PTR

(3:0)P/T(1)

CARD(3:0)

STL(1)

STB(1)

IRQ(1)

AI output(3:0)

256 Rows

AI output timing configuration table*W1(16)

Δt1(16)

16 Rows

Δt2(16)

Δt3(16)

Δt4(16)

W2(16)

W3(16)

W4(16)

Auxiliary Interface response generation

(FPGA Logic)

Default W(16)

Default Δt(16)

Pulse/Toggle

AI Output Number

Delay and Width parameters (per ETID)

AI Output 1

AI Output 2

AI Output 3

AI Output 4

WL1(16)

ΔtL1(16)

WL2(16)

ΔtL2(16)

WL3(16)

ΔtL3(16)

WL4(16)

ΔtL4(16)

WL5(16)

ΔtL5(16)

WL6(16)

ΔtL6(16)

WL7(16)

ΔtL7(16)

Star Trigger Line response generation

(FPGA Logic)

Shared Trigger Bus response generation

(FPGA Logic)

PXI PCI Interupt generation

(FPGA Logic)

PXI Star Trigger lines (7)

PXI Shared Trigger Bus (8)

PXI PCI Interface

PXI PCI IRQ

8 ETID

Card Number (indicating Star Trigger Line)4

Response gen. on PXI Star Trigger Lines enabled (for received ETID)

Response gen. on PXI Shared Trigger Bus enabled (for received ETID)

Generation of PCI interrupts is enabled (for received ETID)

ETID

ES GPS TS

TS rel. to CS CID

ETID

8

7

8

4

Star Trigger Line timing configuration table*

Default AI output timing configuration registers*

* All memory tables and registers are accessible to the MTR FECOS component through the PXI PCI interface.

Card Number (indicating Star Trigger Line)4

4 aux outputs

7 start trigger lines

8 backplane trigger bus

PCI IRQ; VIs

delay propagation compensation fully flexible output configuration concurrent response generation

all events can map to any of the 16 different settings: pulse, delay, width, toggle/pulse (per output, per event)

…but still optimized for resource usage

SNS

  in the period 2008-2011 we cooperated on SNS timing system renovation

  old HW (several VME crates) moved to single Virtex 5 LX-50

  requirements, architecture, development, testing

  did not touch transfer layer, only the master

Cosylab 2011

I2C

16x RF

20MHz

EL

VME

EL_EventTableEL_SoftEventFIFOEL_EventConfigTableEL_SuperCycleTable

RTDL_Table

registers I/F

IRQ handler

generic VME I/F

RTDL

master ref. generator

line frequency measurement

DDS frequency generator

GPIO control

EL_ResetTurnCounter

LineZeroCrossing

EL_to_VME_CycleStart

trigger IRQ

EL_to_VME_5050

clear AutoclearMask_reg & clear RequestRTDLtransmit_reg

EL_t

o_G

PIO

_Cyc

leSt

art

rese

t lat

ched

inpu

ts

and

occu

rrenc

e m

onito

ring

RTDL_LatchTime @ cycle start

RTDL_LoadData @ end of cycle

RTDL_StartTransmit

GPI

O_S

tartR

TDLx

mitR

eqev

ent r

eque

st

RTDL_TransmitDone

IRQ_sources(17:0)

EL_H

Wev

entF

IFO

EL out(LVPECL)

RTDL out(LVPECL)

VME P1 VME P2

20MHz

16x RF

20MHz

VMEclk

VMEclk

Crossing clock domainsØ   DS - double sample clock cross-

domain registersØ   EDS - extend and double sample

clock cross-domain signalsØ   UCF must handle this, to avoid

timing errors

Clock

RTDL_TableUpdateDone

trigger IRQ

HW_m

ask_

input

s(5:

0)

DCMs and buffers

USB I/Fnot supported

Fiber optics module I/F + I2C

not supported

Fiber optics CDR I2Cnot supported

RTC SPInot supported

VMEclk

20MHz

16x RF

SPARE_FRONT_PANEL_OUT

MPS_AR_CARRIER_IN

MPS_LATCH_CARRIER_IN

SPARE_FRONT_PANEL_IN(3:0)

SPARE_P2_OUTPUT(7:0)

SPARE_P2_INPUT(7:0)

VME_FAN_FAIL_LOW_IN

POWER_LINE_REF_IN

GPS_PPS_IN

VME_SYSTEM_FAIL_LOW_IN

GPS_PPS_IN

GPS_TRIGGER_OUT

EL_HWeventFIFO

Turn counter

Collecting, masking & encoding the event

Measuring line phase

Super cycle counter

Overwriting RTDL table

Sending out RTDL frames

Time stamp of the next CycleStart

Reading input and setting

output signals

IRQ control

EL HW eventsEvent masking;

HW-signal sources

EL prim clkCLK_EVENT_LINK_LVDS_0

EL bckp clkCLK_EVENT_LINK_LVDS_2

CLK_LOCAL_HI/LO_IN

CLK_20MHZ16xRF selection

RF measure

System monitor

I2C_SCL_BI,I2C_SDA_BI

SiLabs SI570I2C prog. clock

VMEclk

20x5MHz

20x5MHz

VMEclk

DRP memory mapping

reset

20MHzreset

reset

reset

reset

reset

reset

hold reset after

power up

reset

SNS; Event Link (EL)

Cosylab 2011

sniff for RTDL

start transmit

Ø   there is no RTDL start transmit event in RTDL tableØ   RTDL transmit is either triggered through host

register or RTDL send timeoutØ   once triggered, RTDL start transmit event request is

raised to be stuffed to HW event FIFOØ   once it is to be sent out, RTDL start transmit is

requestedØ   when finished, RTDL transmit done event also

comes through HW event FIFO

TurnCounter19 bit, init: 0

All events should start at turn tick, the only exception is cycle start, which should start 1/16 earlier. This can be handled in several ways; one might be that after sending out first event (cycle start), 1/16 of the delay is injected into the chain (making this period effectively 17/16 long) and all of the subsequent periods 16/16...

EL_SuperCycleTable1k x 16

EL encoding & sending

SuperCycleCounter:Ø   10 bitØ   initialize to 0x000Ø   count; 0-599 and restart

EL sourcespriority

event = 0xFF if nothing there

FSM

RTDL

1 x 32, RSuperCycleCounter_reg

event(7:0)

EL_EventTable16k x 8

EL_HWeventFIFO16 x 8

EL_SoftEventFIFO513 x 8

1

2

3

EL_EventConfigTable512 x 32

1 x 32, RWSoftwareMask_reg

1 x 32, RWAutoclearMask_reg

GPIO control

event configuration(31:0)

supercycle info(11:0)

HW inputs(5:0)

AutoclearMask_reg(5:0)

SoftwareMask_reg(7:0)&

==

CLK_16xRF

EL_ResetTurnCounter

GPS_TRIGGER_OUT

EL out(LVPECL)

CLK distribution

master refgen

1 x 32, R (16xRF resolution)LinePhase_reg

latch @ 0-crossingLineZeroCrossing

turn counter decoderØ   cycle start, @0Ø   5050Ø   EndOfCycle_reg, def 5150 Ø   DelayRTDLsendTimeout_reg, def 500

RTDL send timeout

VME

increment

@ 5050

if(( ))

event_out <= event;else

else event_out <= 0x00;counter value

HW_mask_inputs

EL_to_VME_CycleStart

trigger IRQ

RTDL_LoadData @ end of cycle

RTDL start-transmit logic

1 x 32, RWRequestRTDLtransmit_reg RTDL_Start

Transmit

GPIO_StartRTDL

xmitReq

counter value

GPIO control

EL_to_GPIO_CycleStartreset latched inputs and occurrence monitoring

*auto cleared in VME, init as 0

EL_to_VME_5050

clear AutoclearMask_reg & clear

RequestRTDLtransmit_reg

*auto cleared in VME

Provide a 10 to 100 us pulse corresponding to cycle start.

RTDL_LatchTime @ cycle start

Ø   MPS Auto ResetØ   MPS latchØ   Front panel spare 3Ø   Front panel spare 2Ø   Front panel spare 1Ø   Front panel spare 0

event configuration(31:0) Ø   Host Alive

Ø   Beam OnØ   Diagnostic FastØ   Diagnostic SlowØ   Diagnostic DemandØ   Spare

BEAM_ON_LED_OUT

0x00 means no event

if turn counter > 0x3FFF

event table reads 0

init as 0

if clocked with 32xRF, reset of the turn counter must be done when LSB is 1 (i.e. not to infer EL data polarity shift with respect to the RF)if clocked with 16xRF, it does not matter (think in the frame of reading 32 bit LUTs per turn, which have to be fully sent out)

CLK distribution CLK_16xRF

18:4 – whole turns  £  327673:0 – sub-turn counter £  15

Ø   supercycle counter is incremented @5050(request RTDL transmit reg is also cleared)

Ø   RTDL data is loaded @ EndOfCycle_reg; 5150~5650Ø   RTDL send timeout happens DelayRTDLsendTimeout_reg

time later 200~1000, default 500

signal out @ cycle start

there must be a constant ‘propagation’ delay from the turn counter changeover to the moment appropriate event starts coming out (it’s length is not important as the phase tracking mechanism handles this implicitly)

Another very neat mechanism is to prepare LUTs for all events for the entire RF period (all 32 bits) and just sent that out (parity bit takes care all the phases match and the output stream could consists simply of concatenating appropriate events, no encoding would have to be done this way). 1/16 of the shift would be implemented as all the events except first one to be shifted by 1/16. Event 0 would also be encoded. All events would start with 1 and end with 0!

ODDR, SAME_EDGE

1 x 32, RWDelayRTDLsendTimeout_reg

1 x 32, RWEndOfCycle_reg

CycleStart event jostle must be prohibited; There are at least two ways to achieve that; a) to prohibit loading new events after 16384 (2^14) or b) to have turn-counter to event-out propagation larger or equal than one turn and prohibit loading new events when reset is detected.

implemented in VME, signal passed to EL

(10101010101010101010101010101010)

SNS; catching mains

Cosylab 2011

Fo as function of tuning word T, expressed as offset from T(60Hz)

59.800

59.840

59.880

59.920

59.960

60.000

60.040

60.080

60.120

60.160

60.200

-­‐4,500 -­‐4,000 -­‐3,500 -­‐3,000 -­‐2,500 -­‐2,000 -­‐1,500 -­‐1,000 -­‐500 0 500 1,000 1,500 2,000 2,500 3,000 3,500 4,000 4,500

5x20 MHz

0-crossing@ 5050

0-crossing@ 5050

reset EL counter

reset EL counter

nominally:16,666 ± 500 µs

1/fRF »911-973 ns

0-crossing@ 5050

local 20 MHz counter:Ø  32-bitØ  latch @ line 0-crossingØ  no reset

+

desired output frequency Fo 60 [Hz]adder freqency Fs 100 [MHz]

desired Fo resolution Fores 50 [µHz]

number of bits required N.round 41 [bit]actual Fo resolution Fores.calc 45.5 [µHz]

ideal output-period step ΔTo 12.6 [ns]Fo jitter 1/Fs 10 [ns]

1 x 32, RLinePeriod_reg(50 ns resolution)

1 x 32, RWDDS_tuning_word_reg

preset with 60 Hz value

POWER_LINE_REF_IN

EL_ResetTurnCounter

LineZeroCrossing

overflow detection

ELCLK

distribution CLK_20M

Line frequency counter and the line frequency generator must be derived from the same clock source.

Line frequency counter and the line frequency generator must be derived from the same clock source.

0.033333333 911.2 973.4

Fdesired Tuning word T-T(60) Factual16 x turncounter

16 x turncounter

59.80 1,315,016 -4398 59.80000 293,633 274,87059.83 1,315,749 -3665 59.83334 293,470 274,71759.87 1,316,482 -2932 59.86667 293,306 274,56459.90 1,317,215 -2199 59.90000 293,143 274,41159.93 1,317,948 -1466 59.93334 292,980 274,25959.97 1,318,681 -733 59.96667 292,817 274,10660.00 1,319,414 0 60.00000 292,654 273,95460.03 1,320,147 733 60.03334 292,492 273,80260.07 1,320,880 1466 60.06667 292,330 273,65060.10 1,321,613 2199 60.10000 292,167 273,49860.13 1,322,346 2932 60.13333 292,005 273,34660.17 1,323,079 3665 60.16667 291,844 273,19560.20 1,323,812 4398 60.20000 291,682 273,044

0b1010000 & word(13:0)

Tuning word Factual0b101000000000000000000 1,310,720 59.604640b101000011111111111111 1,327,103 60.34966

LINE_STS_LEDecho POWER_LINE_REF / stretch 50%

CLK_20Mx5

phase measurement register is in EL module

Period MeasureØ   Mains Frequency Ø   TC Reset Frequency

(DDS) 1 x 32, RLineZeroCrossingFreq

1 x 32, RTurnCounterResetFreq MainsTracking

complete timing operation breathes with mains and waits for choppers (rotation inertia)

synthesizing ~60 Hz (50 µHz) to follow mains + PID regulator

Positioning, wrt timing systems

  we are focused on the system-integration level   …and (unlike our partners) we are not focused on the timing

transfer layer   i.e. hardware with sole purpose of real-time clock and data

distribution   we don’t have any preferences for transfer layer

  although, we can make suggestion

  we understand very well how the transfer layer works

  and we know how to add accelerator specific logic   the logic we add is still hard real-time

  FPGA   …but we can be soft, also

  providing drivers or application level SW

  partners with MRF, for even better support

Cosylab 2011

Big picture & services?

  List ALL needed services   Define systems without duplicating services   Understand connections between systems   Define interfaces between systems

Cosylab 2011

Control  System

Timing  System

MachineProtection  System

Service

Service

Service

IOCs

Devices

Device  support

GUI

Service

Service

Service

?