Synchronous Device Interface at NSLS-II Yuke Tian Control Group, NSLS-II, BNL (May 1, 2009 EPICS...

Synchronous Device Interface at NSLS-II

Yuke Tian Control Group, NSLS-II, BNL

(May 1, 2009 EPICS Collaboration Meeting, Vancouver)

Outline

1. Synchronous, deterministic and reliable data distribution

2. Synchronous Device Interface

3. Synchronous Device Interface Extension

4. MicroTCA

5. Summary

EPICS Collaboration Meeting, Vancouver 2009

Synchronous, deterministic and reliable data distribution

Requests to control systems from accelerator physics

Distribute data in accelerator complex with:

● Synchronism: Data acquisition and distribution systems run off a single master oscillator.

● Deterministic : Data distribution with a fixed delay.

● High reliability: Data distribution with redundancy, no single point of failure.

NSLS-II Accelerator Complex


• Goal: run all circuitry in the accelerator off a single master oscillator

• Advantages

Accelerator are driven by many subsystems (RF, magnet/PS, diagnostics etc). Global synchronism offers repeatable, stable behavior. For example, fast orbit feedback.

Synchronous stimulus-response measurement (such as response matrix, tune measurement etc) are easy in a fully synchronous system.

Global synchronism offers global timestamp.

There is no added jitter from crude local clock.

Synchronous, deterministic and reliable data distribution: synchronism


Synchronous, deterministic and reliable data distribution: synchronism


Clock Domains In Accelerators

Synchronous Device Interface: challenges for FOFB


Synchronous Device Interface: answers for NSLS-II FOFB


Synchronous Device Interface: cell controller


• Synchronism in NSLS-II

125MHz clock (in sync with 500 RF master clock) and fiducia will be distributed.

We already have a very fine, low jitter clock. We can synchronize all circuit with this RF clock. We can timestamp every event with 2ns resolution. All the system are talking with the same clock.

We will push the synchronism into power supply control system. Then, all subsystem (BPMs, cell controller, PS control etc) will be fully synchronous with RF clock. This will provide a lot of benefits for accelerator physics.

Synchronous Device Interface: synchronism


Synchronous Device Interface: deterministic

Data per BPM Data per cell Total data to be distributed

GTP 1.25Gbps 8-bit datawidth

GTP 2.5Gbps 16-bit datawidth

10 bytes(X:4; Y: 4; Status:2)

120 Byte(8 BPM4 XBPM)

120 * 30 = 3.6Kbyte

3.6Kbye/1Gbps= 28.8 us

FPGA: 125MHZ

3.6Kbye/2Gbps=14.4 us

FPGA: 125MHZ

Latency calculations:



GTP interface test: 1.25Gbps 8-bit datawidth



GTP interface test: 2.5Gbps 16-bit datawidth


Synchronous Device Interface: reliability

• Synchronous Device Interface reliability:

There are two 2.5Gbps links running in opposite directions. One is the main link and the other is backup link.

When a link (or a pair of links) between two cell controllers is broken, the backup link is used to keep connectivity between any pair of cell controllers in the ring. The system has localized the broken link and re-configured automatically to maintain connectivity using backup link.

The BPM data distribution is abstracted from upper application levels, and therefore corrector setting calculations are not altered when the system falls into the fault recovery operation mode. The fault is identified by the system, and error flags are delivered to the operator level.


Synchronous Device Interface: extension


Synchronous Device Interface: Extension to PS Control


MicroTCA


In 1979, Motorola was developing their new Motorola 68000 CPU and one of their engineers, Jack Kister, decided to set about creating a standardized bus system for 68000-based systems, which he called VERSAbus. http://en.wikipedia.org/wiki/VMEbus

Request from distributed control/computing system(looks familiar ?)

MicroTCA



ANSI/VITA 1.180 MB/s

ANSI/VITA 1.5320+ MB/s VITA 41

3 to 30 GB/sVME with

Switch Fabric on P0

VITA 46VME and/or

Switch Fabrics3U & 6U

IEEE101440 MB/s

VME…VME64…VME64x…VME2eSST…VXS…

1981 1994 1997 2003 2005

VME…VME64…VME64x…VME2eSST…VXS…

1981 1994 1997 2003 2005

Ethernet…10BASE5…10BASE2…10BASE-T…100BASE-T…1000BASE-X…10GBASE-X…

1982 1983 1984 1990 1994 1998 2002

Ethernet…10BASE5…10BASE2…10BASE-T…100BASE-T…1000BASE-X…10GBASE-X…

1982 1983 1984 1990 1994 1998 2002

MicroTCA


VME backplane evolution

MicroTCA

How about a cleaner solution ?

MicroTCA


Kontron OM9140

Emerson OM5080

Schroff MicroTCA crate Elma Blu!Smart

MicroTCA


Technology Bus Bandwidth CPU OS Cost Vendors

VME Parallel+ Serial

VME64: 80MB/s PowerPC LinuxVxWorksRTEMS

Crate: $3-7KCPU: $2-5K

CPU: EmersonCrates: Wiener, Dawn, Elma, Schroff, Rittal

MicroTCA Serials (GigE.PCIe,

RapidIO)

With PCIex41GB/s

Intel multi corePowerPC

LinuxWindows

WindRiver?

$5K (crate, MCH,

and CPU board)

CPU: Emerson, Kontron, GE Fanuc, moreCrates: Dawn, Elma, Schroff, Rittal

Compare of VME and MicroTCA

Challenges of MicroTCA for accelerator control community: 1) IO modules2) Software development.

MicroTCA

Summary

• SDI status:

LBNL group (Larry Doolittle group) is leading the SDI core design. The communication through fiber links are done.

BNL group is designing the SDI extension at PS control system.

Get BPM data into cell controller by using Libera grouping features is under development.

We are evaluating MicroTCA IOC. It will be test on EPICS/Linux. Then we will try to migrate to EPICS/RTEMS if necessary.


Summary

• SDI features:

Use the same clock to synchronously distribute data around nodes;

Two directions transmitter and receiver to make it redundant and single node/connection fail safe;

Serial chain link requires minimum wiring;

Protocol independent of carrier/link speed: Rocket IO 2.5Gbps fiber link; 100Mbps Ethernet link;

• SDI will be an open source design.


Acknowledgement

Larry Doolittle (LBNL)Bob Dalesio (BNL)

Carlos Serrano (LBNL)Joseph Mead (BNL)

Thank you !


Synchronous Device Interface at NSLS-II Yuke Tian Control Group, NSLS-II, BNL (May 1, 2009 EPICS...

Documents

Transcript of Synchronous Device Interface at NSLS-II Yuke Tian Control Group, NSLS-II, BNL (May 1, 2009 EPICS...