SLAC

CONTROL SYSTEMFRONT-END

HARDWARE UPGRADEEric J. SiskindMarch 7, 2002

Stanford Linear Accelerator Center

E. Siskind, March 7, 2002 2

SLAC

Project Goals

• Replace Multibus-I micro hardware with commercial-off-the-shelf personal computers.

• Replace special purpose accelerator networks (SLCnet, KISnet, PNET) with a single commercial-off-the-shelf TCP/IP network such as switched gigabit Ethernet.

• Develop personal computer CAMAC interface to replace MBCD/MBCD-II.

E. Siskind, March 7, 2002 3

SLAC

Project Impetus

• Multibus-I difficult to upgrade, maintain, and support.

• Not able to take advantage of Moore’s Law increases in CPU speed, memory size, and network bandwidth for ~10 years.

• Two orders of magnitude increase in that time:– CPU clock frequency from ~20 MHz to ~2 GHz;

– Memory size from ~10 megabytes to ~1 gigabyte;

– Network bandwidth from 1-10 to 100-1000 megabits/second.

E. Siskind, March 7, 2002 4

SLAC Front-End Real-Time TCP/IP Networking

• Network must deliver 120/360Hz real-time micro-micro traffic in preference to slower Alpha-micro (or generic front-end client to micro) packets. There is currently no expertise in meeting this challenge on the SLAC site.

• Single network is preferable for reliability and cost of purchase/maintenance, but use of a separate real-time network at an early stage is possible.

E. Siskind, March 7, 2002 5

SLAC

New CAMAC Features

• CAMAC interface differs from MBCD/MBCD-II in four major ways:– Two board design to support remote I/O;– Real-time preemptive hardware with recovery;– Maximizes CAMAC utilization when user

software environment in PC is multi-threaded;– Large (i. e. comparable to existing micro) user

programming environment within interface.

E. Siskind, March 7, 2002 6

SLAC

Hardware Architecture

• Hardware consists of two circuit boards connected by up to 10 km of single mode fiber optics:– PCIL (“PCI Link”) board plugs into PCI option

slot in PC;– FECC (“Front-End CAMAC Controller”) is

double-width CAMAC module.

• PC moves from micro’s current location to MCC computer room, leaving behind only FECC.

E. Siskind, March 7, 2002 7

SLAC

PCIL Features

• Basically an intelligent special-purpose Network Interface Card.

• Processor with DMA PCI block transfer master and DMA fiber optic link transmitter/receiver.

• Makes no assumptions about nature of PC’s real-time operating system.

• Semi-static assembly language PROM code with downloaded updates.

• First generation hardware doubles as Alpha SLCnet interface.

E. Siskind, March 7, 2002 8

SLAC

PCIL Code Functions

• Supports PC FECC message passing.• Supports FECC remote read/write access to PC

memory with optional PC interrupt.• Performs function-specific scatter/gather DMA

to/from PC memory for all classes of operations, avoiding unnecessary buffer copying by PC CPU.

• Forwards trigger pattern from PCI registers to FECC (only in 1st generation; pattern forwarding in hardware in 2nd generation).

E. Siskind, March 7, 2002 9

SLAC

FECC Features

• Logical successor to MBCD/MBCD-II.• Accesses host memory via point-to-point fiber

optic link instead of parallel I/O bus.• Gets power from CAMAC backplane and timing

triggers from PDU via auxiliary backplane.• 1st generation supports MBCD-II’s four strings of

SLAC serial CAMAC crate controllers (5 MHz, bit serial, half duplex).

• 2nd generation adds support for four strings of IEEE standard crate controllers (5 MHz, byte serial, full duplex) and Bitbus.

E. Siskind, March 7, 2002 10

SLAC

Rapid Micro Replacement

• All PCs in MCC computer room have only three connections to outside world:– Front-end TCP/IP network interface;

– MPG IDOM connection to hard reset button (replaces PNET reset);

– Fiber optic connection to FECC.

• Only FECC connection is specific to a particular function; i. e. only attachment to PR02’s FECC converts an arbitrary PC on front-end network into PR02.

E. Siskind, March 7, 2002 11

SLAC

Micro Replacement II

• MCC computer room patch panel has fiber optic links to all PCs’ PCILs and all regions’ FECCs.

• Switching to a spare PC requires changing TCP/IP address, updating IDOM bit number, and moving fiber optic patch cord.

• First two accomplished via logical physical micro association table.

E. Siskind, March 7, 2002 12

SLAC

Real-Time Constraints• All accelerator operations based on 360 Hz timing;

2.78 milliseconds/pulse.• Many CAMAC operations must be performed on

each pulse, or on a particular pulse (e.g. F(19) PPYY broadcasts, BPM data acquisition).

• Real-time access to CAMAC for such time-critical usages is currently maintained by limiting all MBCD requests for groups (“packages”) of CAMAC cycles from non-time-critical micro jobs to one millisecond. Only one such micro job may have a single package outstanding in the MBCD at any given time.

E. Siskind, March 7, 2002 13

SLAC PCIL-FECC Real-Time CAMAC Access Scheme

• All time-critical operations either employ hardware that is distinct from that used for non-time-critical operations, or else can interrupt non-time-critical operations on shared hardware.

• All software executing in PCIL or FECC runs under a real-time executive with interrupt-driven prioritized preemptive scheduling.

E. Siskind, March 7, 2002 14

SLAC

Real-Time Latency

• Shared PCI hardware released in three PCI clock ticks (90/45 ns on 33/66 MHz bus).

• Shared 2nd generation fiber optic link accessed on next 4.9 microsecond cell (plus second cell if pattern forwarding required).

• Shared CAMAC hardware released in one CAMAC cycle (~10/~4 microseconds on SLAC/IEEE standard cable) unless locked.

• Real-time executive switches processes in 3-5 microseconds (plus similar interrupt latency if context switch already in progress).

E. Siskind, March 7, 2002 15

SLAC CAMAC Interrupt Recovery & Control

• If a CAMAC block transfer is interrupted, hardware can repeat the previous write operation (address pointer load?) with updated write data before resuming the transfer.

• Hardware can prevent an interrupt between CAMAC cycles transferring two 16-bit halves of a single 32-bit word.

• Hardware can prevent an interrupt of a particular block transfer entirely (needed because of a current PIOP feature).

E. Siskind, March 7, 2002 16

SLAC

MBCD-II Parallelism

• Multibus-I backplane bandwidth is ~1 megabyte/second (2 microseconds/16-bit word).

• SLAC serial CAMAC bandwidth is ~¼ of that (~8 microseconds/16-bit word in block transfer mode).

• Solution to limited bandwidth problem was to operate 4 CAMAC cables in parallel.

E. Siskind, March 7, 2002 17

SLAC MBCD-II Performance Limits

• MBCD-II firmware only looks at one package at a time, so a package must use multiple crates on different cables in order to achieve parallelism.

• Non-time-critical tasks often require multiple packages to read one module, so no parallelism is achieved (e.g. analog status can’t read an entire SAM from collider arcs in a single package).

• Time-critical operations with small numbers of words/crate don’t realize full bandwidth because of limited Multibus-I package fetch bandwidth.

E. Siskind, March 7, 2002 18

SLAC Package Fetch Improvements

• FECC CAMAC hardware has much higher ratio of local memory/CAMAC bandwidth to avoid package fetch bottlenecks.

• Very large PCI DMA bandwidth between PC memory and PCIL local memory, and link DMA bandwidth between PCIL local memory and FECC local memory:– Straw between FECC memory and CAMAC;

– Fire hose between PCIL memory and FECC memory;

– River between PC memory and PCIL memory.

E. Siskind, March 7, 2002 19

SLAC

System Performance

• Many PC jobs now can have simultaneously outstanding long CAMAC packages.

• All packages are sent to FECC firmware ASAP, and parsed into their individual (block transfer) operations in a single crate (“packets”).

• Packets are immediately queued to hardware for the cable accessing the target crate.

• Cable operation begins as soon as any outstanding package has a packet requiring access to a crate on that cable.

• Parallelism now achieved over multiple packages.

E. Siskind, March 7, 2002 20

SLAC

MBCD Intelligence

• Set of four 2911 4-bit slices to make 16-bit microsequencer (AMD 2900 series of bit-slice microprocessors).

• Code from a 32 x 16 bit PROM via a single 82S105 FPLS.

• 5 MHz FPLS clock; 2911 clock generated by FPLS at lower frequency as required.

E. Siskind, March 7, 2002 21

SLAC

MBCD-II Intelligence

• Analog Devices ADSP-2101 12.5 MHz 16-bit DSP.

• 2k x 24 bit on-chip program memory; 1k x 16 bit on-chip data memory; 8k x 8 bit external program/data memory (unused).

• Actual application program uses no more than a few hundred instructions and very little data memory.

E. Siskind, March 7, 2002 22

SLAC

PCIL/FECC Intelligence

• Analog Devices ADSP-21060 “SHARC” 40 MHz 32-bit DSP on each board.

• 40k x 48 bit on-chip program memory; 64k x 32 bit on-chip data memory.

• 512k/2048k x 48 bit external program/data memory (256k/512k x 48 in 2nd generation PCIL).

• External DMA I/O data memory:– 512k/2048k x 32 bit in 1st generation PCIL;– 256k/512k x 64 bit in 2nd generation PCIL;– 1024k/2048 x 32 bit in 2nd generation FECC.

E. Siskind, March 7, 2002 23

SLAC

User FECC Programming• Real-time executive plus MBCD/Bitbus emulation

uses <10% of on-chip program memory and most of on-chip data memory.

• Remaining minimum of 7+ megabytes of memory on 40 MHz 32-bit processor similar to existing 386/486 CPU boards.

• Move existing 360 Hz interrupt driven code from micro to CAMAC interface to minimize CAMAC access delay.

• C run-time environment with dynamic heap allocation; full environment and stack preserved over context switch.

E. Siskind, March 7, 2002 24

SLAC

1st Generation Hardware• PCIL uses 32 bit/33 MHz PCI slot.• Separate fiber optic links for real-time/non-real-

time PCIL FECC communication.• Each link runs at 12.5 megabytes/second half

duplex; back compatible with 10 megabyte/second BIPI link (SLCnet backbone).

• PCIL peeks and pokes at FECC memory.• PCI master and link are FPGA-based DMA

peripherals of PCIL SHARC.• CAMAC is programmed I/O peripheral of FECC

SHARC, but link uses SHARC DMA channels.

E. Siskind, March 7, 2002 25

SLAC

1st Generation Status

• Hardware based on SHARC plus FPGA-based peripherals (PCIL-3; FECC-2).

• PCIL debugged in 1999; FECC in 2000.• Five PCIL/FECC board sets, plus 3 PCILs for

Alpha SLCnet interface.• Now intended primarily for development:

– Link protocol requires 3-4 round trips per message; bandwidth & latency degrade with distance.

– Link TAXI chips no longer available.

E. Siskind, March 7, 2002 26

SLAC

2nd Generation Hardware

• PCIL2 uses 64 bit/66 MHz PCI slot.• Single specialized fiber optic link for PCIL2

FECC2 communication (NLC SBIR).• 125 megabyte/second full duplex link, but

9/19 used for hardware overhead.• All peripherals FPGA-based DMA except

for Bitbus (programmed I/O).• Hardware based on SHARC plus one large

FPGA per board.

E. Siskind, March 7, 2002 27

SLAC

2nd Generation Link

• All traffic divided into 608 byte cells.• 320 bytes of user data (real-time, non-real-time,

pattern, no-op/zero); 32 bytes of hardware header; 256 bytes of error correction code.

• Cell requires 4.864 microseconds; can withstand 128 consecutive bytes or > 1 microsecond of lost data in that time.

• Klystron modulator pulse is > 5 microseconds wide with < 1 microsecond rise and fall times; can survive loss of all data on both rising and trailing edges of a single modulator pulse.

E. Siskind, March 7, 2002 28

SLAC

2nd Generation Link II

• If uncorrectable errors, hardware timeout and retransmission of all lost cells; 100 microsecond (20 cell lengths) typical timeout length.

• Hardware flow control for 16 real-time receive buffers and 16 non-real-time receive buffers.

• No start-of-message sequences to be corrupted by noise after initial turn-on.

• Hardware header supports remote manipulation of initialization register; local and remote hardware error counters.

E. Siskind, March 7, 2002 29

SLAC

Link Header Format

• All fields 16 bits unless noted.• Command :

– 4 bits - destination buffer number;– 1 bit - first cell in buffer;– 1 bit - last cell in buffer;– 2 bits - cell type (Pattern/R-T/N-R-T/NOP);– 1 bit - remote initialization register write enable;– 7 bits - unused.

• Offset in buffer to end of cell, i. e. cumulative buffer length including this cell (64 byte chunks).

E. Siskind, March 7, 2002 30

SLAC

Link Header Format II

• Remote initialization register write data (32 bits).• 16 non-real-time receive buffer available bits

(backpressure flow control).• 16 real-time receive buffer available bits

(backpressure flow control).• 16 non-real-time receive buffer filled handshake

bits.• 16 real-time receive buffer filled handshake bits.

E. Siskind, March 7, 2002 31

SLAC

Link Header Format III

• Outgoing cell sequence number.• Next expected incoming cell sequence number.• Multiple cells acknowledged event count.• Received cell out-of-sequence error count.• Timeout error count.• 8B/10B decoder error count.• Reed-Solomon corrected error count.• Uncorrectable Reed-Solomon error count.• Outgoing and incoming versions of last 8 counters

readable from SHARC.

E. Siskind, March 7, 2002 32

SLAC

2nd Generation Status

• PCIL2 link now debugged; PCI master not yet tested.

• Two PCIL2 boards built in 2001; three more in 2002.

• FECC2 design in progress; complete by mid-summer; debugged by end of CY02.

• Initial build of two FECC2 boards in 2002; also requires I/O concentrator panel (too many connectors for CAMAC front panel).

E. Siskind, March 7, 2002 33

SLAC

3rd Generation Design• Replace SHARCs by PowerPC embedded in each

board’s single FPGA.• Time scale: FPGAs available in CY02; begin in

CY03(???).• Benefits:

– Commercial real-time executive with improved software development tools;

– Clearly the upcoming embedded hardware/software technology.

• Shortcomings:– Need more time;– Need more money.

E. Siskind, March 7, 2002 34

SLAC SLAC CAMAC Bandwidth

• ~7.4 microseconds per 16-bit word in block transfer mode.

• Add round trip propagation delay to target crate controller for each word (up to 2.7 microseconds in SLC arcs).

• Add 1.6 microseconds for 24-bit data.• Subtract 3.2 microseconds for control operation

(F8=1).• Add 3.6/4.8 microseconds for new CNAF in

read/write operations.

E. Siskind, March 7, 2002 35

SLAC IEEE Standard CAMAC Bandwidth

• 3.6 microseconds per 16/24 bit word for read/write with new CNAF on each cycle.

• Add round-trip propagation to furthest crate controller.

• Subtract 0.8 microseconds for control operation.• Includes parity on each byte and message check

byte.• 2-3 times faster than SLAC serial CAMAC

controllers.

E. Siskind, March 7, 2002 36

SLAC Pipelined IEEE Standard CAMAC Mode

• 1.0 microsecond per additional 16/24 bit read/write cycle (full CAMAC crate bandwidth).

• No change to CNAF allowed.• No round-trip propagation delay degradation.• Order of magnitude improvement over SLAC

serial CAMAC controllers.• Only supported by Kinetic Systems model 3952

“enhanced” L-2 crate controller.

E. Siskind, March 7, 2002 37

SLAC

1st Generation Bandwidth

• ~1 megabyte/second bit serial CAMAC bandwidth (four SLAC CAMAC cables combined) (“straw”).

• 12.5 megabytes/second for each link; only one of read/write active at any time on each link (“fire hose”).

• 133 megabytes/second PCI master in block transfer read/write (“river”).

E. Siskind, March 7, 2002 38

SLAC

2nd Generation Bandwidth

• 8 megabytes/second enhanced byte serial CAMAC bandwidth (four IEEE CAMAC cables combined).

• ~1 megabyte/second bit serial CAMAC bandwidth (four SLAC CAMAC cables combined).

• <0.05 megabyte/second Bitbus bandwidth.• 125 megabytes/second link burst rate in each

direction; maximum average rate is 10/19 of this or 66 megabytes/second.

• 533 megabytes/second PCI master in block transfer read/write.

E. Siskind, March 7, 2002 39

SLAC

Bitbus Master

• Existing master based on MCS-51 2 MHz 8 bit microcontroller with SDLC/HDLC serial link.

• 8044BEM is an MCS-51 with mask-programmed firmware (iDCX-51 RTE plus Bitbus application).

• FECC2 has programmed I/O hardware (one SHARC interrupt/32 bits) to send a packet to one slave and receive that slave’s response.

• Poll list, counters, etc. maintained by new SHARC code in FECC2.

• Command queuing no longer an issue.

E. Siskind, March 7, 2002 40

SLAC

Software Group Projects

• Build, boot, and debug iRMX-III in off-the-shelf personal computers.

• Make real-time micro micro data communications streams coexist with slower back-end front-end traffic on switched TCP/IP network.

• Move 360 Hz interrupt processing from micro to FECC.

E. Siskind, March 7, 2002 41

SLAC

Software Projects II

• Change control of 360 Hz interrupt processing from shared memory to network link model.

• Modify fast feedback actuator to specify real-time priority for CAMAC operations.

• Modify slower micro jobs to send larger packages to CAMAC interface to maximize parallelism.

• Modify klystron job to protect PIOP CAMAC operations from interrupts.

E. Siskind, March 7, 2002 42

SLAC

Software Projects III

• Support logical to physical micro mapping.

• Support PC access to file server delivering current versions of PCIL and FECC executable images appropriate to different types of micros.

E. Siskind, March 7, 2002 43

SLAC

Schedule

• SWE development effort is currently in progress (TEG/RCS).

• All optimizations not necessary at early stage.

• Trying to deploy 1-2 PC-micros in production system during accelerator shutdown beginning this July.

• Begin AIP with 2nd generation PCIL-FECC in FY03/04.

E. Siskind, March 7, 2002 44

SLAC Simplified Hardware Configuration

• No PCPC fast feedback (KISnet equivalent) network traffic.

• Use 1st generation PCIL-FECC hardware:– No IEEE CAMAC or Bitbus;– Only need 32 bit/33 MHz PCI slot.

• MPG PC (PNET equivalent) pattern broadcast network traffic on separate real-time TCP/IP network.

• PC-based MPG (MP10/MP11) gets pattern from MP00/MP01 via reflective memory link.