DDL 1

ALICE Detector Data Link (DDL)and it’s use in STAR TOF

J. Schambach

DDL 2

Presentation Outline

• Hardware Overview

• DDL Protocol / Transactions

• RORC

• DDL in TOF

• Software

• Demo

DDL 3

Readout system

Front-end electronicsFront-end electronics

DetectorData Link

DDL SIUDDL SIU

DDL DIUDDL DIU

RORCRORC

SourceInterfaceUnit

DestinationInterfaceUnit

ReadOutReceiver Card

PCPCData Acquisition

PC

Optical Fibre~200 meters

DDL 4

DDL architecture

• Source Interface Unit (SIU) (1)

– Interface to the Front-end Electronics (2)

• Destination Interface Unit (DIU) (3)

– Interface to the Readout Receiver Card (4)

• Full duplex optical link (5)

– Multimode optical cable of up to 200 m

1

2

3

4

5

DDL 5

DDL hardware

DDL 6

SIU Physical Layout

DDL 7

DDL interfaces

• SIU-FEE interface– 3.3V (LVTTL) interface

– 32-bit wide half-duplex data bus (bi-directional bus)

– Bi-directional flow control

– User defined clock (synchronous interface)

– JTAG interface

• DIU-RORC interface– 3.3V (LVTTL) interface

– 32-bit wide full-duplex data bus

– Bi-directional flow control

– User defined clock (synchronous interface)

DDL 8

DDL Interface signals

DDL 9

SIU Connector Pinout

DDL 10

SIU-FEE interface

fbD(31..0) - data lines (bi-directional)fbTEN_N - transfer enable (bi-directional)fbCTRL_N - CONTROL qualifier (bi-directional)fiDIR - bus direction (FEE input)fiBEN_N - bus enable (FEE input)fiLF_N - link full (FEE input)foBSY_N - front-end busy (SIU input)foCLK - interface clock (SIU input)TAP_TCK - JTAG clock (FEE input)TAP_TDI - JTAG data in (FEE input)TAP_TDO - JTAG data out (SIU input)TAP_TMS - JTAG mode select (FEE input)TAP_TRST - JTAG reset (FEE input)

DDL 11

Link management

DDL 12

DDL Configurations

DDL 13

Front-end Commands

DDL 14

Front-end Status Words

Data Transmission Status Word (DTSTW), produced by SIU:

DDL 15

Front-end control

on-line on-line

FECTRL

CTSTW

on-line on-line

DIU SIURORCOnlineOnline

FEE command

Report

FEE

FEE control

idle

foCLK

fiBEN_N

fiDIR

FECTRLfbD

fbTEN_N

fbCTRL_N

DDL 16

Front-end status read

on-line on-line

FESTRD

FESTW

on-line on-line

on-line

DIU SIURORCOnlineOnline

FEE command

Status and report

FEE

FEE status read

CTSTW

foCLK

fiBEN_N

fiDIR

FESTRDfbD

fbTEN_N

fbCTRL_N

HiZ

HiZ

HiZ

foCLK

fiBEN_N

fiDIR

FESTWfbD

fbTEN_N

fbCTRL_N

HiZ

HiZ

HiZ

HiZ

HiZ

HiZ

DDL 17

Event read

foCLK

fiBEN_N

fiDIR

RDYRXfbD

fbTEN_N

fbCTRL_N

HiZ

HiZ

HiZ

foCLK

fiBEN_N

fiDIR

D0fbD

fbTEN_N

fbCTRL_N

HiZ

HiZ

HiZ

fiLF_N

D1 D2 D4 D5

foCLK

fiBEN_N

fiDIR

fbD

fbTEN_N

fbCTRL_N

fiLF_N

FESTWDn-1 Dn D0min. 16 cycles

foCLK

fiBEN_N

fiDIR

fbD

fbTEN_N

fbCTRL_N

fiLF_N

FESTWDn-1 Dn HiZ

HiZ

HiZ

foCLK

fiBEN_N

fiDIR

EOBTRfbD

fbTEN_N

fbCTRL_N

HiZ

HiZ

HiZ

DDL 18

Block write

on-line on-line

STBWR

CTSTW

DIU SIURORCOnlineOnline

FEE command

Report

FEE

STBWRCTSTW

FEE data

EOBTRFEE command

Report

EOBTR

Flow control

Block data data block

foCLK

fiBEN_N

fiDIR

D0fbD

fbTEN_N

fbCTRL_N

foBSY_N

D1 D2 D4 D5STBWR

foCLK

fiBEN_N

fiDIR

fbD

fbTEN_N

fbCTRL_N

foBSY_N

EOBTRDn-1 Dn

DDL 19

Block read

on-line on-line

STBRD

CTSTW

DIU SIURORCOnlineOnline

FEE command

Report

FEE

STBRDCTSTW

FEE data

EOBTR

data block

Block data

FEE command

Report

EOBTR

Flow control

foCLK

fiBEN_N

fiDIR

STBRDfbD

fbTEN_N

fbCTRL_N

HiZ

HiZ

HiZ

foCLK

fiBEN_N

fiDIR

D0fbD

fbTEN_N

fbCTRL_N

HiZ

HiZ

HiZ

fiLF_N

D1 D2 D4 D5

foCLK

fiBEN_N

fiDIR

fbD

fbTEN_N

fbCTRL_N

fiLF_N

FESTWDn-1 Dn HiZ

HiZ

HiZ

foCLK

fiBEN_N

fiDIR

EOBTRfbD

fbTEN_N

fbCTRL_N

HiZ

HiZ

HiZ

DDL 20

RORC features

• Interface between the DIU and PCI local bus– pRORC: 32 bit/33 MHz PCI version, max. throughput 132 MB/s– D-RORC: 64 bit/66 MHz PCI version, max. throughput 528 MB/s

• PCI master capability, data push architecture– Autonomous operation with little software assistance– Supports multi-paged memory management

• Direct data transfer to the PC memory– No local memory on the board– Small elasticity buffers between different clock domains

• Built-in test capability– Internal pattern generator can produce formatted data

DDL 21

D-RORC Hardware

D-RORC with integrated DIU ports• to read out two DDL channels• to support integration with the HLT system

D-RORC with plug-in DIU• to read out single DDL channel• to support the tests of FEE readout sytems

DDL 22

RORC Roadmap

• pRORC: 32-bit, 33 MHz (PCI I/F by ASIC)– well adapted to the prototype version of the DDL

– can be used for the new version of the DDL (adapters)

– already used by several test beams (SDD, HMPID)

• D-RORC I: 64-bit, 66 MHz (PCI I/F by IP core)– well adapted to the new version of the DDL

– will integrate two DIU functions on-board

– will support the DAQ/HLT interface

• D-RORC II: 64-bit, 66/133 MHz (PCI-X I/F by IP core)– to avoid compatibility issues

– to follow the evolution of the PCs

DDL 23

Hardware Architecture

APEXFPGA

64-bit/66 MHz, PCI/PCI-X

MediaI/F 1

MediaI/F 2

BusyI/F

Conf.Flash

JTAGJTAG

P11

P12

P13

P14

CMC I/F

Opt

ical

I/F

LV

DS

I/F

Configuring

250 MB/s

250 MB/s

528 MB/s

DDL 24

Firmware Architecture

PCI core(64-bit master, memory mapped)

Controlregisters

address lengthTransmit

DMA

address lengthReceiveDMA

RX

FIF

O

TX

FIF

O

DIU I/F or DIU core

Slave I/F Master I/F

RAF

TAF

DMA control

TAF – Transmit Address FIFO

RAF – Receive Address FIFO

DIU or Media I/F

PCI bus

DDL 25

The Free FIFO

PRORC PC memory bankFirmware

readout

page address

page address

page address

Free FIFO

PC CPU

Allocation of free pages

DDL 26

Direct Memory Access

PRORCFirmware

PC memory bank

DDL

No involvement

PC CPU

DDL 27

The Ready FIFO

PRORC PC memory bank

readout

DDL

Ready FIFOFirmware

lengthpage status

lengthpage status

lengthpage status

Delivery of filled pages

PC CPU

DDL 28

Software

Event ready test:

1) SOFTWARE

start = curr;start = curr;

do {do {

while (Ready FIFO [curr].status == 0 ) curr = NEXT(curr);while (Ready FIFO [curr].status == 0 ) curr = NEXT(curr);

} while (Ready FIFO [curr].status == 0xffffffff ||} while (Ready FIFO [curr].status == 0xffffffff ||

Ready FIFO [curr].status == 0)Ready FIFO [curr].status == 0)

end = curr;end = curr;

curr = NEXT( curr );curr = NEXT( curr ); Blocks from startstart to endend are now available

2) HARDWARE

delivery of signal, enabling of semaphore or setting of flag

Release event

Firmware

FirmwareFirmware

datalength

transferstatus

Ready FIFOReady FIFO

Event ready?N

Y

Pop descriptor from Free FIFOMove data into buffer until DTSTW

or up to buffer sizeUpdate size

size = 0

DTSTW?

Y

N

size > max size?

N

Y

Throw away rest of datado not increment block length

until DTSTW receivedMark DTSTW “OVERRUN” bit

Push data length in Ready FIFO

Push transfer status = 00000000Optional: update Ready FIFO in memory

Push data length in Ready FIFOPush transfer status = DTSTWUpdate Ready FIFO in memory

Deliver “done” interruptOptional: set stop flag on error

Free FIFO empty?

Y

N

block:Reset entries of Ready FIFO

Push address & size on Free FIFO(if space available in Free FIFO)

Pull data length and transfer statusfrom Ready FIFO

Push pending free blockson Free FIFO (if any)

Load Free FIFO

Initialize Ready FIFO

Load configuration registers

Clear stop flag

Initialize internal data structuresstop flag = ON

acknowledge reset done

Transfer status possible values:ffffffffffffffff unloaded (set by sw)0000000000000000 loaded, no DTSTW (set by fw)elseelse loaded, DTSTW (set by fw)

Free FIFOFree FIFO

blocksize

Index ofReady FIFO

startaddress

stop flag ON?

N

Y

RORCRORC

DDL 29

Test equipments

• Front-end Emulator Interface Card (FEIC)– Fully functional hardware to emulate the detector front-ends

– Formatted data block generation

– Internal (free running) or external (pulse) triggering capabilities

– Adjustable parameters (using front-end control)

– Operates at the nominal speed of the DDL

• Source Interface Unit Simulator (SIMU)– Simulates the behavior of the DDL without any additional

hardware

– Eases the development and the hardware debugging

– Size is similar to the real SIU

DDL 30

Bandwidth: D-RORC to D-RORC

Testing the transfer between two D-RORC cards

• rorc_send –g ... (pattern generator)• rorc_send ... (DMA from)

DDL 31

TOF Essential Model Level 1

Start Detector with

24 Scintillators &24 PMTs

Start DetectorElectronics

Tray

32 RPCWith 6 channels

eachOn-Tray Electronics

Data, Clk, Ctrl

CANbus

30Trays

L0 Trigger

Data, Clk, Ctrl

Dual Fiber DAQTHUB

Master Clock

CA

Nb

us

Tray

32 RPCWith 6 channels

eachOn-Tray Electronics

Data, Clk, Ctrl

CANbus

30Trays

L0 Trigger

Dual Fiber DAQTHUB

Slave Clock

CA

Nb

us

OneDetector Side

CA

Nb

us

Clk

, C

trl

To other THUB’s

Clk, Ctrl

Clk, Ctrl

Control PCwith

CANBus I/F

TCD & BusyLogic

DDL 32

TOF Essential Model Level 2

Start Detector

TOF Tray

TDIG TDIG

TCPU

DIFFERENTIAL DATA AND CLOCK

TINO TINO

MRPC MRPC

COMMANDS

48 CHAN

48 CHAN

48 CHAN

TRAY CAN BUS

MULTIPLICITY

THUB

COPPER: DATA, SAMPLE CLOCK, RESET TRIGGER STROBE & DATA

TOP LEVEL CAN BUS

COPPER LINKS TO 29 TRAYS

RHIC CLOCK

SIU FIBER

CAN BUS TO 29 TRAYS

TCD

DAQ

48 CHAN

4

L0 Trigger

4

PMTs TPMT TSTD TCPU

Diff. Data & Clk

Local CANbus

Co

pper

:D

ata

, S

am

ple

Clk

, Re

set

Trg

Str

obe

& D

ata

Reset & ClockFrom other THUBs

Reset & ClockTo other THUBs

DDL 33

Run 5 TCPU

MCU

PLDALTERA STRATIXEP1S10

TDIG CABLE I/F

TRAY LEVEL CAN BUS I/F

ROM

TOP LEVEL CAN BUS I/F

TCD I/F

DDL-SIU

TDIG CABLE I/F

TDIG CABLE I/F

TDIG CABLE I/F

Clock select and distribute

Leds and switches

Local osc

Ext clock input

Leds, switches,

temperature monitor

Config eprom

Jtag for codetap

Test header

Power

JTAG multiplexer

ICD2header

JTAG for config

Ext clock output

Ext ‘reset’ output

To TDIG cables

DDL 34

DDL Interface Implementation

DDL 35

TCPU Firmware Data Path

FIFO256 X 32

5:1 MUX

FIFO

FIFO

FIFO

FIFO2048 X

32

L2 Trigger Data

MUX FIFO4 TO 32

BITDEMUX

4 TO 32 BIT

DEMUX

4 TO 32 BIT

DEMUX

4 TO 32 BIT

DEMUX

TDIG CABLE

TDIG CABLE

TDIG CABLE

TDIG CABLE

FIFO2048 X

32

MCU I/F

MCU

CAN BUS

TCD I/FFIFO

64 X 32

DDLSIU

TCD CABLE

FIBER

DDL 36

DDL Software

• PCI driver

• API routines for DATE

• Executable utility programs for test and stand-alone (= without FEE or DATE) use of the DDL, such as– Reset the RORC, DIU or SIU

– Display the status of the RORC, DIU or SIU

– Test the functionality and measure the performance of the whole DDL and RORC system

DDL 37

Installation of the utilities

• Utilities are in the directory

./rorc/Linux/• Linux kernel version: 2.4• Driver module must be inserted. As root type:

/sbin/insmod ./Linux/rorc_driver.oor insert a similar line into /etc/rc.d/rc.local

• Checked if physmem and RORC driver are loaded:

cat /proc/modules• Check if RORC card is plugged:

rorc_find

DDL 38

$ /sbin/lspci

00:00.0 Host bridge: Intel Corp. 82840 840 (Carmel) Chipset Host Bridge (Hub A) (rev 01)

00:01.0 PCI bridge: Intel Corp. 82840 840 (Carmel) Chipset AGP Bridge (rev 01)

00:02.0 PCI bridge: Intel Corp. 82840 840 (Carmel) Chipset PCI Bridge (Hub B) (rev 01)

00:1e.0 PCI bridge: Intel Corp. 82801AA PCI Bridge (rev 02)

00:1f.0 ISA bridge: Intel Corp. 82801AA ISA Bridge (LPC) (rev 02)

00:1f.1 IDE interface: Intel Corp. 82801AA IDE (rev 02)

00:1f.2 USB Controller: Intel Corp. 82801AA USB (rev 02)

00:1f.3 SMBus: Intel Corp. 82801AA SMBus (rev 02)

01:05.0 Multimedia audio controller: Cirrus Logic CS 4614/22/24 (rev 01)

01:06.0 Network controller: CERN/ECP/EDU: Unknown device 0033 (rev 01)

01:08.0 Ethernet controller: Accton Technology Corporation SMC2-1211TX (rev 10)

02:1f.0 PCI bridge: Intel Corp. 82806AA PCI64 Hub PCI Bridge (rev 02)

03:00.0 PIC: Intel Corp. 82806AA PCI64 Hub Advanced Programmable Interrupt Controller (rev 01)

03:04.0 Network controller: CERN/ECP/EDU: Unknown device 0033 (rev 02)

03:09.0 SCSI storage controller: Adaptec AIC-7892P U160/m (rev 02)

04:00.0 VGA compatible controller: Matrox Graphics, Inc. MGA G200 AGP (rev 03)

Identifying the RORC cardIdentifying the RORC card

DDL 39

Program ‘rorc_find’

List all plugged and presently not used RORC devices together with their version and serial numbers.

rorc_find

The following device(s) found:

-------------------------------------------------------------------------------

Minor Channel Device type and HW identification (RORC’s FW version)

-------------------------------------------------------------------------------

0 0 D-RORC (FW: 1v35 of July 2 2003)

new DIU DDL card 2v0 LD: 20K60E SP: 2125 Mbps S/N: 00102

0 1 D-RORC

no DIU

1 0 pRORC pRORC 1v1 S/N: 00101 (FW: 1v75 of June 6 2003)

new DIU DDL card 2v0 LD: 20K60E SP: 2125 Mbps S/N: 00118

-------------------------------------------------------------------------------

3 RORC channel(s) not in use was found.

RORC driver reported 2 RORC device(s).

DDL 40

rorc_reset

Initialize the RORC card and the DDL link

rorc_reset [-{M|m} <minor> | 0] [-{D|d|B|b|S|s|F|f|O|o|E|e|C|c}]Where-D or –d reset the DIU-B or –b reset both RORC and DIU-S or -s reset SIU-F or –f clear Free FIFO-O or –o clear RORC’s other FIFOs-E or –e clear RORC error bits-C or –c clear RORC’s byte counterNo option reset RORC

DDL 41

rorc_id

Display RORC’s hardware and software identification wordsrorc_id [-{M|m} <minor> | 0] [-{D|d}{S|s}] [-{T|t} <time-out> | 100000]Where-D or –d display DIU’s firmware id as well-S or –s display SIU’s firmware id ad welltime-out time-out value for DIU or SIU replye.g.:rorc_idRORC driver version: 4.2RORC revision id: 1Hardware identity word of the RORC: pRORC 1v1 S/N:00103`, i.e. Version: 1.1, S/N: 00103. Firmware identity word of the RORC: 0x02190550, i.e. Version: 1.72 Release date : October 16 2002

Free FIFO size: 128 entries

DDL 42

rorc_send_command

Send a DDL command and receive the reply

rorc_send_command [-{M|m} <minor> | 0] -{C|c} <command> [-{T|t} <time-out> | 100000]

[-{V|v} <diu_version> | 2]Wherecommand a hexadecimal number starting with “0x”, or

an ASCII mnemonic of a DLL or pRORC command, e.g.:LBON RORC loop-back onRDYRX ready to receive message to the SIUEOBTR end of block transfer message to the SIU(see the sw manual or the program help for command codes and format)

time-out time-out value for RORC, DIU, SIU or Front-End replydiu_version 1 for prototype, 2 for final version.

DDL 43

rorc_receive

Send and receive data to the Front-End. The most important options:rorc_receive [-{M|m} <minor>|0] [-{G|g}] [-{Y|y}] [-{Z|z}] [-{X|x} <check_level>] [-{K|k} <output file>] [-{E|e} <events>] [-{P|p} <pattern>|0] [-{I|i} <init word>|0] [-Q <GBytes>|1]Where (from version 4.2: rorc_receive)-G send data using RORC’s data generator-Y do not loop-back generated data but send it via the link-Z do not send RDYRX and EOBTR commandsoutput file dump the receives the data into this file without checking itcheck_level 0: do not check the received data, 1: check only the first word,

2: not the first word, 3: check the whole event events # events to sendpattern event pattern to send or receive, it could be: c, a, 0, 1, I, dinit word the first word of each event’s payload (after the event serial number)GBytes display the number of received bytes after each received Gbytes data

(see the sw manual for further options)

DDL 44

feic.menu

Check and set the Front-End Emulator Interface Card

feic.menu [-{M|m} <minor>|0]

This script calls the rorc_send_command program several times to check the FEIC setting or to set new parameters. The following features of the FEIC’s data generator can be modified and displayed:

data pattern alternating, flying 0 or 1, incrementing, decrementing dataevent length 16, 32, 64, …., 256 Kwordstrigger mode push button, ext. trig., 16 or 128 clocks after each event,

every 10 or 100 msseed the seed value for random data length

See the description of rorcArmFeic routine in the sw manual for further details.

DDL 45

Other utilities

program name program function

diu_id DIU hw and fw identification

siu_id SIU hw and fw identification

rorc_status show RORC status

rorc_reg show RORC registers

diu_status ask and display DIU status

siu_status ask and display SIU satatus

rorc_send download data from the PC to the FEE

rorc_send_jtag download JTAG data