Some Data Rate Issues in the L1 Trigger System

Some Data Rate Issues in the L1 Trigger System

July 2004

Outline

• Capacities of data flow at the connection ports.

• Relative data rate evolution in each stage.

L1 Block Diagram

L1 Switch

TSO

PP

L1BServers

ST L1B

BM

CPU

TSO

PP

ST

TSO

PP

ST

TSO

PP

ST

TSO

PP

ST

TSO

PP

ST

TSO

PP

ST

TSO

PP

ST

TSO

PP

ST

TSO

PP

ST

TSO

PP

ST

TSO

PP

ST

BM

CPU

BM

CPU

BM

CPU

BM

CPU

BM

CPU

BM

CPU

BM

CPU

Pixel Data

Time Stamp

Ordering

Cluster Processing

Raw Data to L1B

Data Sharing

Segment Finding

Event Building

Track & Vertex

Processing

L1BServers

L1B

GL1Node

Triplets to L1B

Tracks & Vertices to L1B

Trigger primitives to GL1

Gang Things Together & Reroute Cables

L1 Switch

TSO

PP

L1BServers

ST L1B

BM

CPU

TSO

PP

ST

TSO

PP

ST

TSO

PP

ST

TSO

PP

ST

TSO

PP

ST

TSO

PP

ST

TSO

PP

ST

TSO

PP

ST

TSO

PP

ST

TSO

PP

ST

TSO

PP

ST

BM

CPU

BM

CPU

BM

CPU

BM

CPU

BM

CPU

BM

CPU

BM

CPU L1BServers

L1B

GL1Node

TSO TSO TSO TSO

PP PP PP PP

Events are built during TSO & PP processing.

ST ST ST ST ST ST ST STST/L1BST/L1B ST/L1BST/L1B ST/L1BST/L1B ST/L1BST/L1B

L1 Switch is gone, its functions are absorbed in TSO & PP stages.

Triple links are not needed.

L1B for raw data, triplets

& etc.

System Interconnection (1 Hwy)Time Stamp

Ordering Module

Pixel Pre-processor Module

Segment Tracker & L1B Module

Buffer Manager Module

L1B Server PC

Worker Farm Node

10

16

64

16

3232

A possible GL1 Interconnection

BMHash Sorter

SDRAM

SRAM-ZBT128K x 32

ST2-in

2-out

SDRAM

SRAM-ZBT128K x 32

L1BLogic

ST2-in

2-out

SDRAM

SRAM-ZBT128K x 32

L1BLogic

ST2-in

2-out

SDRAM

SRAM-ZBT128K x 32

L1BLogic

ST2-in

2-out

SDRAM

SRAM-ZBT128K x 32

L1BLogic

GL1Interface

SDRAM

SRAM-ZBT128K x 32

BMHash Sorter

SDRAM

SRAM-ZBT128K x 32

(1) GL1 are sent out by the Worker Nodes.

(2) The TSO modules can be used as concentrators.

(3) The same BM module is used as GL1

interface.

GL1 Node

Farm Nodes

ST/L1B

Throughput Capacities

Media # /hwy

Throughput capacities

Data Rates /hwy Safety factors

PDCB to TSO

2.5 Gbps fibers

120 240 Gbps 53 Gbps

(Doc 3233)

4.5

TSO to PP 500 Mbps x 4 pairs

160 320 Gbps < 53 Gbps > 6

PP to ST/L1B

500 Mbps x 4 pairs

128 256 Gbps < (53+53) Gbps > 2.4

ST/L1B to/from BM/Worker

500 Mbps x 2 pairs each way

128 128 Gbps < (53/4) Gbps > 9.6

128 128 Gbps 200 MB/s > 64

BM to GL1 500 Mbps x 2 pairs each way

4 4 Gbps 47 MB/s > 10

Relative Data Volume Evolution in Different Stages

• When the data are repackaged (such as in PDCB), data rate may be reduced.

• When the data are reordered (such as in Time Stamp Ordering Module), data rate may be reduced.

• When the data are duplicated (such as in Pixel Pre-processor Module), data rate will increase.

Input & Output of PDCB

b04 b03 b02 b01 b00b09 b08 b07 b06 b05b14 b13 b12 b11 b10b15

b04 b03 b02 b01 b00b09 b08 b07 b06 b05b14 b13 b12 b11 b10b15b20 b19 b18 b17 b16b23 b22 b21

Row Column BCO(7:0) ADC 1Hit24

1Status 000Sync24

0 0XXXInvalid coding:

1 1 1 X0

ADC0BCO(11:3)

Row ColumnModule #

Chip #DW0:DW1:

Hits 0 0ADC1ADC2ADC3 BCO(2:0)Optional Continue Word:

DCC et al

Doc 2621

12.3 Ghits/s x 24 b/hit = 0.3 Tbps (total), 37.5 Gbps/hwy

12.3 Ghits/s /(2.5 hits/group)x 48 b/group = 236 Gbps (total), 29.5 Gbps/hwy

Doc 3233

Idling, an Important Operation


1 1 1 X0Status etc.Idle Words: 1 1 1 X0Status etc.

00

ADC0BCO(11:3)

Row ColumnModule #

Chip #DW0:DW1:

Hits 0 0ADC1ADC2ADC3 BCO(2:0)Optional Continue Word:

ADC0BCO(11:3)

Row ColumnModule #

Chip #

Hits 0 0ADC1ADC2ADC3 BCO(2:0)

Output of Time Stamp Ordering Module

Hits: 12.3 Ghits/s /(2.5 hits/group)x 32 b/group = 157 Gbps (total), 19.7 Gbps/hwy


Hits 0 0ADC4ADC5ADC6CW: ADC3

NSADC0

Row ColumnModule #

Chip #DW0:DW1: ADC1ADC2Hits

BCO(21:9)

BCO(9:0)

Idle Words:1 1 1 X0

Status etc.00

0 0

0 0

1

1

1

0

0

1

0

1

0

1

Station 1

Station 2

Station 0

Station 0

BCO0

BCO0+1

Output of Pixel Pre-processor Module

Raw Hits: 12.3 Ghits/s /(2.5 hits/cluster)x 32 b/cluster = 157 Gbps (total), 19.7 Gbps/hwy


XY: 12.3 Ghits/s /(2.5 hits/group)x (32+16)b/(2 groups) = 118 Gbps (total), 14.7 Gbps/hwy

Hits 0 0ADC3ADC4ADC5Same Column CW:

NSADC0

Row ColumnModule #

Chip #DW0:DW1: ADC1ADC2Hits

BCO(21:9)

BCO(9:0)

Idle Words:1 1 1 X0

Status etc.00

0 0

0 0

1

1

Separator TypeSeparator Word: 1 1 1 X0

x0x(12:9)XY Words: Vx0x or y0y(8:0)

1

0

NSy0x(8:5)

y0x or x0y(4:0)

Vx1x or y1y(8:0) 0y1x or x1y(4:0)Additional XY Words:

Hits 1 0ADC3ADC4ADC5Cross Column CW: Del Row2

Data Rates

Media # /hwy

Throughput capacities

Data Rates /hwy Safety factors

PDCB to TSO

2.5 Gbps fibers

120 240 Gbps 29.5 Gbps 8.1

TSO to PP 500 Mbps x 4 pairs

160 320 Gbps 19.7 Gbps 16.2

PP to ST/L1B

500 Mbps x 4 pairs

128 256 Gbps (19.7+14.7) Gbps 7.4

ST/L1B to/from BM/Worker

500 Mbps x 2 pairs each way

128 128 Gbps < (19.7/4) Gbps > 26

128 128 Gbps 200 MB/s > 64

BM to GL1 500 Mbps x 2 pairs each way

4 4 Gbps 47 MB/s > 10

Conclusion:• We have large safety factors.

I/O of Time Stamp Ordering Moduleb04 b03 b02 b01 b00b09 b08 b07 b06 b05b14 b13 b12 b11 b10b15

ADC0BCO(11:3)

Row Column

Module #

Hits

Chip #

1 1 1 X0

0 0ADC1ADC2ADC3

Status etc.

DW0:DW1:

BCO(2:0)Optional Continue Word:

Idle Words: 1 1 1 X0Status etc.

00

Station #


ADC0

BCO(23:7)

Row Column

Module #

Chip #

HitsADC1ADC2

BCO(7:0) 0 0Header Long Word:

Hits Long Word:

Idle Long Words:

I/O of Time Stamp Ordering Module

Station #

BCO(23:7)

BCO(7:0) 0 0Header Long Word:

Hits: 12.3 Ghits/s /(2.5 hits/group)x 32 b/group = 157 Gbps (total), 19.7 Gbps/hwy


b25 b24b30 b29 b28 b27 b26b31 b20 b19 b18 b17 b16b23 b22 b21


ADC0BCO(11:3)

Row Column

Module #

Chip #DW0:DW1:

Hits 0 0ADC1ADC2ADC3BCO(2:0)Optional Continue Word:

ADC0

Row Column

Module #

Chip #

HitsADC1ADC2Hits Long Word: NX

Headers (1/hwy): 7.6 MBOC/s x (1/8hwy) x 32b/BCO/port x 480 ports = 14.6 Gbps/hwy

Some Data Rate Issues in the L1 Trigger System

Documents

Transcript of Some Data Rate Issues in the L1 Trigger System