Design of a Diversified Router: Dedicated CRF for IPv4 Metarouter

John DeHart, Brandon [email protected], [email protected]

http://arl.wustl.edu/projects/techX/

Design of aDiversified Router:

Dedicated CRFfor IPv4 Metarouter

2 - JDD - 04/19/23

Revision History 5/22/06 (JDD):

» Created» Buffer descriptor stuff probably needs updating.

6/1/06 (JDD):» Updating data going between blocks, still in progress.

6/2/06 (JDD):» More cleanup of data going between blocks.» Buffer descriptor details still need updating.

6/5/06 (JDD):» Slight change to format for Lookup Key and defining what goes in each word in the NN ring.» Add IP Pkt Length to data Demux passes to Parse

6/6/06 (JDD):» Reorganized the Lookup Result given to Hdr Format to distinguish between MR portion and

Substrate portion.» Clean up labeling of data to Parse (MN vs. IP Pkt)

Output from Parse is still IP Pkt Offset and Length.» Data from Parse to Lookup needs update to reflect case where lookup is just for Substrate

mapping of MI to LC. 6/7/06 (JDD):

» Updated notes about Parse block’s input/output and functionality 6/15/06 (JDD):

» Removed CRC from Rx to Demux data. MSF does not pass us a CRC like we thought so we will skip the CRC checking.

» Updated data going from Demux to Parse, Parse to Lookup and Lookup to Hdr Format

3 - JDD - 04/19/23

Revision History 6/19/06 (BDH):

» Split Header Format into MR Header Format and Substrate Encap» Demux is now Substrate Decap» Reorganization of all slides into logical and physical formats, coloring scheme» IPv4 MR now has own section, integrated JL’s internal format slides

6/21/06 (BDH):» H Flags nuked» MN Pkt Length into Lookup is now substrate-defined» Logical communication added from Lookup to Substrate Encap» Port fields are all 4 bits now

6/26/06 (BDH):» Substrate Decap to Parse format changed» Changed block diagram to better show that substrate encloses the MR-specific portions» Added details on Substrate Decap» Moved IPv4 slides to techX\bdh4\techx\IPv4_MR_shared. These slides should be done by

mid-July. 6/29/06 (BDH):

» Updated format of Tx input 6/30/06 (BDH):

» Updated format of Hdr Format to Substrate Encap data, only handles IPv4 NH_MN_ADDRs now

4 - JDD - 04/19/23

Dedicated CRF Slide Organization

BlockInput Data Output Data

In the “at-a-glance” format, all blocks are logical» Logical inputs and outputs» High-level overview of processing» Each logical block is like an Intel microblock, not necessarily an ME

In the detailed format, all blocks are physical» Physical inputs and outputs» Specific functionality and implementation notes

Color scheme» Blue = Substrate, should not change!» Green = Metarouter, different for each MR

Substrate

LookupRx TxQMSubstrDecap

SubstrEncap

L1 L2L3 L3 L3

L2 L2 L1

ParseHeaderFormat

MR MR

Metarouter

Logical Formats

6 - JDD - 04/19/23

Receive

RxRBUF

Buffer Handle

Ethernet Frame Len

Port

Coordinate transfer of packets from RBUF to DRAM


SubstrEncap

L1 L2L3 L3 L3

L2 L2 L1

ParseHeaderFormat

MR MR

7 - JDD - 04/19/23

Substrate Decap

Substr Decap

Read and validate Ethernet header from DRAM Read and validate substrate header from DRAM Extract Source ID Calculate MN frame length and offset

Buffer Handle

Ethernet Frame Len

Port

Destination MPE

Source ID

MN Frame Length

MN Frame Offset

Buffer Handle


SubstrEncap

L1 L2L3 L3 L3

L2 L2 L1

ParseHeaderFormat

MR MR

8 - JDD - 04/19/23

Parse

Parse

Substrate matches the destination MPE Read and align MN header (includes IPv4 Hdr) from DRAM MR-specific

» Consume internal header (if packet from other MPE of MR)» Header validation» Header modification» Exception checks» Extract lookup key and set lookup flags

Write aligned modified IPv4 header back to DRAM

Destination MPE

Source ID

MN Frame Length

MN Frame Offset

Buffer Handle

Lookup Flags

Lookup Key

Buffer Handle

Source ID

MR Data to MR Hdr Format

to Lookup

MN Pkt Length


SubstrEncap

L1 L2L3 L3 L3

L2 L2 L1

ParseHeaderFormat

MR MR

9 - JDD - 04/19/23

Lookup

Lookup

Buffer Handle

Perform lookup in TCAM Increment counters based on Stats Index Priority resolution of results from multiple databases,

if needed

Lookup Input Flags

Lookup Key

Buffer Handle

MN Pkt Length

Lookup Result Flags

MR Lookup ResultSource ID

Dest Addr

Output Port

QID

to MR Header Format

to Substrate Encap


SubstrEncap

L1 L2L3 L3 L3

L2 L2 L1

ParseHeaderFormat

MR MR

10 - JDD - 04/19/23

Header Format

MR Hdr Format

Process Lookup result For exceptions, generate internal header Decide substrate type

MR Data

from Lookup

from MR Parse

Buffer Handle

MN Frame Offset

Buffer Handle

Lookup Result Flags

MR Lookup Result Substrate Type

Substr. Type-dep. Data

MN Frame Length


SubstrEncap

L1 L2L3 L3 L3

L2 L2 L1

ParseHeaderFormat

MR MR

11 - JDD - 04/19/23

Substrate Encap

Substr Encap

Buffer Handle

Output Port

Write substrate and ethernet headers

MN Frame Length

MN Frame Offset

Substrate Type

Substr. Type-dep. Data

QID

MN Frame Length

Buffer Handle

Dest Addr

Output Port

QID

from Lookup

from MR Header Format


SubstrEncap

L1 L2L3 L3 L3

L2 L2 L1

ParseHeaderFormat

MR MR

12 - JDD - 04/19/23

Queue Manager

QM

CRF queue management for Meta Interface queues

WRR? Details

Buffer Handle

Output Port

QID

MN Frame Length


SubstrEncap

L1 L2L3 L3 L3

L2 L2 L1

ParseHeaderFormat

MR MR

Buffer Handle

Output Port

Valid

13 - JDD - 04/19/23

Transmit

Tx

Coordinate transfer of packets from DRAM to TBUFs Recycle buffer handle

TBUF


SubstrEncap

L1 L2L3 L3 L3

L2 L2 L1

ParseHeaderFormat

MR MR

Buffer Handle

Output Port

Valid

Physical Formats

15 - JDD - 04/19/23

Receive

RBUF format details here Buf Handle details here Notes:

» We’ll pass the Buffer Handle which contains the SRAM address of the buffer descriptor.

» From the SRAM address of the descriptor we can calculate the DRAM address of the buffer data.

VLAN

Packet_Next

MR_ID

TxMI

Free_List

Packet_Size

Buffer_Next

Offset

BufferDescriptor

Buffer_Size

RBUF Buf Handle(32b)

Port(8b)

Reserved(8b)

Eth. FrameLen (16b)

Buf Handle(32b)

Size (16b) Offset (16b)

Hdr Type(8b)

Freelist(4b)

Port (16b)Rx

status(4b)

currently:

will be:

16 - JDD - 04/19/23

Substrate Decap

SourceID:» specifies RxMI or MPE (each 15-bit)

VLAN

Packet_Next

MR_ID

TxMI

Free_List

Packet_Size

Buffer_Next

Offset

BufferDescriptor

Buffer_Size

currently:

Buf Handle(32b)

Port(8b)

Reserved(8b)

Eth. FrameLen (16b)

Source ID(16b)

Buf Handle(32b)

MN Frm Offset (16b)MN Frm Length(16b)

Dest MPE (16b)

Buf Handle(32b)

Size (16b) Offset (16b)

Hdr Type(8b)

Freelist(4b)

Port (16b)Rx

status(4b)

will be:

17 - JDD - 04/19/23

Substrate Decap Functions

Read Ethernet VLAN and Substrate header from DRAM Validate Ethernet VLAN packet

»Valid Length?»Known protocol (VLAN)?»Broadcast/Multicast source?»Multicast destination?»Broadcast destination?»Local Dest?

Validate Substrate header»Known substrate header type (Internal or Ingress)?»Substrate-reported MN frm len == Enet-deduced MN frm len?

Fill NN ring fields

18 - JDD - 04/19/23

Substrate Decap Implementation 8 threads, ordered thread execution 121 cycles per thread per packet, common case ~670 cycles of latency, within 1360 cycle limit for 8

threads Resource use:

»SRAM refs: 1 per counter to increment (disabled currently)

»DRAM refs: 3 8B reads: Enet and Substrate header 2 8B reads: Enet checksum

Optimizations could reduce cycle count further»projected: 80-100 cycles»combined initial error-check to remove branch mispredicts» remove/combine DRAM read signals» remove volatile keywords»single-critical-section ordered threading

19 - JDD - 04/19/23

Parse

Lookup Key[111-80] (32b)

MR Data (28b)

Buf Handle(32b)

Lookup Key[ 79-48] (32b)


Lookup Key[15- 0] (16b) Reserved (16b)

Lookup Key[143-112] MR/MI (32b)

LFlags(4b)

VLAN

Packet_Next

MR_ID

TxMI

Free_List

Packet_Size

Buffer_Next

Offset

BufferDescriptor

Buffer_Size

Can Parse adjust the buffer/packet size and offset? Can Parse do something like, terminate a tunnel and strip off an outer

header?

MR Data (16b)MN Pkt Len (16b)Source ID (16b)

Buf Handle(32b)

MN Frm Offset (16b)MN Frm Length(16b)

Dest MPE (16b)

20 - JDD - 04/19/23

Lookup

L Flags: » bit 0: 0: Normal, 1: Substrate Lookup» bit 1: 0: Normal, 1: NH MN Address present in

Key Word[1] Key Word[0] = MR/MI Bit 1 should never be set without bit 0 also being

set.

Port(4b)

MR Lookup Result (32b)

QID(20b)DA(8b)


Buf Handle(32b)

MR Data (28b)Rsv(4b)

VLAN

Packet_Next

MR_ID

TxMI

Free_List

Packet_Size

Buffer_Next

Offset

BufferDescriptor

Buffer_Size

Lookup Key[111-80] (32b)

MR Data (28b)

Buf Handle(32b)

MR Data (16b)



Lookup Key[15- 0] (16b) Reserved (16b)

Lookup Key[143-112] MR/MI (32b)

LFlags(4b)

MN Pkt Len (16b) MR Data (16b)MN Pkt Len (16b)

21 - JDD - 04/19/23

Header Format

Egress Simple and Internal formats use just the dest ID, source ID, ad SH type

MAC fields used for MAC_ADDR Egress format

NH MN Addr field used for NH_MN_Addr format

VLAN

Packet_Next

MR_ID

TxMI

Free_List

Packet_Size

Buffer_Next

Offset

BufferDescriptor

Buffer_Size

Port(4b)


QID(20b)DA(8b)


Buf Handle(32b)

MR Data (28b)Rsv(4b)

MR Data (16b)MN Pkt Len (16b)MN Pkt Length (16b)

Buffer Handle(32b)

MN Pkt Offset (16b)

Source ID(16b)

NH MN IPv4 Addr / MAC Lo (32b)

Port(4b) QID(20b)DA(8b)

Dest ID(16b)

MAC Hi (16b)Rsv(8b)

SH Type(8b)

22 - JDD - 04/19/23

Substrate Encapsulation

Substrate header types/formats here?

VLAN

Packet_Next

MR_ID

TxMI

Free_List

Packet_Size

Buffer_Next

Offset

BufferDescriptor

Buffer_SizeMN Pkt Length (16b)

Buffer Handle(32b)

Reserved (16b)


MN Pkt Length (16b)

Buffer Handle(32b)

MN Pkt Offset (16b)

Source ID(16b)

NH MN IPv4 Addr / MAC Lo (32b)


Dest ID(16b)

MAC Hi (16b)Rsv(8b)

SH Type(8b)

23 - JDD - 04/19/23

Queue Manager

Text VLAN

Packet_Next

MR_ID

TxMI

Free_List

Packet_Size

Buffer_Next

Offset

BufferDescriptor

Buffer_SizeMN Pkt Length (16b)

Buffer Handle(32b)

Reserved (16b)


Buffer Handle (24b)Port(4b)

Rsv(3b)

V(1b)

24 - JDD - 04/19/23

Transmit

Text

TBUF

VLAN

Packet_Next

MR_ID

TxMI

Free_List

Packet_Size

Buffer_Next

Offset

BufferDescriptor

Buffer_Size

Buffer Handle (24b)Port(4b)

Rsv(3b)

V(1b)

IPv4 Metarouter

Look at:

techx\bdh4\techx\IPv4_MR_shared… for Metarouter-specific IPv4 slides

26 - JDD - 04/19/23

Extra The next set of slides are for templates or extra

information if needed

27 - JDD - 04/19/23

Text Slide Template

28 - JDD - 04/19/23

Image Slide Template

29 - JDD - 04/19/23

At-a-glance Block Template

LookupRx TxQMMR

ParseMR HdrFormat

SubstrDecap

SubstrEncap

L1 L2L3 L3 L3

L2 L2 L1

BlockRBUF

Buffer Handle

Ethernet Frame Len

Port

Text

30 - JDD - 04/19/23

Detailed Block Template

Text VLAN

Packet_Next

MR_ID

TxMI

Free_List

Packet_Size

Buffer_Next

Offset

BufferDescriptor

Buffer_Size

RBUF Buf Handle(32b)

Port(8b)

Reserved(8b)

Eth. FrameLen (16b)

31 - JDD - 04/19/23

QM/Scheduler on Multiple MEs

QM/Schd(1 ME)

QM/Schd(1 ME)

NN/Scratch Rings

InputHlpr

(1 ME)

NN Ring

HeaderFormat TxMR-1

MR-n. . .

QID(32b):» Reserved (8b)» QM ID (3b)» QID(17b): 1M queues per QM

Input Hlpr would use QM ID to select Scratch ring on which to put request.

QM/Sched then sends on its output NN/scratch ring to its associated Tx With 64 entries in Q-Array and 16 entries in CAM, max number of

QM/Schds is probably 4 (2 bits). » We’ll set aside 3 bits to give us flexibility in the future.

Tx

QID(20b)

IP Pkt Length (16b)

Buffer Handle(32b)

Rsv(4b)

Reserved (16b)

Port(8b)

Buffer Handle(32b)

Port(8b)Reserved (24b)

32 - JDD - 04/19/23

Packet Buffer Descriptor Tradeoffs Why use a Buffer Descriptor at all?

» QM needs something to link packets/buffers in queues» ME-to-ME communications costs vs. SRAM access costs

33 - JDD - 04/19/23

Packet Buffer Descriptor def Meta Data structure of Packet Buffers (LSB to MSB)

» buffer_next 32 bits Next Buffer Pointer (in a chain of buffers)» offset 16 bits Offset to start of data in bytes» BufferSize 16 bits Length of data in the current buffer in bytes» header_type 8 bits type of header at offset bytes in to the buffer» rx_stat 4 bits Receive status flags» free_list 4 bits Freelist ID» packet_size 16 bits (Total packet size across multiple buffers)» output_port 16 bits Output Port on the egress processor» input_port 16 bits Input Port on the ingress processor» nhid_type 4 bits Nexthop ID type.» reserved 4 bits Reserved» fabric_port 8 bits Output port for fabric indicating blade ID.» nexthop_id 16 bits NextHop IP ID» color 8 bits Qos Color » flow_id 24 bits QOS flow ID or MPLS label/flow id» reserved 16 bits Reserved» class_id 16 bits Class ID» packet_next 32 bits pointer to next packet (unused in cell mode)

34 - JDD - 04/19/23

Packet Buffer Descriptor Gets buffer_next: tx Offset: rx, tx, fwd BufferSize: tx, fwd header_type: tx, fwd rx_stat: NONE free_listpacket_size: NONE output_port: qm(?), tx input_port: rx, fwd nhid_type: NONE fabric_port: qm(?), tx nexthop_id color flow_id class_id packet_next

35 - JDD - 04/19/23

Meta Data Caching Meta Data can be cached in one of three places:

»SRAM Xfer Registers»DRAM Xfer Registers»GPR Registers

Size of Meta Data Cache is controlled by #define META_CACHE_SIZE

Macro dl_meta_load_cache[] loads meta data cache»buffer_handle: buffer handle for which meta data is to be fetched»dl_meta: read transfer register prefix

Xbuf_alloc[] should be used to allocate the needed registers»signal_number:»START_LW: starting long word for fetch»NUM_LW: number of long words to fetch

Each microengine (microblock?) can use Meta Data Caching differently.

36 - JDD - 04/19/23

Meta Data Caching In the ipv4_v6_forwarder sample app,

» dl_meta_load_cache() used in: Egress

ethernet_arp.uc pkt_tx_16p.uc statistics_util.uc tx_helper.uc

Ingress ethernet_arp.uc pkt_tx_16p.uc statistics_util.uc tx_helper.uc

» dl_meta_get_*[] used in: Egress

ethernet_arp.uc pkt_tx_16p.uc tx_helper.uc

Ingress Ether.uc Ipv4_fwder.uc Ipv4_fwder_util.uc Ipv6_fwder.uc V6v4_tunnel_decap.uc V6v4_tunnel_encap.uc pkt_tx_16p.uc tx_helper.uc

» dl_meta_set_*[] used in: Egress

ethernet_arp.uc pkt_rx_init.uc pkt_rx_two_me_util.uc

Ingress pkt_rx_init.uc pkt_rx_two_me_util.uc Ether.uc Ipv4_fwder_util.uc Ipv6_fwder.uc V6v4_tunnel_decap.uc V6v4_tunnel_encap.uc pkt_tx_16p.uc tx_helper.uc

37 - JDD - 04/19/23

Buffer Handle

38 - JDD - 04/19/23

Buffer Descriptor Usage Is there a different Buffer Descriptor defn for LC and PE? Will we support Multi-Buffer Packets?

» If not, we do not need buffer_next(32b) or buffer_size(16b) QM uses packet_next for its packet chaining in qarray. Output Port and Input Port probably translate to TxMI and RxMI Next Hop fields (nhid_type(4b) and nexthop_id(16b)) probably can

go away. QOS fields (color(8b) and flow_id(24b)) probably can go away. Two reserved fields 4b and 16b can go away. class_id(16b) (virtual queue id?) can probably go away. fabric_port can probably go away.

39 - JDD - 04/19/23

Buffer Descriptor Usage PE Buffer Descriptor:

» MR_ID (16b)» TxMI (16b)» VLAN (16b)

» buffer_next 32 bits Next Buffer Pointer (in a chain of buffers)» offset 16 bits Offset to start of data in bytes» BufferSize 16 bits Length of data in the current buffer in bytes

header_type 8 bits type of header at offset bytes in to the buffer rx_stat 4 bits Receive status flags

» free_list 4 bits Freelist ID» packet_size 16 bits (Total packet size across multiple buffers)

output_port 16 bits Output Port on the egress processor input_port 16 bits Input Port on the ingress processor nhid_type 4 bits Nexthop ID type. reserved 4 bits Reserved fabric_port 8 bits Output port for fabric indicating blade ID. nexthop_id 16 bits NextHop IP ID color 8 bits Qos Color flow_id 24 bits QOS flow ID or MPLS label/flow id reserved 16 bits Reserved class_id 16 bits Class ID

» packet_next 32 bits pointer to next packet (unused in cell mode)

40 - JDD - 04/19/23

Buffer Descriptor Usage PE Buffer Descriptor:

» LW0: buffer_next 32 bits Next Buffer Pointer (in a chain of buffers)» LW1: offset 16 bits Offset to start of data in bytes» LW1: BufferSize 16 bits Length of data in the current buffer in bytes» LW2: reserved 8 bits reserved/unused» LW2: reserved 4 bits reserved/unused» LW2: free_list 4 bits Freelist ID» LW2: packet_size 16 bits (Total packet size across multiple buffers)» LW3: MR_ID 16 bits Meta Router ID» LW3: TxMI 16 bits Transmit Meta Interface» LW4: VLAN 16 bits VLAN» LW4: reserved 16 bits reserved/unused» LW5: reserved 32 bits reserved/unused» LW6: reserved 32 bits reserved/unused» LW7: packet_next 32 bits pointer to next packet (unused in cell mode)

Leave multi-buffer fields there as a template for the dedicated blade implementation of a jumbo-frame MR.» Also reduces changes to Rx, Tx, and QM and reduces potential problems.

41 - JDD - 04/19/23

Multicast Alternatives At least Three Options

»Force MRs that need Multicast to be Dedicated Blade MRs and do their own Multicast

For our short term goals this is probably sufficient and the best course. Perhaps longer term we can look at adding it to the CRF

»Treat as exception and send to Xscale»Provide support in CRF for Multicast

Use Multi-Hit Lookup capability of the TCAM MI Bit mask defined in Lookup Result

Will put a bound on the number of MIs that can be supported on an MR because of the size of the lookup result.

Has issues of mapping bits in the bit mask to actual MIs. Lookup Result contains an index into a table containing MI bit masks Allow but do not force MRs to provide code to interpret Lookup Result.

– This would also allow other possible extensions on an MR-specific basis– This carries with it the problem of bounding the execution time of the MR-specific

code in the Lookup block. For general multicast, this could be a serious issue.– There are also issues with generating a QID based on an MI when the QID is not

included in the Lookup Result.

»Other options?

42 - JDD - 04/19/23

CRF Support for Multicast

HeaderFormat

MR-1

MR-n. . .

DRAM Buf Ptr

MR Id

Output MI

Buffer Offset

QID

DRAM Buf Ptr

MR Id

MR Lookup Key

MR Ctrl Blk Ptr

MR Mem Ptr

DRAM Buf Ptr

MR Id

Output MI

QID

MR SpecificLookup Result

Stats Index

MR Ctrl Blk Ptr

MR Mem Ptr

Parse

MR-1

MR-n. . .

DRAM Buf Ptr

MR Id

Input MI

MR Ctrl Blk Ptr

MR Mem Ptr

MRInterpLo

oku

p

Post

Pro

cess

Default/Unicast path

MR-SpecificPath

43 - JDD - 04/19/23


DRAM Buf Ptr

MR Id

MR Lookup Key

MR Ctrl Blk Ptr

MR Mem Ptr

MRInterpLo

oku

p

Post

Pro

cess

Default path

MR-SpecificPath

DRAM Buf Ptr

MR Id

Output MI

QID


Stats Index

MR Ctrl Blk Ptr

MR Mem Ptr

Copy Cnt

We will need some kind of copy count or multicast bit and last copy bit to let TX know when it can release the DRAM buffer that holds the packet.

DRAM Buf Ptr

MR Id

Output MI

QID


Stats Index

MR Ctrl Blk Ptr

MR Mem Ptr

Copy Cnt=1

44 - JDD - 04/19/23


DRAM Buf Ptr

MR Id

MR Lookup Key

MR Ctrl Blk Ptr

MR Mem Ptr

MRInterpLooku

p

Post

Pro

cess

Default path

MR-SpecificPath

DRAM Buf Ptr

MR Ctrl Blk Ptr

MR Mem Ptr


MR Lookup Key Output MI

Copy Cnt

Output MI

Copy Cnt

Output MI

Copy Cnt

DRAM Buf Ptr

MR Id

Output MI

QID


Stats Index

MR Ctrl Blk Ptr

MR Mem Ptr

Copy Cnt

DRAM Buf Ptr

MR Id

Output MI

QID


Stats Index

MR Ctrl Blk Ptr

MR Mem Ptr

Copy Cnt

DRAM Buf Ptr

MR Id

Output MI

QID


Stats Index

MR Ctrl Blk Ptr

MR Mem Ptr

Copy Cnt

We will need some kind of copy count or multicast bit and last copy bit to let TX know when it can release the DRAM buffer that holds the packet.

45 - JDD - 04/19/23

OLD The rest of these are old slides that should be deleted

at some point.

46 - JDD - 04/19/23

Common Router Framework (CRF) Functional Blocks

Parse

Lookup

HeaderFormat

DeMuxRx TxQMMR-1

MR-n. . .

MR-1

MR-n. . .

RBUF

Rx:»Function

Coordinate transfer of packets from RBUF to DRAM»Notes:

We’ll pass the Buffer Handle which contains the SRAM address of the buffer descriptor.

From the SRAM address of the descriptor we can calculate the DRAM address of the buffer data.

Buf Handle(32b)

VLAN

Packet_Next

MR_ID

TxMI

Free_List

Packet_Size

Buffer_Next

Offset

BufferDescriptor

Buffer_Size

47 - JDD - 04/19/23


Parse

Lookup

HeaderFormat

DeMuxRx TxQMMR-1

MR-n. . .

MR-1

MR-n. . .

DeMux» Function

Read Pkt Header from DRAM Use VLAN from Ethernet header to determine destination MR in order to locate:

MR Parse code MR specific memory pointers

Write MR Id to Buffer Descriptor Write VLAN to Buffer Descriptor

VLAN

Packet_Next

MR_ID

TxMI

Free_List

Packet_Size

Buffer_Next

Offset

BufferDescriptor

Buffer_Size

MR Id(16b)

Input MI(16b)

MR Mem Ptr(32b)

Buf Handle(32b)

DRAM Buf Ptr(32b)

Buffer Offset(16b)

Buf Handle(32b)

48 - JDD - 04/19/23


Parse

Lookup

HeaderFormat

DeMuxRx TxQMMR-1

MR-n. . .

MR-1

MR-n. . .

Parse» Function

MR-specific header processing Generate MR-specific lookup key (16 Bytes) from packet

Need CRF functionality to managed multiple MRs in shared PE. Notes:

» Can Parse adjust the buffer/packet size and offset? » Can Parse do something like, terminate a tunnel and strip off an outer header?

VLAN

Packet_Next

MR_ID

TxMI

Free_List

Packet_Size

Buffer_Next

Offset

BufferDescriptor

Buffer_SizeMR Id(16b)

Input MI(16b)

MR Mem Ptr(32b)

Buf Handle(32b)

DRAM Buf Ptr(32b)

Buffer Offset(16b)

MR Lookup Key(16B)

MR Id(16b)

Input MI(16b)

MR Mem Ptr(32b)

Buf Handle(32b)

DRAM Buf Ptr(32b)

Buffer Offset(16b)

49 - JDD - 04/19/23

CRF Wrapper Around Parse

Parse

MR-1

MR-n

. . .

MRSelector

MR Id

Input MI

MR Mem Ptr

Buf Handle(32b)

DRAM Buf Ptr

Buffer Offset

DRAM Buf Ptr

Input MI

MR Mem Ptr

Buffer Offset

MR Lookup Key

Buffer Offset

MR Lookup Key

MR Id

Input MI

MR Mem Ptr

Buf Handle(32b)

DRAM Buf Ptr

Buffer Offset

50 - JDD - 04/19/23


Parse

Lookup

HeaderFormat

DeMuxRx TxQMMR-1

MR-n. . .

MR-1

MR-n. . .

Lookup»Function

Perform lookup in TCAM based on MR Id and lookup key

»Result: Output MI QID Stats index MR-specific Lookup

Result (flags, etc. ?) How wide can/should

this be?

VLAN

Packet_Next

MR_ID

TxMI

Free_List

Packet_Size

Buffer_Next

Offset

BufferDescriptor

Buffer_Size

MR Lookup Key(16B)

MR Id(16b)

Input MI(16b)

MR Mem Ptr(32b)

Buf Handle(32b)

DRAM Buf Ptr(32b)

Buffer Offset(16b)

Buffer Handle(32b)

MR Id(16b)

Lookup Result(Nb)

MR Mem Ptr(32b)

DRAM Buf Ptr(32b)

Buffer Offset(16b)

51 - JDD - 04/19/23


Parse

Lookup

HeaderFormat

DeMuxRx TxQMMR-1

MR-n. . .

MR-1

MR-n. . .

Buffer Handle(32b)

QID(16b) Header Format» Function

MR specific packet header formatting

MR specific Lookup Result processing

Drop and Miss bits Need CRF functionality

to managed multiple MRs in shared PE.» Pulls out QID, Length

and Port from MR Result, etc.

» Checks for Drop and Miss bits and deals with those actions.

VLAN

Packet_Next

MR_ID

TxMI

Free_List

Packet_Size

Buffer_Next

Offset

BufferDescriptor

Buffer_Size

Buffer Handle(32b)

MR Id(16b)

Lookup Result(Nb)

MR Mem Ptr(32b)

DRAM Buf Ptr(32b)

Buffer Offset(16b) Size (16b)

Port(8b)

Includes dropand miss bits

52 - JDD - 04/19/23

CRF Wrapper Around Header Format

HeaderFormat

MR-1

MR-n. . .

Buffer Offset

MRSelector

Gets written to Buffer DescriptorMay also cause size(s) in Descriptor to be updated.

(what about trimming data,What if it is a buffer’s worth

Which would change the chaining,Can they add/trim at either end?

Buffer Handle

MR Id

Lookup Result

MR Mem Ptr

DRAM Buf Ptr(32b)

Buffer Offset

DRAM Buf Ptr

Output MI


MR Mem Ptr

Buffer Offset

Buffer Handle

QID

Size

Port

53 - JDD - 04/19/23


Parse

Lookup

HeaderFormat

DeMuxRx TxQMMR-1

MR-n. . .

MR-1

MR-n. . .

QM»Function

CRF queue management for Meta Interface queues

For performance reasons, QM may actually be implemented as multiple instances»Each instance on a separate ME

would support a separate set of Meta Interfaces.

»See next slide for more details…

Buffer Handle(32b)

QID(16b)

VLAN

Packet_Next

MR_ID

TxMI

Free_List

Packet_Size

Buffer_Next

Offset

BufferDescriptor

Buffer_Size

Buf Handle(32b)

Size (16b)

Port(8b)

54 - JDD - 04/19/23

QM/Scheduler on Multiple MEs

OutputHlpr

(1 ME)

QM/Schd(1 ME)

QM/Schd(1 ME)

Scratch Rings

InputHlpr

(1 ME)

NN Ring NN Ring

. . .

HeaderFormat Tx

Buffer Handle(32b)

QID(32b) Buf Handle(32b)

MR-1

MR-n. . .

QID(32b):» Reserved (8b)» QM ID (4b)» QID(20b): 1M queues per QM

Input Hlpr would use QM ID to select Scratch ring on which to put request. Output Hlpr would process all Scratch rings coming from QM/Schd MEs and

multiplex onto one NN ring to TX With 64 entries in Q-Array and 16 entries in CAM, max number of QM/Schds is

probably 4 (2 bits). » We’ll set aside 4 bits to give us flexibility in the future.

Size (16b)

Port(8b)

55 - JDD - 04/19/23


Parse

Lookup

HeaderFormat

DeMuxRx TxQMMR-1

MR-n. . .

MR-1

MR-n. . .

TBUF

Tx»Function

Coordinate transfer of packets from DRAM to TBUF

Buffer Handle(32b)

VLAN

Packet_Next

MR_ID

TxMI

Free_List

Packet_Size

Buffer_Next

Offset

BufferDescriptor

Buffer_Size

56 - JDD - 04/19/23

Old Template

LookupDeMuxRx TxQM

TBUF

Tx»Function

Coordinate transfer of packets from DRAM to TBUF

VLAN

Packet_Next

MR_ID

TxMI

Free_List

Packet_Size

Buffer_Next

Offset

BufferDescriptor

Buffer_Size

HeaderFormatParse

Buffer Handle(32b)

Port(8b)Reserved (24b)

57 - JDD - 04/19/23

Old Rejected Overly Busy Slide

LookupRx TxQMMR

ParseMR HdrFormat

SubstrDecap

SubstrEncap

L1 L2L3 L3 L3

L2 L2 L1

BlockInput Data

Input Data

VLANPacket_Next

MR_IDTxMI

Free_ListPacket_Size

Buffer_Next

Offset

BufferDescriptor

Buffer_Size

Output Data

Output Data

Logical interface» Data passing between

layers

Notes here

Physical format» Actual Format of data » Shows type of

communication Scratch Ring NN Ring SRAM Rings

Buf Descriptor» shows fields

read/written

Design of a Diversified Router: Dedicated CRF for IPv4 Metarouter

Documents

Transcript of Design of a Diversified Router: Dedicated CRF for IPv4 Metarouter