Cavium Networks OCTEON Plus CN50XX Hardware ... 2008 Cavium Networks OCTEON Plus CN50XX Hardware...

Cavium Networks OCTEON Plus CN50XXHardware Reference Manual

Contents of this document are subject to change without notice.

The exact features and specifications may change prior to the V1.0 manual revision.

CN50XX-HM-0.99E PRELIMINARY

Cavium Networks Proprietary and Confidential DO NOT COPY

July 2008

PUBLISHED BYCavium Networks805 East Middlefield RoadMountain View, CA 94043Phone: 650-623-7000Fax: 650-625-9751Email: [email protected]: http://www.caviumnetworks.com

2003-2008 by Cavium Networks

All rights reserved. No part of this manual may be reproduced in any form, or transmitted by any means, without the written permission of Cavium Networks.

Cavium Networks makes no warranty about the use of its products, and reserves the right to change this document at any time, without notice. Whereas great care has been taken in the preparation of this manual, Cavium Networks, the publisher, and the authors assume no responsibility for errors or omissions.

OCTEON is a trademark of Cavium Networks.

MIPS and MIPS64 are registered trademarks of MIPS Technologies. cnMIPS is a trademark of MIPS Technologies; Cavium is a licensee of cnMIPS.

All other trademarks or service marks referred to in this manual are the property of their respective owners.

Table of Contents

Preface............................................................................................................. 33Chapter 1 Introduction ............................................................................................................... 39

OCTEON Plus CN50XX ........................................................................................................... 39Overview .................................................................................................................................... 40

1.1 Principles of Operation ............................................................................................ 431.1.1 CPU Cores ....................................................................................................... 431.1.2 Coherent Multicore and I/O L2/DRAM Sharing ........................................... 431.1.3 Core Partitioning ............................................................................................ 431.1.4 Flexible Packet/Control Interfacing............................................................... 431.1.5 In-line Packet-Processing Hardware Acceleration ....................................... 441.1.6 Hardware-Assisted Dynamic Memory Allocation/Deallocation ................... 441.1.7 Hardware Work Queuing, Scheduling, Ordering, and Synchronization ..... 441.1.8 Essential Quality of Service (QoS) Functions Implemented in Hardware.. 451.1.9 Security Features............................................................................................ 451.1.10 Coprocessor Accelerators................................................................................ 461.1.11 Debug Support ................................................................................................ 46

1.2 CN50XX System Applications ................................................................................. 46

1.3 Remaining Chapters ................................................................................................ 471.3.1 Coherent Memory Bus (CMB), Level-Two Cache Controller (L2C),

and DRAM Controller..................................................................................... 471.3.2 I/O Bus and I/O Bridge ................................................................................... 471.3.3 CPU Cores ....................................................................................................... 471.3.4 Packet Order / Work Unit (POW) .................................................................. 471.3.5 Free Pool Unit (FPA) ...................................................................................... 471.3.6 Packet Input Processing/Input Packet Data Unit (PIP/IPD) ....................... 471.3.7 Packet Output Unit (PKO)............................................................................. 481.3.8 PCI Unit .......................................................................................................... 481.3.9 Timer Unit (TIM)............................................................................................ 481.3.10 Central Interrupt Unit (CIU)......................................................................... 481.3.11 Boot Bus Unit.................................................................................................. 481.3.12 RGMII/GMII/MII Unit (GMX) ....................................................................... 481.3.13 TDM/PCM Unit............................................................................................... 481.3.14 GPIO Unit ....................................................................................................... 491.3.15 UART Unit ...................................................................................................... 491.3.16 TWSI Unit ....................................................................................................... 491.3.17 System Management Interface (SMI)............................................................ 491.3.18 Random Number Generator (RNG/RNM) ..................................................... 491.3.19 SPI/MPI Unit .................................................................................................. 491.3.20 USB Unit......................................................................................................... 491.3.21 Electrical Specifications ................................................................................. 491.3.22 AC Characteristics.......................................................................................... 491.3.23 Mechanical Specifications .............................................................................. 491.3.24 Signal Descriptions......................................................................................... 501.3.25 Ball Assignments ............................................................................................ 50

1.4 Configuration and Status Registers (CSRs) ............................................................ 501.4.1 CSR Field Types ............................................................................................. 52

Chapter 2 Coherent Memory Bus, Level-2 Cache Controller, DRAM Controller ................... 532.1 Coherent Memory Bus (CMB) ................................................................................. 54

CN50XX-HM-0.99E PRELIMINARY Cavium Networks Proprietary and Confidential DO NOT COPY 3

Table of Contents

2.1.1 CMB Overview ................................................................................................ 542.1.2 CMB Buses ...................................................................................................... 542.1.3 CMB Description............................................................................................. 542.1.4 CMB Memory Coherence Support ................................................................. 552.1.5 CMB Transactions .......................................................................................... 58

2.2 Level-2 Cache Controller (L2C) ............................................................................... 602.2.1 L2 Cache and Data Store ............................................................................... 602.2.2 L2C Memory Coherence ................................................................................. 612.2.3 L2 Cache Indexing (Set Selection) ................................................................. 622.2.4 L2 Cache Replacement and Way-Partitioning .............................................. 632.2.5 L2 Cache-Block Locking ................................................................................. 642.2.6 Cache-Block Flush and Unlocking................................................................. 652.2.7 Memory Input Queue Arbitration.................................................................. 662.2.8 COMMIT and FILL Bus Arbitration ............................................................. 662.2.9 L2C ECC Codes ............................................................................................. 67

2.3 DRAM Controller (LMC) ......................................................................................... 682.3.1 Main Memory DRAM Addressing.................................................................. 712.3.2 DRAM Part Addressing.................................................................................. 712.3.3 DRAM Transaction Examples........................................................................ 722.3.4 DRAM Programming ...................................................................................... 782.3.5 DRAM Refreshes............................................................................................. 782.3.6 DRAM Scheduler Performance ...................................................................... 782.3.7 DRAM Chip Selects and ODT ........................................................................ 792.3.8 DRAM Controller Initialization ..................................................................... 802.3.9 DDR Clock-Speed Programming Tables........................................................ 832.3.10 DRAM ECC Codes ......................................................................................... 83

2.4 L2C Registers ........................................................................................................... 84L2C_CFG ........................................................................................................ 85L2T_ERR ........................................................................................................ 86L2D_ERR ........................................................................................................ 87L2D_FADR ..................................................................................................... 87L2D_FSYN0 ................................................................................................... 88L2D_FSYN1 ................................................................................................... 88L2C_DBG ....................................................................................................... 89L2C_LFB0 ...................................................................................................... 90L2C_LFB1 ...................................................................................................... 91L2C_LFB2 ...................................................................................................... 91L2C_LFB3 ...................................................................................................... 91L2C_DUT ........................................................................................................ 92L2C_LCKBASE .............................................................................................. 93L2C_LCKOFF ................................................................................................ 94L2C_SPAR0 .................................................................................................... 94L2C_SPAR4 .................................................................................................... 94L2C_PFCTL ................................................................................................... 95L2C_PFC(0..3) ................................................................................................ 97L2D_BST0 ...................................................................................................... 98L2D_BST1 ...................................................................................................... 98L2D_BST2 ...................................................................................................... 99L2D_BST3 ...................................................................................................... 99L2D_FUS0 ...................................................................................................... 100L2D_FUS1 ...................................................................................................... 100L2D_FUS2 ...................................................................................................... 101L2D_FUS3 ...................................................................................................... 101L2C_BST0 ...................................................................................................... 103L2C_BST1 ...................................................................................................... 103L2C_BST2 ...................................................................................................... 104

2.5 LMC Registers ......................................................................................................... 105

4 Cavium Networks Proprietary and Confidential DO NOT COPY CN50XX-HM-0.99E PRELIMINARY

Table of Contents

LMC_MEM_CFG0 ......................................................................................... 106LMC_MEM_CFG1 ......................................................................................... 109LMC_CTL ....................................................................................................... 111LMC_DDR2_CTL ........................................................................................... 113LMC_FADR .................................................................................................... 115LMC_COMP_CTL .......................................................................................... 115LMC_WODT_CTL .......................................................................................... 116LMC_ECC_SYND .......................................................................................... 117LMC_IFB_CNT_LO ....................................................................................... 117LMC_IFB_CNT_HI ........................................................................................ 118LMC_OPS_CNT_LO ...................................................................................... 118LMC_OPS_CNT_HI ....................................................................................... 118LMC_DCLK_CNT_LO ................................................................................... 118LMC_DCLK_CNT_HI .................................................................................... 119LMC_RODT_CTL .......................................................................................... 119LMC_DELAY_CFG ........................................................................................ 119LMC_CTL1 ..................................................................................................... 120LMC_DUAL_MEMCFG ................................................................................. 121LMC_RODT_COMP_CTL ............................................................................. 123LMC_PLL_CTL .............................................................................................. 123LMC_PLL_STATUS ...................................................................................... 124LMC_BIST_CTL ............................................................................................ 124LMC_BIST_RESULT ..................................................................................... 124

Chapter 3 I/O Busing, I/O Bridge (IOB) andFetch and Add Unit (FAU) ......................................................................................... 125

3.1 CN50XX I/O Busing ................................................................................................. 1263.1.1 I/O Busing Overview....................................................................................... 1263.1.2 I/O Bus Flow Examples .................................................................................. 127

3.2 IOB Architecture ...................................................................................................... 1293.2.1 IOB Architecture Overview............................................................................ 129

3.3 Dont-Write-Back Engine ......................................................................................... 130

3.4 Fetch and Add Unit (FAU) ....................................................................................... 130

3.5 Fetch-and-Add Operations ....................................................................................... 1323.5.1 Load Operations.............................................................................................. 1323.5.2 IOBDMA Operations ...................................................................................... 1343.5.3 Store Operations ............................................................................................. 136

3.6 IOB Registers ........................................................................................................... 137IOB_FAU_TIMEOUT .................................................................................... 138IOB_CTL_STATUS ........................................................................................ 138IOB_INT_SUM ............................................................................................... 138IOB_INT_ENB ............................................................................................... 139IOB_PKT_ERR ............................................................................................... 139IOB_INB_DATA_MATCH ............................................................................. 139IOB_INB_CONTROL_MATCH ..................................................................... 140IOB_INB_DATA_MATCH_ENB ................................................................... 140IOB_INB_CONTROL_MATCH_ENB ........................................................... 140IOB_OUTB_DATA_MATCH ......................................................................... 140IOB_OUTB_CONTROL_MATCH ................................................................. 141IOB_OUTB_DATA_MATCH_ENB ............................................................... 141IOB_OUTB_CONTROL_MATCH_ENB ....................................................... 141IOB_BIST_STATUS ...................................................................................... 142

Chapter 4 cnMIPS Cores ......................................................................................................... 143Overview .................................................................................................................................... 144

4.1 Summary of cnMIPS Core Features ........................................................................ 1444.1.1 MIPS64 Version 2.0 Implementation ............................................................ 1444.1.2 Cavium-Specific Architectural Additions ...................................................... 145


Table of Contents

4.1.3 Full Privileged Architecture (i.e. Coprocessor 0) Support ............................ 1464.1.4 Full EJTAG Version 3.10 Support ................................................................. 147

4.2 cnMIPS Core Non-Privileged State ......................................................................... 148

4.3 Cavium-Specific Instruction Summary ................................................................... 149

4.4 cnMIPS Core Instruction Set Summary ................................................................. 151

4.5 cnMIPS Core Virtual Addresses and CVMSEG ..................................................... 156

4.6 Physical Addresses ................................................................................................... 157

4.7 IOBDMA Operations ................................................................................................ 160

4.8 cnMIPS Core-Memory Reference Ordering ............................................................. 161

4.9 cnMIPS Core CSR Ordering .................................................................................... 162

4.10 cnMIPS Core Write Buffer ....................................................................................... 163

4.11 cnMIPS Core Coprocessor 0 Privileged Registers .................................................. 165Index Register ............................................................................................... 167Random Register ........................................................................................... 167EntryLo0, EntryLo1 Registers ..................................................................... 167 Context Register .......................................................................................... 168PageMask Register ........................................................................................ 168PageGrain Register ....................................................................................... 168Wired Register ............................................................................................... 169HWREna Register .......................................................................................... 169BadVAddr Register ........................................................................................ 169Count Register ............................................................................................... 169EntryHi Register ............................................................................................ 169Compare Register .......................................................................................... 170Status Register ............................................................................................... 170IntCtl Register ............................................................................................... 171SRSCtl Register ............................................................................................. 171Cause Register ............................................................................................... 171Exception Program Counter .......................................................................... 172PRId Register ................................................................................................. 172EBase Register ............................................................................................... 172Config Register ............................................................................................... 172Config1 Register ............................................................................................. 173Config2 Register ............................................................................................. 173Config3 Register ............................................................................................. 174WatchLo Register ........................................................................................... 174WatchHi Register ........................................................................................... 174XContext Register .......................................................................................... 175Debug Register ............................................................................................... 175Debug Exception Program Counter Register ............................................... 176Performance Counter Control Register ........................................................ 176Performance Counter Counter Register ....................................................... 178ErrorEPC ........................................................................................................ 178DESAVE Register .......................................................................................... 178

4.11.1 Cavium Networks-Specific Coprocessor 0 Registers..................................... 179CacheErr (Icache) .......................................................................................... 179CacheErr (Dcache) ......................................................................................... 179TagLo Register (Icache) ................................................................................. 180TagLo Register (Dcache) ................................................................................ 180DataLo Register (Icache) ............................................................................... 180DataLo Register (Dcache) .............................................................................. 181TagHi Register .............................................................................................. 181DataHi Register (Icache) ............................................................................... 181DataHi Register (Dcache) .............................................................................. 182CvmCtl Register ............................................................................................. 182CvmMemCtl Register .................................................................................... 184CvmCount Register ........................................................................................ 185


Table of Contents

Multicore Debug Register .............................................................................. 186

4.12 cnMIPS Core EJTAG DRSEG Registers ............................................................. 186Debug Control Register (DCR) ...................................................................... 187Instruction Breakpoint Status (IBS) Register ............................................. 187Instruction Breakpoint Address (IBA0...3) Register .................................... 187Instruction Breakpoint Address Mask (IBM0...3) Register ......................... 187Instruction Breakpoint ASID (IBASID0...3) Register ................................. 188Instruction Breakpoint Control (IBC0...3) Register ..................................... 188Data Breakpoint Status (DBS) Register ....................................................... 188Data Breakpoint Address (DBA0...3) Register ............................................. 188Data Breakpoint Address Mask (DBM0...3) Register .................................. 189Data Breakpoint ASID (DBASID0...3) Register ........................................... 189Data Breakpoint Control (DBC0...3) Register .............................................. 189Data Breakpoint Value (DBV0...3) Register ................................................ 189

4.13 cnMIPS Core EJTAG TAP Registers ................................................................... 190Device ID Register Format ............................................................................ 190Implementation Register Format (TAP Instruction IMPCODE) ................ 190Data Register (TAP Instruction DATA, ALL, or FASTDATA) .................... 191Address Register (TAP Instruction ADDRESS or ALL) .............................. 191EJTAG Control Register (ECR) (TAP Instruction CONTROL or ALL) ...... 191PC Sample Register Format (TAP Instruction PCSAMPLE) ...................... 191EJTAG Boot Indication ................................................................................. 192Bypass Register .............................................................................................. 192Fastdata Register ........................................................................................... 192

4.14 cnMIPS Core Pipelines ............................................................................................. 193

4.15 Special MUL Topics .................................................................................................. 194

4.16 COP2 Latencies ........................................................................................................ 196

4.17 cnMIPS Core Hardware Debug Features ................................................................ 1974.17.1 Multicore Debug Support ............................................................................... 1984.17.2 System Debug Characteristics ....................................................................... 199

4.18 cnMIPS Core Load-Linked / Store-Conditional ...................................................... 200

4.19 cnMIPS Core Exceptions .......................................................................................... 200

Chapter 5 Packet Order / Work Unit (POW) .............................................................................. 205Overview .................................................................................................................................... 206

5.1 POW Work Flow, Operations, and Ordering .......................................................... 207

5.2 Software Architecture Example ............................................................................... 2135.2.1 Defragmentation ............................................................................................. 2165.2.2 IPSEC Decryption........................................................................................... 2165.2.3 Lookup ............................................................................................................. 2165.2.4 Process............................................................................................................. 2175.2.5 IPSEC Encrypt................................................................................................ 2175.2.6 Output Queue.................................................................................................. 217

5.3 POW Internal Architecture ................................................................................... 217

5.4 Work-Queue Entry Format ...................................................................................... 220

5.5 Core and Fetch-and-Add Pending Switch Bits ....................................................... 221

5.6 POW Interrupts ........................................................................................................ 222

5.7 POW QOS Features .................................................................................................. 2255.7.1 Thresholds....................................................................................................... 2255.7.2 Scheduling....................................................................................................... 225

5.8 POW Debug Visibility .............................................................................................. 227

5.9 POW Performance Considerations .......................................................................... 228

5.10 Forward Progress Constraints ................................................................................. 229

5.11 POW Operations ....................................................................................................... 231


Table of Contents

5.11.1 Load Operations.............................................................................................. 2315.11.2 IOBDMA Operations ...................................................................................... 2385.11.3 Store Operations ............................................................................................. 238

5.12 POW ECC Codes ...................................................................................................... 239

5.13 POW Registers ......................................................................................................... 240POW_PP_GRP_MSK0/1 ................................................................................ 241POW_WQ_INT_THR(0..15) ........................................................................... 241POW_WQ_INT_CNT(0..15) ........................................................................... 243POW_QOS_THR(0..7) .................................................................................... 243POW_QOS_RND(0...7) ................................................................................... 245POW_WQ_INT ............................................................................................... 245POW_WQ_INT_PC ........................................................................................ 246POW_NW_TIM .............................................................................................. 246POW_ECC_ERR ............................................................................................. 248POW_NOS_CNT ............................................................................................ 249POW_PF_RST_MSK ...................................................................................... 249POW_WS_PC(0..15) ....................................................................................... 249POW_WA_PC(0..7) ......................................................................................... 249POW_IQ_CNT(0..7) ........................................................................................ 249POW_WA_COM_PC ...................................................................................... 250POW_IQ_COM_CNT ..................................................................................... 250POW_TS_PC .................................................................................................. 250POW_DS_PC .................................................................................................. 250POW_BIST_STAT .......................................................................................... 251

Chapter 6 Free Pool Unit (FPA) ................................................................................................. 253Overview .................................................................................................................................... 254

6.1 Free Pool Unit Operations ....................................................................................... 2566.1.1 Load Operations.............................................................................................. 2566.1.2 IOBDMA Operations ...................................................................................... 2576.1.3 Store Operations ............................................................................................. 257

6.2 FPA Registers .......................................................................................................... 258FPA_INT_SUM .............................................................................................. 259FPA_INT_ENB ............................................................................................... 260 FPA_CTL_STATUS ...................................................................................... 261FPA_QUE(0..7)_AVAILABLE ....................................................................... 261FPA_BIST_STATUS ...................................................................................... 262FPA_QUE(0..7)_PAGE_INDEX .................................................................... 262FPA_QUE_EXP .............................................................................................. 262FPA_QUE_ACT .............................................................................................. 263

Chapter 7 Packet Input Processing/Input Packet Data Unit (PIP/IPD) ................................... 265Overview .................................................................................................................................... 266

7.1 Input Ports ................................................................................................................ 266

7.2 Input Packet Formats and Pre-IP Parsing ............................................................. 2667.2.1 Packet Instruction Header ............................................................................. 2687.2.2 PCI Instruction-to-Packet Conversion........................................................... 2707.2.3 Parse Mode and Skip Length Selection ......................................................... 2717.2.4 PIP/IPD L2 Parsing and Is_IP Determination.............................................. 2727.2.5 Pre-IP Parsing Summary ............................................................................... 2727.2.6 Packet Input CRC........................................................................................... 2747.2.7 Packet Length Checks .................................................................................... 2747.2.8 Legal SKIP Values.......................................................................................... 276

7.3 Packet Buffering ....................................................................................................... 277

7.4 Packet Scheduling .................................................................................................... 2827.4.1 RAWFULL and RAWSCHED Packets ......................................................... 2827.4.2 QOS.................................................................................................................. 282


Table of Contents

7.4.3 Grp ................................................................................................................... 2827.4.4 TT..................................................................................................................... 2837.4.5 Tag ................................................................................................................... 283

7.5 Work-Queue Entry ................................................................................................... 284

7.6 Input Packet Data Unit (IPD) Quality of Service ................................................... 301

7.7 PIP/IPD Per-QOS Admission Control ..................................................................... 303

7.8 PIP Registers ............................................................................................................ 306PIP_BIST_STATUS ....................................................................................... 307PIP_INT_REG ................................................................................................ 308PIP_INT_EN .................................................................................................. 309PIP_STAT_CTL ............................................................................................. 309PIP_GBL_CTL ............................................................................................... 310PIP_GBL_CFG ............................................................................................... 311PIP_SFT_RST ................................................................................................ 312PIP_IP_OFFSET ............................................................................................ 313PIP_TAG_SECRET ........................................................................................ 313PIP_TAG_MASK ............................................................................................ 314PIP_TODO_ENTRY ....................................................................................... 314PIP_DEC_IPSEC(0..3) ................................................................................... 314PIP_RAW_WORD .......................................................................................... 314PIP_QOS_VLAN(0..7) .................................................................................... 315PIP_QOS_WATCH(0..7) ................................................................................ 315PIP_FRM_LEN_CHK0/1 ............................................................................... 315PIP_PRT_CFG(0..2, 32/33) ............................................................................ 316PIP_PRT_TAG(0..2, 32/33) ............................................................................ 317PIP_QOS_DIFF(0..63) ................................................................................... 318PIP_TAG_INC(0..63) ..................................................................................... 318

7.8.1 PIP Statistics Counters .................................................................................. 319PIP_STAT0_PRT(0..2, 32/33) ........................................................................ 319PIP_STAT1_PRT(0..2, 32/33) ........................................................................ 319PIP_STAT2_PRT(0..2, 32/33) ........................................................................ 319PIP_STAT3_PRT(0..2, 32/33) ........................................................................ 320PIP_STAT4_PRT(0..2, 32/33) ........................................................................ 320PIP_STAT5_PRT(0..2, 32/33) ........................................................................ 320PIP_STAT6_PRT(0..2, 32/33) ........................................................................ 320PIP_STAT7_PRT(0..2, 32/33) ........................................................................ 320PIP_STAT8_PRT(0..2, 32/33) ........................................................................ 321PIP_STAT9_PRT(0..2, 32/33) ........................................................................ 321

7.8.2 PIP Inbound Statistics Registers ................................................................... 322PIP_STAT_INB_PKTS(0..2, 32/33) ............................................................... 322PIP_STAT_INB_OCTS(0..2, 32/33) ............................................................... 322PIP_STAT_INB_ERRS(0..2, 32/33) ............................................................... 322

7.9 IPD Registers ............................................................................................................ 323IPD_1ST_MBUFF_SKIP ............................................................................... 324IPD_NOT_1ST_MBUFF_SKIP ..................................................................... 324IPD_PACKET_MBUFF_SIZE ....................................................................... 324IPD_CTL_STATUS ........................................................................................ 325IPD_WQE_FPA_QUEUE .............................................................................. 326IPD_PORT(0..2, 32/33)_BP_PAGE_CNT ...................................................... 326IPD_SUB_PORT_BP_PAGE_CNT ............................................................... 326IPD_1ST_NEXT_PTR_BACK ....................................................................... 327IPD_2ND_NEXT_PTR_BACK ...................................................................... 327IPD_INT_ENB ............................................................................................... 327IPD_INT_SUM ............................................................................................... 328IPD_SUB_PORT_FCS ................................................................................... 328IPD_QOS(0..7)_RED_MARKS ...................................................................... 328IPD_PORT_BP_COUNTERS_PAIR(0..2, 32/33) .......................................... 329IPD_RED_PORT_ENABLE .......................................................................... 329


Table of Contents

IPD_RED_QUE(0..7)_PARAM ...................................................................... 330IPD_PTR_COUNT ......................................................................................... 330IPD_BP_PRT_RED_END .............................................................................. 331IPD_QUE0_FREE_PAGE_CNT .................................................................... 331IPD_CLK_COUNT ......................................................................................... 331IPD_PWP_PTR_FIFO_CTL .......................................................................... 331IPD_PRC_HOLD_PTR_FIFO_CTL .............................................................. 332IPD_PRC_PORT_PTR_FIFO_CTL ............................................................... 332IPD_PKT_PTR_VALID .................................................................................. 332IPD_WQE_PTR_VALID ................................................................................ 333IPD_BIST_STATUS ....................................................................................... 333

Chapter 8 Packet Output Processing Unit (PKO) .................................................................... 335Overview .................................................................................................................................... 336

8.1 Output Ports ............................................................................................................. 337

8.2 Output Packet Format and TCP/UDP Checksum Insertion .................................. 338

8.3 PKO Output Queue .................................................................................................. 339

8.4 PKO Commands ....................................................................................................... 340

8.5 PKO Queue Arbitration Algorithm ......................................................................... 346

8.6 PKO Dont-Write-Back (DWB) Calculation ............................................................ 348

8.7 PKO Performance ..................................................................................................... 349

8.8 PKO Operations ........................................................................................................ 3498.8.1 Store Operations ............................................................................................. 349

8.9 PKO Registers .......................................................................................................... 351PKO_REG_FLAGS ........................................................................................ 352PKO_REG_READ_IDX .................................................................................. 352PKO_REG_CMD_BUF .................................................................................. 352PKO_REG_GMX_PORT_MODE ................................................................... 353PKO_REG_QUEUE_MODE .......................................................................... 353PKO_REG_BIST_RESULT ........................................................................... 353PKO_REG_ERROR ........................................................................................ 354PKO_REG_INT_MASK ................................................................................. 354PKO_REG_DEBUG0 ..................................................................................... 354PKO_REG_DEBUG1 ..................................................................................... 354PKO_REG_DEBUG2 ..................................................................................... 354PKO_REG_DEBUG3 ..................................................................................... 355PKO_REG_QUEUE_PTRS1 .......................................................................... 355PKO_MEM_QUEUE_PTRS .......................................................................... 355PKO_MEM_QUEUE_QOS ............................................................................ 357PKO_MEM_COUNT0 .................................................................................... 357PKO_MEM_COUNT1 .................................................................................... 358PKO_MEM_DEBUG0 .................................................................................... 358PKO_MEM_DEBUG1 .................................................................................... 358PKO_MEM_DEBUG2 .................................................................................... 359PKO_MEM_DEBUG3 .................................................................................... 359PKO_MEM_DEBUG4 .................................................................................... 359PKO_MEM_DEBUG5 .................................................................................... 360PKO_MEM_DEBUG6 .................................................................................... 360PKO_MEM_DEBUG7 .................................................................................... 361PKO_MEM_DEBUG8 .................................................................................... 361PKO_MEM_DEBUG9 .................................................................................... 361PKO_MEM_DEBUG10 .................................................................................. 362PKO_MEM_DEBUG11 .................................................................................. 362PKO_MEM_DEBUG12 .................................................................................. 362PKO_MEM_DEBUG13 .................................................................................. 363

Chapter 9 PCI Bus ...................................................................................................................... 365Overview .................................................................................................................................... 366


Table of Contents

9.1 CN50XX PCI Features ............................................................................................ 366

9.2 CN50XX Addressing as a PCI Target ...................................................................... 3679.2.1 BAR0 - Memory-Mapped CSR Region ........................................................... 3679.2.2 BAR1 - 32-Bit Memory-Mapped Region ........................................................ 3689.2.3 BAR2 - 64-bit Memory-Mapped Region......................................................... 3709.2.4 Expansion ROM .............................................................................................. 371

9.3 PCI Instruction Input From an External Host ....................................................... 3719.3.1 PCI Instruction Format.................................................................................. 3719.3.2 PCI Input Packet ............................................................................................ 3739.3.3 DPTR Formats ................................................................................................ 375

9.4 PCI Packet Output From CN50XX .......................................................................... 3779.4.1 Info-Pointer Mode ........................................................................................... 3799.4.2 Buffer-Pointer-Only Mode .............................................................................. 380

9.5 PCI DMA Engine Access From Cores ..................................................................... 3819.5.1 PCI DMA Instruction-Header Format........................................................... 3829.5.2 PCI DMA Instruction Local-Pointer Format................................................. 3839.5.3 PCI DMA Instruction PCI Components and Processing .............................. 3859.5.4 PCI DMA Instruction Fetching...................................................................... 3869.5.5 PCI DMA Instruction Ordering and Completion .......................................... 3879.5.6 PCI DMA Engine Dont-Write-Back Calculation .......................................... 3889.5.7 Host Output Queueing Via the PCI DMA Engine ........................................ 388

9.6 PCI Memory Space Loads/Stores to BAR1/2 ........................................................... 3899.6.1 Referencing L2/DRAM With CN50XX as a PCI Target................................ 389

9.7 CN50XX PCI Internal Arbiter ................................................................................. 391

9.8 CN50XX PCI MSI Support ....................................................................................... 391

9.9 Endian Swapping ..................................................................................................... 3919.9.1 PASS_THRU MODE (== 0)............................................................................ 3919.9.2 64b_BYTE_SWAP Mode (== 1) ...................................................................... 3929.9.3 32b_BYTE_SWAP Mode (== 2) ...................................................................... 3939.9.4 32b_LW_SWAP Mode (== 3) .......................................................................... 393

9.10 PC Bus Operations ................................................................................................... 3949.10.1 Load/Store Operations ................................................................................. 3949.10.2 IOBDMA Operations ...................................................................................... 3949.10.3 RSL Access Space (SubDID == 0) .................................................................. 3949.10.4 PCI Config / IACK / Special Space (SubDID == 1)........................................ 3959.10.5 PCI I/O Space (SubDID == 2)......................................................................... 3969.10.6 Memory Space (SubDID == 3, 4, 5, 6)............................................................ 3969.10.7 PCI-Related, NCB-Direct, PCICONFIG, and PCI_NCB CSR Access (SubDID ==

7) ...................................................................................................................... 397

9.11 PCI Reset Sequence .................................................................................................. 3979.11.1 PCI Reset Sequence in Host Mode................................................................. 3979.11.2 PCI Reset Sequence in Non-Host Mode......................................................... 399

9.12 PCI Checklist ............................................................................................................ 401

9.13 PCI Configuration Registers ................................................................................... 403PCI_CFG00 .................................................................................................... 404PCI_CFG01 .................................................................................................... 404PCI_CFG02 .................................................................................................... 405PCI_CFG03 .................................................................................................... 405PCI_CFG04 .................................................................................................... 405PCI_CFG05 .................................................................................................... 405PCI_CFG06 .................................................................................................... 406PCI_CFG07 .................................................................................................... 406PCI_CFG08 .................................................................................................... 406PCI_CFG09 .................................................................................................... 406


Table of Contents

PCI_CFG10 .................................................................................................... 406PCI_CFG11 .................................................................................................... 407PCI_CFG12 .................................................................................................... 407PCI_CFG13 .................................................................................................... 407PCI_CFG15 .................................................................................................... 407PCI_CFG16 .................................................................................................... 408PCI_CFG17 .................................................................................................... 409PCI_CFG18 .................................................................................................... 409PCI_CFG19 .................................................................................................... 410PCI_CFG20 .................................................................................................... 411PCI_CFG21 .................................................................................................... 411PCI_CFG22 .................................................................................................... 412PCI_CFG58 .................................................................................................... 413PCI_CFG59 .................................................................................................... 413PCI_CFG60 .................................................................................................... 414PCI_CFG61 .................................................................................................... 414PCI_CFG62 .................................................................................................... 414PCI_CFG63 .................................................................................................... 414

9.14 PCI Bus Registers .................................................................................................... 4159.14.1 PCI_NCB-Type Registers ............................................................................... 417

PCI_BAR1_INDEX(0...31) ............................................................................. 417PCI_READ_CMD_6 ....................................................................................... 417PCI_READ_CMD_C ....................................................................................... 418PCI_READ_CMD_E ....................................................................................... 418PCI_CTL_STATUS_2 .................................................................................... 419NPI_MSI_RCV ............................................................................................... 422PCI_INT_ENB2 ............................................................................................. 423PCI_INT_SUM2 ............................................................................................. 424

9.14.2 PCI-Type Registers ......................................................................................... 426PCI_WIN_WR_ADDR .................................................................................... 426PCI_WIN_RD_ADDR .................................................................................... 426PCI_WIN_WR_DATA .................................................................................... 427PCI_WIN_WR_MASK ................................................................................... 427PCI_WIN_RD_DATA ..................................................................................... 427PCI_INT_SUM ............................................................................................... 427PCI_INT_ENB ............................................................................................... 429PCI_PKTS_SENT0/1 ..................................................................................... 429PCI_PKT_CREDITS0/1 ................................................................................. 430PCI_PKTS_SENT_INT_LEV0/1 ................................................................... 430PCI_PKTS_SENT_TIME0/1 .......................................................................... 430PCI_DBELL0/1 .............................................................................................. 430PCI_INSTR_COUNT0/1 ................................................................................ 431PCI_DMA_CNT0/1 ......................................................................................... 431PCI_DMA_INT_LEV0/1 ................................................................................ 431PCI_DMA_TIME0/1 ....................................................................................... 431PCI_MSI_RCV ............................................................................................... 431

9.15 NPI Registers ............................................................................................................ 432NPI_RSL_INT_BLOCKS ............................................................................... 433NPI_DBG_SELECT ....................................................................................... 433NPI_CTL_STATUS ........................................................................................ 434NPI_INT_SUM ............................................................................................... 434NPI_INT_ENB ............................................................................................... 436NPI_MEM_ACCESS_SUBID(3..6) ............................................................... 438NPI_PCI_READ_CMD .................................................................................. 438NPI_NUM_DESC_OUTPUT0/1 .................................................................... 438NPI_BASE_ADDR_INPUT0/1 ...................................................................... 439NPI_SIZE_INPUT0/1 .................................................................................... 439PCI_READ_TIMEOUT .................................................................................. 439NPI_BASE_ADDR_OUTPUT0/1 ................................................................... 439


Table of Contents

NPI_PCI_BURST_SIZE ................................................................................ 440NPI_BUFF_SIZE_OUTPUT0/1 ..................................................................... 440NPI_OUTPUT_CONTROL ............................................................................ 441NPI_LOWP_IBUFF_SADDR ........................................................................ 441NPI_HIGHP_IBUFF_SADDR ....................................................................... 441NPI_LOWP_DBELL ...................................................................................... 442NPI_HIGHP_DBELL ..................................................................................... 442NPI_DMA_CONTROL ................................................................................... 443NPI_PCI_INT_ARB_CFG .............................................................................. 443NPI_INPUT_CONTROL ............................................................................... 444NPI_DMA_LOWP_COUNTS ........................................................................ 444NPI_DMA_HIGHP_COUNTS ....................................................................... 444NPI_DMA_LOWP_NADDR ........................................................................... 445NPI_DMA_HIGHP_NADDR ......................................................................... 445NPI_P0/1_PAIR_CNTS .................................................................................. 445NPI_P0/1_DBPAIR_ADDR ............................................................................ 445NPI_P0/1_INSTR_CNTS ............................................................................... 446NPI_P0/1_INSTR_ADDR .............................................................................. 446NPI_WIN_READ_TO ..................................................................................... 446DBG_DATA .................................................................................................... 446NPI_PORT_BP_CONTROL ........................................................................... 447NPI_PORT32/33_INSTR_HDR ..................................................................... 447NPI_BIST_STATUS ...................................................................................... 448

Chapter 10 Timer ........................................................................................................................... 449Overview .................................................................................................................................... 450

10.1 Timer Features ......................................................................................................... 450

10.2 Timer Support ........................................................................................................... 451

10.3 Software Responsibilities ......................................................................................... 452

10.4 Timer Registers ....................................................................................................... 454TIM_REG_FLAGS ......................................................................................... 455TIM_REG_READ_IDX .................................................................................. 455TIM_REG_BIST_RESULT ............................................................................ 455TIM_REG_ERROR ........................................................................................ 456TIM_REG_INT_MASK .................................................................................. 456TIM_MEM_RING0 ........................................................................................ 456TIM_MEM_RING1 ........................................................................................ 457TIM_MEM_DEBUG0 .................................................................................... 457TIM_MEM_DEBUG1 .................................................................................... 457TIM_MEM_DEBUG2 .................................................................................... 458

Chapter 11 Central Interrupt Unit (CIU) ....................................................................................... 459Overview .................................................................................................................................... 460

11.1 Central Interrupt Collection and Distribution ....................................................... 460

11.2 Per-Core Mailbox Registers ..................................................................................... 464

11.3 Per-Core Watchdog Timers ...................................................................................... 465

11.4 Four General Timers ................................................................................................ 466

11.5 Core Availability and Reset ..................................................................................... 467

11.6 Core Debug-Mode Observability .............................................................................. 467

11.7 Core Debug-Interrupt Generation ........................................................................... 468

11.8 Core Non-Maskable Interrupt Generation .............................................................. 468

11.9 Chip Soft-Reset Initiation ........................................................................................ 468

11.10CIU Registers ........................................................................................................... 469CIU_INT(0..3,32)_SUM0 ............................................................................... 470CIU_INT_SUM1 ............................................................................................ 470CIU_INT(0..3,32)_EN0 .................................................................................. 471CIU_INT(0..3,32)_EN1 .................................................................................. 471


Table of Contents

CIU_INT0/1_SUM4 ....................................................................................... 472CIU_INT0/1_EN4_0 ....................................................................................... 473CIU_INT0/1_EN4_1 ....................................................................................... 473CIU_TIM(0..3) ................................................................................................ 474CIU_WDOG0/1 ............................................................................................... 474CIU_PP_POKE0/1 ......................................................................................... 474CIU_MBOX_SET0/1 ...................................................................................... 475CIU_MBOX_CLR0/1 ...................................................................................... 475CIU_PP_RST .................................................................................................. 475CIU_PP_DBG ................................................................................................. 475CIU_GSTOP ................................................................................................... 475CIU_NMI ........................................................................................................ 476CIU_DINT ...................................................................................................... 476CIU_FUSE ..................................................................................................... 476CIU_BIST ....................................................................................................... 476CIU_SOFT_BIST ........................................................................................... 476CIU_SOFT_RST ............................................................................................. 477CIU_SOFT_PRST .......................................................................................... 477CIU_PCI_INTA .............................................................................................. 477

Chapter 12 Boot Bus .................................................................................................................... 479Overview .................................................................................................................................... 480

12.1 Boot-Bus Addresses .................................................................................................. 481

12.2 Boot-Bus Address Matching and Regions ............................................................... 481

12.3 Boot-Bus Reset Configuration and Booting ............................................................ 482

12.4 Boot-Bus Region Timing .......................................................................................... 48312.4.1 Static-Timed Read Sequences ........................................................................ 48512.4.2 Static-Timed Write Sequences ....................................................................... 49012.4.3 Static-Timed Page-Read Sequences............................................................... 49312.4.4 Dynamic-Timed Sequences ............................................................................ 497

12.5 Boot-Bus Request Queuing ...................................................................................... 498

12.6 Boot-Bus Connections .............................................................................................. 499

12.7 Boot-Bus Operations ................................................................................................ 50012.7.1 Load Operations.............................................................................................. 50012.7.2 IOBDMA Operations ...................................................................................... 50112.7.3 Store Operations ............................................................................................. 502

12.8 Boot-Bus Registers ................................................................................................... 502MIO_BOOT_REG_CFG0 ............................................................................... 503MIO_BOOT_REG_CFG(1..7) ......................................................................... 504MIO_BOOT_REG_TIM0 ................................................................................ 505MIO_BOOT_REG_TIM(1..7) ......................................................................... 505MIO_BOOT_LOC_CFG0/1 ............................................................................ 506MIO_BOOT_LOC_ADR ................................................................................. 506MIO_BOOT_LOC_DAT ................................................................................. 506MIO_BOOT_ERR ........................................................................................... 507MIO_BOOT_INT ............................................................................................ 507MIO_BOOT_THR ........................................................................................... 507MIO_BOOT_COMP ....................................................................................... 508MIO_BOOT_BIST_STAT .............................................................................. 508

Chapter 13 CN50XX Packet Interface .......................................................................................... 509Overview .................................................................................................................................... 510

13.1 Packet Interface Introduction .................................................................................. 510

13.2 RGMII Features ........................................................................................................ 51213.2.1 Flow Control.................................................................................................... 51213.2.2 Receive Preamble............................................................................................ 51413.2.3 Receive Packet Dropping................................................................................ 514


Table of Contents

13.2.4 Receive-Packet Inspection.............................................................................. 51613.2.5 Receive Link Status ........................................................................................ 51713.2.6 Packet Transmission ...................................................................................... 51713.2.7 Transmit-Packet Options ............................................................................... 51813.2.8 Collisions ......................................................................................................... 51813.2.9 Bursts .............................................................................................................. 518

13.3 Errors/Exceptions ..................................................................................................... 51913.3.1 Receive Error/Exception Checks .................................................................... 51913.3.2 Transmit Error/Exception Checks ................................................................. 52113.3.3 Transmit Error Propagation .......................................................................... 522

13.4 Link ........................................................................................................................... 52213.4.1 Link Status...................................................................................................... 52213.4.2 Link Status Changes ...................................................................................... 52213.4.3 Configuration Based on Mode ........................................................................ 523

13.5 Statistics ................................................................................................................... 523

13.6 Loopback ................................................................................................................... 524

13.7 Initialization ............................................................................................................. 525

13.8 GMX Registers .......................................................................................................... 526GMX0_RX(0..2)_INT_REG ............................................................................ 530GMX0_RX(0..2)_INT_EN .............................................................................. 531GMX0_PRT(0..2)_CFG ................................................................................... 531GMX0_RX(0..2)_FRM_CTL ........................................................................... 532GMX0_RX(0..2)_FRM_CHK .......................................................................... 534GMX0_RX(0..2)_JABBER .............................................................................. 534GMX0_RX(0..2)_DECISION .......................................................................... 535GMX0_RX(0..2)_UDD_SKP ........................................................................... 536GMX0_RX(0..2)_STATS_CTL ....................................................................... 536GMX0_RX(0..2)_IFG ...................................................................................... 537GMX0_RX(0..2)_RX_INBND ......................................................................... 537GMX0_RX(0..2)_PAUSE_DROP_TIME ........................................................ 537GMX0_RX(0..2)_STATS_PKTS ..................................................................... 538GMX0_RX(0..2)_STATS_OCTS ..................................................................... 538GMX0_RX(0..2)_STATS_PKTS_CTL ............................................................ 538GMX0_RX(0..2)_STATS_OCTS_CTL ............................................................ 539GMX0_RX(0..2)_STATS_PKTS_DMAC ........................................................ 539GMX0_RX(0..2)_STATS_OCTS_DMAC ........................................................ 539GMX0_RX(0..2)_STATS_PKTS_DRP ........................................................... 540GMX0_RX(0..2)_STATS_OCTS_DRP ........................................................... 540GMX0_RX(0..2)_STATS_PKTS_BAD ........................................................... 540GMX0_RX(0..2)_ADR_CTL ........................................................................... 541GMX0_RX(0..2)_ADR_CAM_EN ................................................................... 541GMX0_RX(0..2)_ADR_CAM(0..5) .................................................................. 542GMX0_TX(0..2)_CLK ..................................................................................... 542GMX0_TX(0..2)_THRESH ............................................................................. 542GMX0_TX(0..2)_APPEND ............................................................................. 543GMX0_TX(0..2)_SLOT ................................................................................... 543GMX0_TX(0..2)_BURST ................................................................................ 543GMX0_SMAC(0..2) ......................................................................................... 543GMX0_TX(0..2)_PAUSE_PKT_TIME ........................................................... 544GMX0_TX(0..2)_MIN_PKT ............................................................................ 544GMX0_TX(0..2)_PAUSE_PKT_INTERVAL ................................................. 545GMX0_TX(0..2)_SOFT_PAUSE .................................................................... 545GMX0_TX(0..2)_PAUSE_TOGO ................................................................... 546GMX0_TX(0..2)_PAUSE_ZERO .................................................................... 546GMX0_TX(0..2)_STATS_CTL ........................................................................ 546GMX0_TX(0..2)_CTL ..................................................................................... 546GMX0_TX(0..2)_STAT0 ................................................................................. 547GMX0_TX(0..2)_STAT1 ................................................................................. 547


Table of Contents

GMX0_TX(0..2)_STAT2 ................................................................................. 547GMX0_TX(0..2)_STAT3 ................................................................................. 548GMX0_TX(0..2)_STAT4 ................................................................................. 548GMX0_TX(0..2)_STAT5 ................................................................................. 548GMX0_TX(0..2)_STAT6 ................................................................................. 549GMX0_TX(0..2)_STAT7 ................................................................................. 549GMX0_TX(0..2)_STAT8 ................................................................................. 549GMX0_TX(0..2)_STAT9 ................................................................................. 550GMX0_BIST ................................................................................................... 550GMX_RX_PRTS ............................................................................................. 550GMX0_RX_BP_DROP(0..2) ........................................................................... 550GMX0_RX_BP_ON(0..2) ................................................................................ 551GMX0_RX_BP_OFF(0..2) .............................................................................. 552GMX0_TX_PRTS ............................................................................................ 552GMX0_TX_IFG ............................................................................................... 552GMX0_TX_JAM ............................................................................................. 552GMX0_TX_COL_ATTEMPT ......................................................................... 553GMX0_TX_PAUSE_PKT_DMAC .................................................................. 553GMX0_TX_PAUSE_PKT_TYPE ................................................................... 553GMX0_TX_OVR_BP ...................................................................................... 553GMX0_TX_BP ................................................................................................ 554GMX0_TX_CORRUPT ................................................................................... 554GMX0_RX_PRT_INFO .................................................................................. 554GMX0_TX_LFSR ............................................................................................ 555GMX0_TX_INT_REG ..................................................................................... 555GMX0_TX_INT_EN ....................................................................................... 555GMX0_NXA_ADR .......................................................................................... 556GMX_BAD_REG ............................................................................................ 556GMX_STAT_BP ............................................................................................. 556GMX0_TX_CLK_MSK0/1 .............................................................................. 556GMX0_RX_TX_STATUS ............................................................................... 557GMX0_INF_MODE ........................................................................................ 557

13.9 ASX Registers ........................................................................................................... 558ASX0_RX_PRT_EN ........................................................................................ 558ASX0_TX_PRT_EN ........................................................................................ 559ASX0_INT_REG ............................................................................................. 559ASX0_INT_EN ............................................................................................... 559ASX0_RX_CLK_SET(0..2) ............................................................................. 560ASX0_PRT_LOOP .......................................................................................... 561ASX0_TX_CLK_SET(0..2) ............................................................................. 562ASX0_TX_COMP_BYP .................................................................................. 562ASX0_TX_HI_WATER(0..2) .......................................................................... 562ASX0_GMII_RX_CLK_SET ........................................................................... 563ASX0_GMII_RX_DAT_SET ........................................................................... 563ASX0_MII_RX_DAT_SET ............................................................................. 563

Chapter 14 PCM/TDM Interface .................................................................................................... 569Overview .................................................................................................................................... 570

14.1 Signal Usage ............................................................................................................. 571

14.2 Clocking ..................................................................................................................... 57214.2.1 BCLK Generation ........................................................................................... 57214.2.2 FSYNC Generation ......................................................................................... 57314.2.3 BCLK Reception.............................................................................................. 57314.2.4 FSYNC Reception ........................................................................................... 57314.2.5 Examples BCLK/FSYNC Waveforms ............................................................ 574

14.3 TDM Engines ............................................................................................................ 57514.3.1 TDM Engine Configuration............................................................................ 57614.3.2 DMA Engines .................................................................................................. 577

14.4 Initialization Sequence ............................................................................................. 581


Table of Contents

14.5 PCM/TDM Registers ................................................................................................. 582PCM_CLK0/1_CFG ........................................................................................ 584PCM_CLK0/1_GEN ...............

Cavium Networks OCTEON Plus CN50XX Hardware ... 2008 Cavium Networks OCTEON Plus CN50XX Hardware...

Documents

Transcript of Cavium Networks OCTEON Plus CN50XX Hardware ... 2008 Cavium Networks OCTEON Plus CN50XX Hardware...