Post on 19-Jan-2016
June 16, 2002 SPC Tutorial 1WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
A Smart Port Card (SPC and SPC-II)
Tutorial---
Hardware
John DeHartWashington University
jdd@arl.wustl.eduhttp://www.arl.wustl.edu/~jdd
June 16, 2002 SPC Tutorial 2WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Motivation
• Active Networking• Network Probe• High performance router architectures
– PC as router is VERY limited
– (Gigabit/s + Processing) on each port
– MSR: Multi-Service multiport Router
June 16, 2002 SPC Tutorial 3WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
The Original Smart Port Card
• Hardware:– SPC as a PC
• How do they each boot?
– SPC Hardware Components• What roles do they play?
June 16, 2002 SPC Tutorial 4WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Typical Pentium PC
CPU North-BridgeCache DRAM
SouthBridge (PIIX3)(PIC, PIT, …)
PCI Bus
ISA Bus
PCIDevices
ISADevices BIOSSuper-IO BIOS
RT
CU
art
sK
bd
/Ms
eF
lop
py
Pa
rall
el
...
Addr/Data Ctrl
Ctrl
Addr/Data/Ctrl
Intr
NM
IIN
IT
CPU/Memory Bus
June 16, 2002 SPC Tutorial 5WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Typical Pentium PC: CPU/Memory Bus
CPU North-BridgeCache DRAM
SouthBridge (PIIX3)(PIC, PIT, …)
PCI Bus
ISA Bus
PCIDevices
ISADevices BIOSSuper-IO BIOS
RT
CU
art
sK
bd
/Ms
eF
lop
py
Pa
rall
el
...
Addr/Data Ctrl
Ctrl
Addr/Data/Ctrl
Intr
NM
IIN
IT
CPU/Memory Bus
June 16, 2002 SPC Tutorial 6WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Typical Pentium PC: PCI Bus
CPU North-BridgeCache DRAM
SouthBridge (PIIX3)(PIC, PIT, …)
PCI Bus
PCIDevices
ISADevices BIOSSuper-IO BIOS
RT
CU
art
sK
bd
/Ms
eF
lop
py
Pa
rall
el
...
Addr/Data Ctrl
Ctrl
Addr/Data/Ctrl
Intr
NM
IIN
IT
CPU/Memory Bus
ISA Bus
June 16, 2002 SPC Tutorial 7WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Typical Pentium PC: ISA Bus
CPU North-BridgeCache DRAM
SouthBridge (PIIX3)(PIC, PIT, …)
PCI Bus
ISA Bus
PCIDevices
ISADevices BIOSSuper-IO BIOS
RT
CU
art
sK
bd
/Ms
eF
lop
py
Pa
rall
el
...
Addr/Data Ctrl
Ctrl
Addr/Data/Ctrl
Intr
NM
IIN
IT
CPU/Memory Bus
June 16, 2002 SPC Tutorial 8WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
How NetBSD Boots on a PC
Components:– Pentium
– Boot ROM (part of BIOS in modern systems?)
– BIOS
– Bootloader
– Kernel
June 16, 2002 SPC Tutorial 9WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Sketch of How a PC Boots
• Pentium after Reset:– fetches its first instruction from location 0xFFFFFFF0
• Boot Code must be located at 0xFFFFFFF0
• Boot Code jumps to BIOS located in ROM– Boot Code may actually be part of the BIOS...
• BIOS copies itself into memory (Shadow)
• BIOS remaps memory – future accesses to BIOS addresses go to memory instead of ROM.
• BIOS performs system configuration (some proprietary)– Motherboard Details
– Pentium Details
– NB/SB Chipset Details
– Device configuration: IRQs, Memory maps, ...
June 16, 2002 SPC Tutorial 10WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
How a PC Boots (continued…)• BIOS loads bootloader into memory (from disk…)• BIOS jumps to bootloader• Bootloader performs some more configuration:
– Pentium control registers– Cache configuration– Memory/Page model
• Bootloader determines what to run next.• Bootloader may have to do some device configuration.
– e.g. to get OS from a disk.
• Bootloader loads OS kernel into memory• Bootloader jumps to start of OS kernel• Kernel does some OS-specific configuration:
– for NetBSD look in: sys/arch/i386/i386/locore.s– Determines what CPU it has (“cpuid” instruction)– Paging– Virtual Memory
June 16, 2002 SPC Tutorial 11WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Typical Pentium PC (Again…)
CPU North-BridgeCache DRAM
SouthBridge (PIIX3)(PIC, PIT, …)
PCI Bus
ISA Bus
PCIDevices
ISADevices BIOSSuper-IO BIOS
RT
CU
art
sK
bd
/Ms
eF
lop
py
Pa
rall
el
...
Addr/Data Ctrl
Ctrl
Addr/Data/Ctrl
Intr
NM
IIN
IT
CPU/Memory Bus
June 16, 2002 SPC Tutorial 12WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
What SPC Needs
CPU North-BridgeCache DRAM
SouthBridge(PIC, PIT, …)
PCI Bus
APIC
BIOSBIOS
Addr/Data Ctrl
Ctrl
Addr/Data/Ctrl
Intr
NM
IIN
IT
RT
CU
art
s
CPU/Memory Bus
June 16, 2002 SPC Tutorial 13WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
What SPC Needs: CPU/Memory Bus
CPU North-BridgeCache DRAM
SouthBridge(PIC, PIT, …)
PCI Bus
APIC
BIOSBIOS
Addr/Data Ctrl
Ctrl
Addr/Data/Ctrl
Intr
NM
IIN
IT
RT
CU
art
s
CPU/Memory Bus
June 16, 2002 SPC Tutorial 14WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
What SPC Needs: PCI Bus
CPU North-BridgeCache DRAM
SouthBridge(PIC, PIT, …)
PCI Bus
APIC
BIOSBIOS
Addr/Data Ctrl
Ctrl
Addr/Data/Ctrl
Intr
NM
IIN
IT
RT
CU
art
s
CPU/Memory Bus
June 16, 2002 SPC Tutorial 15WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
What SPC Needs: ISA Bus?
CPU North-BridgeCache DRAM
SouthBridge(PIC, PIT, …)
PCI Bus
APIC
BIOSBIOS
Addr/Data Ctrl
Ctrl
Addr/Data/Ctrl
Intr
NM
IIN
IT
RT
CU
art
s
CPU/Memory Bus
June 16, 2002 SPC Tutorial 16WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
System FPGA
Intel Embedded Module
SPC Architecture
CPU North-BridgeCache DRAM
PCI Bus
APIC
Addr/Data Ctrl
Ctrl
Addr/Data/Ctrl
Intr
NM
IIN
IT
PITPICRTC’
BIOS ROMUART1 Interface
UART2 Interface
UART1
UART2Link Interface
Switch Interface
June 16, 2002 SPC Tutorial 17WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
SPC Components• APIC• Pentium Embedded Module
– 166 MHz MMX Pentium Processor• L1 Cache: 16KB Data, 16KB Code
– L2 cache: 512 KB– NorthBridge - 33 MHz, 32 bit PCI Bus
• System FPGA– Xilinx XC4020XLA-1 FPGA
• Memory– EDO DRAM: 64MB (Max for current design)– SO DIMM
• Switch and Link Interfaces – Each 1 Gb Utopia• UART supports two Serial Ports:
• NetBSD system console• TTY port
June 16, 2002 SPC Tutorial 18WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
System FPGA
• Coded in VHDL
• PCI slave device
• Replaces some of the PIIX3 (south bridge)
• Replaces some of the BIOS
• Replaces some of the Super IO Chip
• Provides reset capability
June 16, 2002 SPC Tutorial 19WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
System FPGA: PIIX3 Functionality
• Programmable Interrupt Controller (PIC)– Four Interrupts supported and statically assigned:
• PIT (IRQ 0)
• APIC (IRQ 5)
• COM1 (IRQ 4)
• COM2 (IRQ 3)
– Static fully-nested interrupt priority structure.
– Specific End of Interrupt is the only EOI mode supported
• Programmable Interval Timer (PIT)– generates a clock interrupt for NetBSD every ~10ms
• Reset - covered in a later slide
June 16, 2002 SPC Tutorial 20WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
System FPGA: BIOS Functionality
• Interrupt functionality replaced by static values• Simple 16 word by 32-bit “ROM”
– implements loop waiting for location 0xFFE00 to change value
– then jumps to boot loader code
• Does NOT perform configuration of Northbridge– This will be done by the boot loader
• Does NOT perform PCI configuration of APIC– This will be done by the APIC Driver
June 16, 2002 SPC Tutorial 21WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
System FPGA: Super IO Chip Functionality
• UART Interface– Two Serial lines supported
– Fixed IRQs
• Real Time Clock– only the register accesses of the RTC are supported
– no interrupts supported
– i.e. supported only so NetBSD didn’t need to change
– i.e. no alarms will be generated
June 16, 2002 SPC Tutorial 22WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
System FPGA: Reset
• SPC needs a reset before every download:– switch reset:
• causes SPC to be reset
• causes all connections in switch to be lost
– System FPGA reset• causes SPC to be reset
• has no effect on the switch
• Normal SouthBridge reset:– I/O Register: 0xCF9
– Hard Reset: assert CPURST, PCIRST#, and RSTDRV• write 0xCF9 0x02 (00000010b)
• write 0xCF9 0x06 (00000110b)
– Soft Reset: assert INIT• write 0xCF9 0x00 (00000000b)
• write 0xCF9 0x04 (00000100b)
bits
June 16, 2002 SPC Tutorial 23WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
System FPGA: Reset
• SPC Reset:– a sequence of two writes to memory addresses
– APIC Control cells can write to• memory addresses
• configuration registers
• NOT I/O Registers! Argh...
– To mimic the reset structure of the SB we use:• 0xFFFFFFF0
• 0xFFFFFFF4
– Hard Reset (all we really care about)• write 0xFFFFFFF0 0x02 (00000010b)
• write 0xFFFFFFF4 0x06 (00000110b)bits
June 16, 2002 SPC Tutorial 24WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
References: SPC-I (on Kits Web Pages)
• Intel Embedded Module:– Data Sheet
– Design Guide
• 430HX Chipset– NorthBridge
– SouthBridge
• System FPGA
• Memory
• Mobile Pentium with MMX– Software Developer Manuals 1,2,3
– Datasheet
• APIC
• Cache
June 16, 2002 SPC Tutorial 25WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
SPC-II
• Motivation– Faster Processor
– More Memory
– Faster Memory Bus
– Simpler• Real BIOS
• No System FPGA to build a fake Southbridge
– Interchangeable modules• 700 MHz already available
• Other low-power modules also available
• You can buy new/faster CPU modules if you want…
• More details to be given by Dave Richard on Tuesday at the Workshop
June 16, 2002 SPC Tutorial 26WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
ETX Embedded Module
P-IIICPU
North-Bridge
VideoDRAM
PCI Bus
FPGA
Video DB15
Link InterfaceSwitch Interface
SPC-II Architecture
South-Bridge BIOS
ISA Bus
I/O
AGP Bus
FLASH DISKIDE Bus
KeyboardMouseSerial Ports
APIC
June 16, 2002 SPC Tutorial 27WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
SPC-II Architecture
TI
IPP
OP
P
SWITCH
Link
256 MBSDRAM
500/700 MHzPentium-III
256 KB L2 Cache
Intel BXNorth Bridge
Port 1APICPort 0
SPC-II16/32 bit
32 bit
PCIBUS
SouthBridge
ISA Bus
ISADevices Super-IO
BIOS
FPGA
FPX
16 bit
16 bit
100
MH
z M
emor
y B
us
June 16, 2002 SPC Tutorial 28WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
SPC-II• Components
– Intel Module• 500 MHz Pentium III• 256 KB L2 Cache• Northbridge and Southbridge• 33 MHz, 32 bit PCI Bus• 100 MHz memory bus • BIOS• Video, Mouse, Keyboard
– APIC– FPGA: for ATM level routing– Extra PCI Slot: Debugging possibilities.– 256 MB of Memory– 32 MB IDE DOM (Disk On Module)
• right now holds binary boot code• could later hold actual kernel• larger ones are possible
June 16, 2002 SPC Tutorial 29WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Booting an SPC-II
• IDE DOM contains boot code– modified Stage2 from AAL5_download
• configured to boot in place of a kernel– No special BIOS tricks to play
• contains APIC code to put kernel into memory• kernel downloaded via AAL5 frames
– uses Stage3 from AAL5_download
• kernel booted when download complete
June 16, 2002 SPC Tutorial 30WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
SPC II FPGA Architecture
FPXLC Switch
APIC
PCI Bus Port
Port 1Port 0
SPC-II FPGA
VPI[0]=1
VPI[0]=0
64<=VCI<=127???
Reset
VPI[0]=1VCI = 38
OSC
June 16, 2002 SPC Tutorial 31WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
SPC-II References and more info
• APIC References the same as before• Other references to come soon• Dave Richard will talk more about SPC-II on
Wednesday at the Workshop
June 16, 2002 SPC Tutorial 32WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
• The rest of the slides are a Gallery of SPC Photos