June 16, 2002 SPC Tutorial 1 Washington WASHINGTON UNIVERSITY IN ST LOUIS A Smart Port Card (SPC and...

June 16, 2002 SPC Tutorial 1WashingtonWASHINGTON UNIVERSITY IN ST LOUIS

A Smart Port Card (SPC and SPC-II)

Tutorial---

Hardware

John DeHartWashington University

[email protected]://www.arl.wustl.edu/~jdd


Motivation

• Active Networking• Network Probe• High performance router architectures

– PC as router is VERY limited

– (Gigabit/s + Processing) on each port

– MSR: Multi-Service multiport Router


The Original Smart Port Card

• Hardware:– SPC as a PC

• How do they each boot?

– SPC Hardware Components• What roles do they play?


Typical Pentium PC

CPU North-BridgeCache DRAM

SouthBridge (PIIX3)(PIC, PIT, …)

PCI Bus

ISA Bus

PCIDevices

ISADevices BIOSSuper-IO BIOS

RT

CU

art

sK

bd

/Ms

eF

lop

py

Pa

rall

el

...

Addr/Data Ctrl

Ctrl

Addr/Data/Ctrl

Intr

NM

IIN

IT

CPU/Memory Bus


Typical Pentium PC: CPU/Memory Bus



PCI Bus

ISA Bus

PCIDevices


RT

CU

art

sK

bd

/Ms

eF

lop

py

Pa

rall

el

...

Addr/Data Ctrl

Ctrl

Addr/Data/Ctrl

Intr

NM

IIN

IT

CPU/Memory Bus


Typical Pentium PC: PCI Bus



PCI Bus

PCIDevices


RT

CU

art

sK

bd

/Ms

eF

lop

py

Pa

rall

el

...

Addr/Data Ctrl

Ctrl

Addr/Data/Ctrl

Intr

NM

IIN

IT

CPU/Memory Bus

ISA Bus


Typical Pentium PC: ISA Bus



PCI Bus

ISA Bus

PCIDevices


RT

CU

art

sK

bd

/Ms

eF

lop

py

Pa

rall

el

...

Addr/Data Ctrl

Ctrl

Addr/Data/Ctrl

Intr

NM

IIN

IT

CPU/Memory Bus


How NetBSD Boots on a PC

Components:– Pentium

– Boot ROM (part of BIOS in modern systems?)

– BIOS

– Bootloader

– Kernel


Sketch of How a PC Boots

• Pentium after Reset:– fetches its first instruction from location 0xFFFFFFF0

• Boot Code must be located at 0xFFFFFFF0

• Boot Code jumps to BIOS located in ROM– Boot Code may actually be part of the BIOS...

• BIOS copies itself into memory (Shadow)

• BIOS remaps memory – future accesses to BIOS addresses go to memory instead of ROM.

• BIOS performs system configuration (some proprietary)– Motherboard Details

– Pentium Details

– NB/SB Chipset Details

– Device configuration: IRQs, Memory maps, ...


How a PC Boots (continued…)• BIOS loads bootloader into memory (from disk…)• BIOS jumps to bootloader• Bootloader performs some more configuration:

– Pentium control registers– Cache configuration– Memory/Page model

• Bootloader determines what to run next.• Bootloader may have to do some device configuration.

– e.g. to get OS from a disk.

• Bootloader loads OS kernel into memory• Bootloader jumps to start of OS kernel• Kernel does some OS-specific configuration:

– for NetBSD look in: sys/arch/i386/i386/locore.s– Determines what CPU it has (“cpuid” instruction)– Paging– Virtual Memory


Typical Pentium PC (Again…)



PCI Bus

ISA Bus

PCIDevices


RT

CU

art

sK

bd

/Ms

eF

lop

py

Pa

rall

el

...

Addr/Data Ctrl

Ctrl

Addr/Data/Ctrl

Intr

NM

IIN

IT

CPU/Memory Bus


What SPC Needs


SouthBridge(PIC, PIT, …)

PCI Bus

APIC

BIOSBIOS

Addr/Data Ctrl

Ctrl

Addr/Data/Ctrl

Intr

NM

IIN

IT

RT

CU

art

s

CPU/Memory Bus


What SPC Needs: CPU/Memory Bus



PCI Bus

APIC

BIOSBIOS

Addr/Data Ctrl

Ctrl

Addr/Data/Ctrl

Intr

NM

IIN

IT

RT

CU

art

s

CPU/Memory Bus


What SPC Needs: PCI Bus



PCI Bus

APIC

BIOSBIOS

Addr/Data Ctrl

Ctrl

Addr/Data/Ctrl

Intr

NM

IIN

IT

RT

CU

art

s

CPU/Memory Bus


What SPC Needs: ISA Bus?



PCI Bus

APIC

BIOSBIOS

Addr/Data Ctrl

Ctrl

Addr/Data/Ctrl

Intr

NM

IIN

IT

RT

CU

art

s

CPU/Memory Bus


System FPGA

Intel Embedded Module

SPC Architecture


PCI Bus

APIC

Addr/Data Ctrl

Ctrl

Addr/Data/Ctrl

Intr

NM

IIN

IT

PITPICRTC’

BIOS ROMUART1 Interface

UART2 Interface

UART1

UART2Link Interface

Switch Interface


SPC Components• APIC• Pentium Embedded Module

– 166 MHz MMX Pentium Processor• L1 Cache: 16KB Data, 16KB Code

– L2 cache: 512 KB– NorthBridge - 33 MHz, 32 bit PCI Bus

• System FPGA– Xilinx XC4020XLA-1 FPGA

• Memory– EDO DRAM: 64MB (Max for current design)– SO DIMM

• Switch and Link Interfaces – Each 1 Gb Utopia• UART supports two Serial Ports:

• NetBSD system console• TTY port


System FPGA

• Coded in VHDL

• PCI slave device

• Replaces some of the PIIX3 (south bridge)

• Replaces some of the BIOS

• Replaces some of the Super IO Chip

• Provides reset capability


System FPGA: PIIX3 Functionality

• Programmable Interrupt Controller (PIC)– Four Interrupts supported and statically assigned:

• PIT (IRQ 0)

• APIC (IRQ 5)

• COM1 (IRQ 4)

• COM2 (IRQ 3)

– Static fully-nested interrupt priority structure.

– Specific End of Interrupt is the only EOI mode supported

• Programmable Interval Timer (PIT)– generates a clock interrupt for NetBSD every ~10ms

• Reset - covered in a later slide


System FPGA: BIOS Functionality

• Interrupt functionality replaced by static values• Simple 16 word by 32-bit “ROM”

– implements loop waiting for location 0xFFE00 to change value

– then jumps to boot loader code

• Does NOT perform configuration of Northbridge– This will be done by the boot loader

• Does NOT perform PCI configuration of APIC– This will be done by the APIC Driver


System FPGA: Super IO Chip Functionality

• UART Interface– Two Serial lines supported

– Fixed IRQs

• Real Time Clock– only the register accesses of the RTC are supported

– no interrupts supported

– i.e. supported only so NetBSD didn’t need to change

– i.e. no alarms will be generated


System FPGA: Reset

• SPC needs a reset before every download:– switch reset:

• causes SPC to be reset

• causes all connections in switch to be lost

– System FPGA reset• causes SPC to be reset

• has no effect on the switch

• Normal SouthBridge reset:– I/O Register: 0xCF9

– Hard Reset: assert CPURST, PCIRST#, and RSTDRV• write 0xCF9 0x02 (00000010b)

• write 0xCF9 0x06 (00000110b)

– Soft Reset: assert INIT• write 0xCF9 0x00 (00000000b)

• write 0xCF9 0x04 (00000100b)

bits


System FPGA: Reset

• SPC Reset:– a sequence of two writes to memory addresses

– APIC Control cells can write to• memory addresses

• configuration registers

• NOT I/O Registers! Argh...

– To mimic the reset structure of the SB we use:• 0xFFFFFFF0

• 0xFFFFFFF4

– Hard Reset (all we really care about)• write 0xFFFFFFF0 0x02 (00000010b)

• write 0xFFFFFFF4 0x06 (00000110b)bits


References: SPC-I (on Kits Web Pages)

• Intel Embedded Module:– Data Sheet

– Design Guide

• 430HX Chipset– NorthBridge

– SouthBridge

• System FPGA

• Memory

• Mobile Pentium with MMX– Software Developer Manuals 1,2,3

– Datasheet

• APIC

• Cache


SPC-II

• Motivation– Faster Processor

– More Memory

– Faster Memory Bus

– Simpler• Real BIOS

• No System FPGA to build a fake Southbridge

– Interchangeable modules• 700 MHz already available

• Other low-power modules also available

• You can buy new/faster CPU modules if you want…

• More details to be given by Dave Richard on Tuesday at the Workshop


ETX Embedded Module

P-IIICPU

North-Bridge

VideoDRAM

PCI Bus

FPGA

Video DB15

Link InterfaceSwitch Interface

SPC-II Architecture

South-Bridge BIOS

ISA Bus

I/O

AGP Bus

FLASH DISKIDE Bus

KeyboardMouseSerial Ports

APIC


SPC-II Architecture

TI

IPP

OP

P

SWITCH

Link

256 MBSDRAM

500/700 MHzPentium-III

256 KB L2 Cache

Intel BXNorth Bridge

Port 1APICPort 0

SPC-II16/32 bit

32 bit

PCIBUS

SouthBridge

ISA Bus

ISADevices Super-IO

BIOS

FPGA

FPX

16 bit

16 bit

100

MH

z M

emor

y B

us


SPC-II• Components

– Intel Module• 500 MHz Pentium III• 256 KB L2 Cache• Northbridge and Southbridge• 33 MHz, 32 bit PCI Bus• 100 MHz memory bus • BIOS• Video, Mouse, Keyboard

– APIC– FPGA: for ATM level routing– Extra PCI Slot: Debugging possibilities.– 256 MB of Memory– 32 MB IDE DOM (Disk On Module)

• right now holds binary boot code• could later hold actual kernel• larger ones are possible


Booting an SPC-II

• IDE DOM contains boot code– modified Stage2 from AAL5_download

• configured to boot in place of a kernel– No special BIOS tricks to play

• contains APIC code to put kernel into memory• kernel downloaded via AAL5 frames

– uses Stage3 from AAL5_download

• kernel booted when download complete


SPC II FPGA Architecture

FPXLC Switch

APIC

PCI Bus Port

Port 1Port 0

SPC-II FPGA

VPI[0]=1

VPI[0]=0

64<=VCI<=127???

Reset

VPI[0]=1VCI = 38

OSC


SPC-II References and more info

• APIC References the same as before• Other references to come soon• Dave Richard will talk more about SPC-II on

Wednesday at the Workshop


• The rest of the slides are a Gallery of SPC Photos

June 16, 2002 SPC Tutorial 1 Washington WASHINGTON UNIVERSITY IN ST LOUIS A Smart Port Card (SPC and...

Documents

Transcript of June 16, 2002 SPC Tutorial 1 Washington WASHINGTON UNIVERSITY IN ST LOUIS A Smart Port Card (SPC and...