A look at memory issues

Data-transfers must occur between system memory and the

network interface controller

Typical Chipset Layout

MCHMemory Controller Hub

(Northbridge)

ICHI/O Controller Hub

(Southbridge)

CPUCentral Processing Unit

DRAMDynamic RandomAccess Memory

NICNetwork Interface

Controller

HDCHard Disk Controller

ACAudio Controller

GraphicsController

Timer Keyboard Mouse Clock

MultimediaController

FirmwareHub

USBcontroller

Typical Chipset Layout

MCHMemory Controller Hub

(Northbridge)

ICHI/O Controller Hub

(Southbridge)

CPUCentral Processing Unit

DRAMDynamic RandomAccess Memory

NICNetwork Interface

Controller

HDCHard Disk Controller

ACAudio Controller

GraphicsController

Timer Keyboard Mouse Clock

MultimediaController

FirmwareHub

USBcontroller

DMA

PCI Bus Master DMA82573L i/o-memory

RX and TX FIFOs(32-KB total)

Host’s Dynamic Random Access Memory

Descriptor Queue

packet-buffer

packet-buffer

packet-buffer

packet-buffer

packet-buffer

packet-buffer

packet-buffer

DMA

on-chip RX descriptors

on-chip TX descriptors

Memory-mapped I/O

• We mentioned that Intel’s x86 architecture originally was designed with two separate address-spaces, one for memory and the other for I/O ports, unlike the designs for CPUs by many of Intel’s competitors in which I/O access was “memory-mapped”

• But now the newer Intel processors also can support memory-mapped I/O as well

Address-bus widths

• The Intel Core-2 Quad processors in our classroom and Lab machines potentially could address 236 physical memory cells (i.e., 64GB), although only 4GB of RAM actually are installed at the present time

• Some PCI-compliant hardware devices were designed for a 32-bit address-bus, thus they must be “mapped” below 4G

Physical-address assignments

The CPU’s physical address-space

Devices’ registers must be mapped to addresses in the bottom 4G

Dynamic Random Access Memory

Virtual addresses

• Software running on the x86 processor is unable to use actual memory addresses, but instead uses ‘virtual’ addresses that map to physical addresses by means of mapping-tables which Linux dynamically defines for each different process it runs

• This complicates the steps software must take to arrange for the DMA to take place

Our ‘dram.c’ module

• To help us confirm that our hardware-level network software is working as we intend, or to diagnose our ‘bugs’ if it isn’t, we can use an LKM we’ve written that implements a character-mode device-driver for system memory, allowing us to view the contents of physical memory as if it were a file; for example, by using our ‘fileview.cpp’ tool

How to view system memory

• Download ‘dram.c’ from course website

• Compile it using our ‘mmake’ utility

• Install ‘dram.ko’ by using ‘/sbin/insmod

• Insure the ‘/dev/dram’ device-node exists

• Download ‘fileview.cpp’ from website and compile it with ‘g++’ (or with ‘make’)

• Execute ‘fileview /dev/dram’ and use the arrow-keys to navigate (or hit <ENTER>)

“canonical” addresses0000000001000100001100100001010011000111

01000010010101001011011000110101110011111000010001100101001110100101011011010111

1100011001110101101111100111011111011111

“non-canonical”(invalid)

virtual addresses

“canonical” addresses

“canonical” addresses

0xFFFFFFFFFFFFFFFF … 0xFFFF800000000000

0x00007FFFFFFFFFFF … 0x0000000000000000

Analogy using 5-bit values

64-bit “vrtual”address space

4-Levels of mapping

PageMap

Level-4Table

CR3

PageDirectoryPointerTable

PageDirectory

PageTable

PageFrame(4KB)

offset

64-bit ‘canonical’ virtual address

sign-extension PML4 PDPT PDIR PTBL 63 48 47 39 38 30 29 21 20 12 11 0

Each mapping-table contains up to 512 quadword-size entries

4-level address-translation

• The CPU examines any virtual address it encounters, subdividing it into five fields

offset into page-frame

index into page-table

63 48 47 39 38 30 29 21 20 12 11 0

16-bits 9-bits 12-bits

index into page-

directory

index into page-

directorypointer table

index intolevel 4

page-maptable

9-bits9-bits9-bits

sign-extension

Any 48-bit virtual-address is sign-extended to a 64-bit “canonical” address

Only “canonical” 64-bit virtual-addresses are legal in 64-bit mode

Reserved(must be 0)

Format of 64-bit table-entries

63 62 52 51 40 39 32

31 12 11 9 8 7 6 5 4 3 2 1 0

Page-frame physical base-address[31..12]

Page-framephysical

base-address[39..32]

EXB

PWUPWT

PCD

A

avl

avl

Meaning of these bits varies with the table

Legend: P = Present (1=yes, 0=no) PWT = Page Cache Disable (1=yes, 0=no) W = Writable (1=yes, 0=no) PWT = Page Write-Through (1=yes, 0=no) U = User-page (1=yes, 0=no) avl = available for user-defined purposes A = Accessed (1=yes, 0=no) EXB = Execution-disabled Bit (if EFER.NXE=1)

Our ‘mem64.c’ module

• We wrote an LKM to create a pseudo-file that will let us see how the virtual memory is being utilized by an application program

• Download this file, compile it with ‘mmake’ and install ‘mem64.ko’ in the Linux kernel

• Then view the virtual-memory mapping that is being used by the ‘cat’ program:

$ cat /proc/mem64

The NIC’s PCI ‘resources’

StatusRegister

CommandRegister

DeviceID0x109A

VendorID0x8086

BISTCacheLineSize

Class CodeClass/SubClass/ProgIF

RevisionID

Base Address 0

SubsystemDevice ID

SubsystemVendor ID

CardBus CIS Pointer

reservedcapabilities

pointer Expansion ROM Base Address

MinimumGrant

InterruptPin

reserved

LatencyTimer

HeaderType

Base Address 1

Base Address 2Base Address 3

Base Address 4Base Address 5

InterruptLine

MaximumLatency

31 0 31 0

16 doublewords

Dwords

1 - 0

3 - 2

5 - 4

7 - 6

9 - 8

11 - 10

13 - 12

15 - 14

Mechanisms compared

kernel memory-space

NIC i/o-memory

CPU’s ‘virtual’ address-space

io

user memory-space

Each NIC register has its own address in memory (allows one-step access)

addr data

Access to all of the NIC’s registers is muliplexed through a pair of I/O-ports(requires multiple instructions)

CPU’s ‘I/O’ address-space

‘nicstatus.c’

• Here’s an LKM that creates a pseudo-file (called ‘/proc/nicstatus’) which will allow a user to view the current value in our Intel 82573L Network Interface Controller’s ‘DEVICE_STATUS’ register

• It uses the I/O-port interface to the NIC’s registers, rather than a ‘memory-mapped’ interface to those device-registers

0

Device Status (0x0008)

? 0 0 0 0 0 0 0 0 0 0 0GIO

MasterEN

0 0 0

0 0 0 0 PHYRA ASDV

ILOS

SLU

0 TXOFF 0 0

FD

15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0

FunctionID

LU

31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16

SPEED

FD = Full-DuplexLU = Link UpTXOFF = Transmission PausedSPEED (00=10Mbps,01=100Mbps, 10=1000Mbps, 11=reserved)ASDV = Auto-negotiation Speed Detection ValuePHYRA = PHY Reset Asserted

82573L

some undocumented functionality?

‘82573.c’

• This is a more elaborate example of an LKM which not only creates a pseudo-file (i.e., ‘/proc/82573’) that we can view using the Linux ‘cat’ command and that lets us see our NIC’s PCI Configuration Space, but also implements some device-driver functions that let us view the NIC’s device registers by using our ‘fileview.cpp’ tool

Linux PCI helper-functions

#include <linux/pci.h>

struct pci_dev *devp; unsigned int mmio_base; unsigned int mmio_size; void *io;

devp = pci_get_device( VENDOR_ID, DEVICE_ID, NULL );if ( devp == NULL ) return –ENODEV;

mmio_base = pci_resource_start( devp, 0 );mmio_size = pci_resource_len( devp, 0 );io = ioremap_nocache( mmio_base, iomm_size );if ( io == NULL ) return –ENOSPC;

In-class exercise

• Two of the NIC’s 32-bit device registers are used to hold its 48-bit Ethernet MAC address – which will be a different value for each of the hosts in our classroom

• These two registers are located at offsets 0x5400 and 0x5404 in device-memory

• The six bytes occur in network byte-order

• Write code to show the MAC address!