Building a Reliable Onboard Network with Ethernet: A GSFC Prototype Jane Marquart NASA/GSFC.

Building a Reliable Onboard Network with Ethernet: A GSFC Prototype

Jane MarquartNASA/GSFC

8 June 2004 2Fourth Space Internet Workshop

Requirements

• Ethernet/IP Onboard Network Bus

• Critical real-time data must be deliveredreliably onboard the spacecraft

(target is 5 milliseconds)


Protocol Options

TCPTCP works but doesn’t support reliability features required by typical flight software

UDPUDP is a better fit, but requires reliability be implemented by the user

UDP +Reliability

Where to implement reliability? Applicationlayer or data link layer?


Reliability Trade

• Application layer:

– Rx & tx packet latency through the stack is highly variable and non-deterministic (regardless of direction)

– Latency up to several milliseconds on a MCP750 @ 233mhz

• Datalink layer:

– avoids IP stack latency, but;

– the issue is standardization and portability of a datalink layer solution.


IEEE 802.3 LLC

• LLC – “Logical Link Control” offers a IEEE standardized datalink layer reliability protocol, adding a 3 byte header to the ethernet frame

• Supplies 3 major types of service;– Type 1 (unreliable packet exchange, same as plain ethernet)– Type 2 (reliable, statefull, connection oriented; used by FDDI, Wireless,

Token Ring)– Type 3 (reliable, stateless, connectionless)

• Type 3 selected– Stateless but reliable– Well-defined (and simple) state machine– Suitable for a variety of physical layers– Packet overhead is 3 bytes


Data-link Layer Reliability: LLC

• Ack/retry algorithm implemented within the NIC driver.

• Latency & delay well defined since control is right at the hardware.

• Widely supported, if not tolerated, by common OS IP stacks, routers, etc.

• Well defined & supported but ad-hoc mapping onto standard ethernet framing.

• Reliability protocol only useable between conformant drivers.


Packet data

Packet dataEthernet frame header LLC header

PDU Formats

ethernet dest

(6 bytes)

ethernet source

(6 bytes)

ethernet type

(2 bytes)DSAP

(1 byte)SSAP

(1 byte)Control(1 byte)

user data &fill if req’d

Ethernet destination(6 bytes)

Ethernet source(6 bytes)

Ethernet type(2 bytes)

User data &fill if req’d

Default datalink

pdu

Reliable pdu (data

packet)

Source SAPDestination SAP

Reliable pdu (ack

packet)

ack pdu's swap DSAP & SSAP, along with ethernet source/destination so reply is sent to originating host.

Standard ethernet frame w/ EthernetType >= 0x600

EthernetType < 0x600, set to frame length indicating a 802.2 LLC pdu

Ethernet frame header

Packet dataEthernet frame header LLC header

ethernet dest

(6 bytes)

ethernet source

(6 bytes)

ethernet type

(2 bytes)DSAP

(1 byte)SSAP

(1 byte)Control(1 byte)

No user data,Fill to min. valid pkt length


Ethernet/IP Multi-node Testbed

SWITCH

Subsys ‘A’

Coldfire

MEDIACONVERTER

Ethernet Hub

"Comm Card"Dell PC

ITOS GSE

Router

Router

Serial

Channel Link Simulator

Serial

Serial

4 Mbps

50 kbits 150kbits

150 kbits

4 Mbps

1 kbits

200 kbits

Flight NIC

NetMgr

SB

TO

CICFDP

NIC Driver

IP OS Stack

MEDIACONVERTER

CFDP

Subsys‘B’PC


NIC Driver Architecture

IP stack

RX packets

TX packets

RX

interru

pt

Packet Scheduler

Async Pkt Channel Queues

SOIS Independent Layer

SOIS dependent Data-link layer

Packet validation &

LLC processing

Packet retrieved

TX

don

e interru

pt

Packet submitted

to NIC

LLC ack/error packets

Flight NICEth1Eth0

RXpackets

TX packets


Use Case Examples

nominalPkt sent with reliable QOS

Driver returns ‘ACK’

Receive ACK Done

pkt dropPkt sent with reliable QOS

ACK Time-out,Resend pkt Driver returns

‘ACK’Receive ACK Done

ACK dropPkt sent with reliable QOS


ACK Time-out,Resend pkt, Log error

Driver returns ‘ACK’ Discards redundant pktLogs error

Receive ACK Done

pkt lostPkt ‘N’ sent with reliable QOS

ACK Time-out,Resend pkt ‘N’Log error

Driver returns ‘ACK’; logs error

Receive ACK Done

ACK lost

Pkt N+1 sent with reliable QOS


ACK Time- out,Resend pkt, Log error

Driver returns ‘ACK’ Discards redundant pktLogs error

Receive ACK Done

ACK Time-out,Log error

Pkt ‘N’ sent with reliable QOS Driver returns

‘ACK’

ACK Time- out,Log error

Pkt N+1 sent with reliable QOS

Pktlost

Pktlost

Pktlost

ACKlost

ACKlost

ACKlost


NIC/OS Transmit Path Measurements

User app. opens socket

to send packet (UDP Client)

Stack Processes Packet

Key:

= start receive markers

Tx_Stack_to_Staging_Buffer

NIC receives pkt.

Metrics:1.) Stack Process Time

2.) Driver copy from stack to Staging buffer3.) DMA vs. Programmed I/O time4.) Switch Latency

Switch

= logic analyzer timestamp= timestamp marker (software)

Tx Stack_to_NIC

TxDMA/PIO

Tx App_to_NIC

NIC-NIC

Separate test with SmartBits Analyzer


NIC/OS Receive Path Measurements

Packet Recv’d

Interrupt

Key:

= start receive marker

Metrics:1.) DMA vs. Programmed I/O

2.) Copy time from staging buffer(receive_buffer) to stack

= timestamp marker (software)

Rx DMA/PIO

Rx_NIC_to_Stack

App receives pkt.

Rx_receive_buffer _to_stack

Stack Process Time (TBD)


Preliminary Performance Results

26.14820.064159.421 7.740TX_STACK_TO_NIC*

23.84917.256144.600 5.100TX_BUFFER_TO_NIC

(PIO or DMA)*

2.523 14.880 1.260

TX_STACK_ PROCESSING**

TX_STACK_TO_STAGING BUFFER*

39.15627.843299.58118.481RX_NIC_TO_STACK*

0.481 2.525 3.360 1.500Rx_BUFF_to_STACK*

39.07325.264296.40016.560Rx_NIC_to_BUFF (PIO or DMA)*

Std.Dev(us)

AvgWidth(us)

MaxWidth(us)

Min.Width(us)

* Numbers are for Programmed I/O (which can be up to ~4 times slower than DMA), using TCP (ftp), MTU sizes vary between 64-1500 bytes

** Numbers are for Programmed I/O, UDP, MTU sizes between 64-1500

TBD TBD TBD TBD

Std.Dev(us)

AvgWidth(us)

MaxWidth(us)

Min.Width(us)

Without LLC With LLC

TBD TBD TBD TBD

TBD TBD TBD TBD

TBD TBD TBD TBD

TBD TBD TBD TBD

TBD TBD TBD TBD

TBD TBD TBD TBD

TBD TBD TBD TBD

2.346


Status

• LLC working in lab (2 millisecond timeout)

• Additional performance testing includes:– LLC

• Bi-directional benchmark

• Worst case footprint of LLC from CPU & network throughput cost

– Fixed packet size (minimum and maximum)

– Linux OS (should be worse due to virtual memory model)


Conclusion

Preliminary tests indicate that:

the onboard network architecture, using LLC for reliability, will meet the 5 millisecond requirement for packet transfer


The Team

Software

Greg Menke

Art Ferrer

Freemon Johnson

Alan Cudmore

Jonathan Wilmot

Jane Marquart

Hardware

Mike Lin

Scott Edfors

For more information contact: Jane Marquart at [email protected]

Building a Reliable Onboard Network with Ethernet: A GSFC Prototype Jane Marquart NASA/GSFC.

Documents

Transcript of Building a Reliable Onboard Network with Ethernet: A GSFC Prototype Jane Marquart NASA/GSFC.