Ben Teitelbaum <[email protected]>January 25th, 2001TEQUILA Workshop on Internet Design for SLS Delivery
Service Specification and TE for the QBone
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 2
architecture
deployment
How We Got Here (short version)
• Began chanting: “enable advanced applications,…”
• Assessed requirements• Recommended DiffServ• Selected “Premium” service to meet demands of loss/jitter sensitive apps
• Charted QBone initiative • Specified QBone architecture• Now proceeding to implement it and tweak the architecture
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 3
IntServ/RSVP
IntServ/RSVP vs DiffServ
BB BB
•Per-flow service state at every hop
•Abstract/manage each cloud’s resources (BBs) •Packets colored to indicate forwarding “behavior”•Focus on aggregates not individual flows•Policing at edge to get services
•Scalability problems
DiffServ
•Focus on multipoint multicast
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 4
DiffServ Overview
Applications contract for specific QoS profiles • Policing at network periphery• “Color” packets with a few simple, differentiated per-hop forwarding behaviors (PHBs)
–Indicated in packet header–Applied to PHB traffic aggregates
• PHBs + policing rules = range of services
DS domains contract with each other for aggregate QoS traffic profiles
• Policing at cloud-cloud boundary• Supports simple, bilateral business agreements
Exploits edge/core distinction for scalability
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 5
Example Service #1: Premium
Assurance: like a leased line
PHB: Expedited Forwarding (RFC 2598)• EF in separate queue configured with minimum departure rate
• Example mechanisms: strict priority, MDRR, WFQ
Policing: police to a specified peak rate and drop out-of-profile packets; effectively a leaky bucket with depth 1 MTU
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 6
Example Service #2: Controlled Load
Assurance: network looks “lightly-loaded” for conforming traffic
PHB: Assured Forwarding (RFC 2597)• 4 independent AF classes• 3 drop preference levels within each class• Example mechanisms: WRED, WFQ
Policing: police to specified rate and burst profile, remarking out-of-profile packets to have higher drop probability
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 7
Example Service #3: CoS
Assurance: “better than Joe”
PHB: “drop the lower classes first ” (AF or class selector PHBs)
Policing: could be based on anything (e.g. higher priority for the CEO)
A.K.A.“Olympic” classes of BE service (e.g. Gold, Silver, Bronze)
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 8
QBone Architecture
A Service: QBone Premium Service• Built on Expedited Forwarding (EF) (RFC 2598)• Assurance: near-zero loss & low, bounded jitter for marked traffic conforming to a specified peak rate
–a.k.a. “virtual leased line”, “virtual wire”
Reservation Setup Protocol• Now: long-lived, manual setup• Proposed: SIBBS protocol between QBone domains; RSVP end-to-end between hosts
QBone Measurement Architecture• Uniform collection of QoS metrics• Uniform dissemination interface
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 9
QBone Measurement Architecture1/2
Collection metrics, EF and BE...• Active metrics (paths)
– One-way delay-variation
– One-way loss
– Traceroutes
– e.g. IPPM Surveyors
• Passive metrics (interfaces)– Load
– EF reservation load
– Discards (suggested)
– Link bandwidths (suggested)
– e.g. OCxMon, RTFM, MIBs
Boundary Router
Intra-Domain Premium Path
Inter-Domain Premium Path
ActiveMeasurements
MIB-basedstatistics
PassiveMeasurements
PassiveMeasurements
PM node PM node
AM node
QBone Domain2
QBone Domain1
QBone Domain3
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 10
QBone Measurement Architecture2/2
Dissemination • Standard URL query syntax:
• whois server to learn canonical names for QBone domains, routers, sniffers, etc
label ::= <alphanum> { <alphanum> }router ::= label"-ROUTER”probe ::= label"-PROBE”sniffer ::= label "-SNIFFER”host ::= router | probe | snifferpath ::= host "/" hostPHB ::= "BE" | "EF”metric ::= "LOSS" | "ONEWAY" | "PING" | "IPDV" | "LOAD" | "TRACEROUTE" | "COMMITMENT" | "RESERVATION"year ::= digit digit digit digitmonth ::= "01" | "02" | ... | "12”day ::= "01" | "02" | ... | "31”YYYYMMDD ::= year month dayaggregation ::= <unsigned_integer>prefix ::= “http” | “ftp” | <other>query ::= prefix ”://" path "/" PHB "/" metric "/” YYYYMMDD "/" metric "/" aggregation
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 11
QBone E2E Picture
GigaPoPA
CampusA
CampusB
GigaPoPB
CampusC
CampusD
Backbone
X Kbps of QPS from hither to………..yon
Key
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 12
SC2000 Interdomain QoS DemoNovember 6-9, 2000
Premium service over two wide-area paths• LBNL-ESnet-Abilene-SCinet-Internet2 booth• Stanford-CalREN2-Abilene-SCinet-Internet2 booth
Congestion induced at multiple points
CD-quality interactive audio application shown with/without QoS
ESnet and Abilene QoS capabilities will form nucleus of QBone
SC2000 Network Challenge Winner: Award for "Most Captivating and Best Tuned" Demo
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 13
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 14
Project Overview
Internet2 & NetworkResearch
Internet2 QoS
Abilene QoS
Future Directions
Questions
Abilene QoS...
0
15
30
45
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 15
Abilene TopologyJanuary, 2001
•47 connectors •183 participants
•34 connections to 20 peer networks
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 16
Miami
NYC (60 Hudson)
LA
AbileneSingaren
SEATTLE
NYC (Telehouse
25 Broadway)
STARTAP
ArgentinaBrazilChileColumbiaBrazil
Developing International Peering
Courtesy: Linda Winkler, STAR TAP
CERNSINET
AbileneESnetvBNS
JAnetSURFnetNORDUnet
AbileneESnetCA*net3AARnetCERNET
CUDIHARNETSINET
AbilenevBNSESnetDRENNRENNISN
CA*net3SURFnetMIRnetNORDUnetREUNA
IUCCAPANCERNSINETSingarenTanet
GEMNETRenater
DFNINFNDANTECA*net3
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 17
Abilene Load Snapshot
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 18
APS Participation
Goal:• Make APS a reference implementation of the QBone architecture
Current Participants• MAGPI (U. Penn)• iCAIR• PSC (Penn State)• OARNet (Ohio State)• NASA EOS
Others in the wings• TF-NGN (through DANTE)• MIRnet• Various other international
• ANL• UIUC• DOE Science GRID (peering transit network)
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 19
Initial Engineering Plan (obsolete)
Measurement
Edge Policing
Manual Setup
EF Core Forwarding
Automated Setup
EF Edge Forwarding
Shaping
Sweetwater Midland Odessa Pecos
(GTS)
(Surveyor + SNMP + HTTP)
(“Firehose” CAR)
(Whiteboard + CLI)
(MDRR)
(MDRR)
(BB)
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 20
APS Measurements
Status: • Collecting EF/BE loads and CAR conform/exceed stats• Not currently monitoring IPDV, but Abilene Surveyor nodes now OC-3 connected and operational
• Ohio-ITEC hosting APS measurements and QBone-wide whois server
Near Future: • IPDV along edge-to-edge QBone paths• Abilene Surveyor timing improvements
–Better NTP–New CDMA timing sources (can't get GPS in Qwest POPs)
• Collection of AS-level traffic matrices
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 21
Committed Access Rate (CAR)
Classifies traffic based on certain matching criteria (interface, DSCP, or ACL) and meters it to a leaky bucket traffic profile
Depending on metering result, different actions applied (drop, transmit, set DSCP,…)
Syntax:rate-limit {input | output} [access-group [rate-limit] acl-index] bps burst-normal burst-max conform-action action exceed-action action
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 22
CAR Experience
For the most part, CAR is exactly what the DiffServ doctor ordered
However, there are some limitations…• PPS performance cost• Quirky constraints on token bucket depths• Not easy to do "virtual trunk" style classification
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 23
CAR Limitation 1: Performance
Problem:• On E0 GSR edge cards, lack of ASIC support for CAR results in non-trivial pps hit
Solution:• Really out of our hands; must wait for E3 edge cards, which will have CAR in hardware
• Load on access interfaces is still light, so performance hit not a big issue for current participants
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 24
CAR Limitation 2: Policing Granularity
Problem:• Cisco CAR Doc:
–“burst-normal Normal burst size in bytes. The minimum value is bps divided by 2000”
• i.e. burst-normal max (mtu, bps/2000)• But, QPS demands token bucket depth of 1 MTU!• Implication: bps 3Mbps (for Ethernet MTU)
Solution:• Again, out of our hands• Have raised concern with Cisco and are hoping E3 cards will address this limitation too
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 25
Solution: with additional rope,CAR can also classify by qos-group (Cisco proprietary)
• Packets assigned to QoS groups through QoS Policy Propagation via BGP (QPPB)
CAR Limitation 3: Classification
Problem: we want...
…is what we have
("virtual trunk")
("firehose")But this...
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 26
Abilene Architecture Limitation: “Porous” Edge Problem
DoQoS problem with current architecture
MDRR (EF forwarding) on interior interfaces easily subverted by unpoliced connectors
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 27
How to “Crisp” the Edge?
Problem: EF requires that all connectors be policed
Solution:• Short term: Stochastically detect illegal EF traffic with NetFlow and/or OCxMon passive monitoring; gigaPoP would be asked to police
• Longer term: Wait for E3 edge cards, deploy them aggressively, and police everywhere
• Road not taken:Re-write all non-participant traffic with DSCP 000000 (using PIRC hack); need to pass some DSCP values (reason why coming up…)
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 28
Current Engineering Plan
Measurement
Edge Policing
Manual Setup
EF Core Forwarding
Automated Setup
EF Edge Forwarding
Shaping
Sweetwater Midland Odessa Pecos
(GTS)
(Surveyor + SNMP + HTTP + whois + traffic matrices)
(CAR + QPPB)
(Whiteboard + CLI)
(MDRR)
(MDRR)
(SIBBS? + DSTE?)
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 29
Project Overview
Internet2 & NetworkResearch
Internet2 QoS
Abilene QoS
Future Directions
Questions
Future Directions...
0
15
30
45
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 30
QBone Premium Service Outlook
Good News:• DiffServ functionality in most modern routers• Many hosts support QoS signaling • Lots of isolated testbed trials• Some partial backbone implementations
Bad News:• Deploying QPS "requires upgrading the world"• Low demand for QPS (app/net chicken/egg dynamic)• Vendor implementations don’t always live up to hype• Requires a lot of elbow grease, which is in short supply in most campus IT organizations
Bottom line:• Progress can be made towards end-to-end real-time services (QPS), but it is going to be slow
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 31
Looking forward...
QBone Premium• Revise/complete architecture (joint work with TF-NGN)• Make measured progress on deployment• Focus on ad hoc QoS solutions where it needed, but deploy in a way consistent with QPS architecture
Big measurement push• QBone measurement architecture• Broader Internet2 measurement push• E2E Performance Initiative
Reap lower-hanging fruits of DiffServ• Internet2 Scavenger Service (I2SS)
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 32
Internet2 Scavenger Service
Basic idea• A lower priority class of best-effort• Voluntary marking hints to network that degraded service is OK (think of a "nice" for the network)
Intended uses• Non-time-critical traffic (e.g. server-to-server NNTP, anonymous FTP, network backups)
• Bulk data transfers using TCP• Non-mission applications (e.g. Napster, games, etc)• New kinds of distributed applications that attempt to use idle network capacity
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 33
I2SS Service Specification
Rigorously defining the E2E I2SS is difficult!• Hoping to define relative to best effort
I2SS traffic indicated by DSCP 001000• Modification of class selector PHB• Note that the I2SS codepoint has global significance
I2SS domain requirements• Traffic leaving must be marked I2SS, if it entered so
Router requirements (all SHOULDs)• Forward I2SS independently giving it a lower probability of timely forwarding OR forward in same manner as BE
• Offer I2SS a very small minimum departure rate• Offer I2SS all used bandwidth
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 34
Project Overview
Internet2 & NetworkResearch
Internet2 QoS
Abilene QoS
Future Directions
Questions
Any Questions?
0
15
30
45
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 35
For more information...
Internet2 Home:• http://www.internet2.edu/
Internet2 QoS Working Group Home:• http://www.internet2.edu/wg/qos/
QBone Home:• http://qbone.internet2.edu/
Abilene Premium Service Home:• http://www.internet2.edu/abilene/qos/
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 36
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 37
What is ''QBone"?
Internet2 effort to build an interdomain diff-serv testbed
Service objective: Premium/VLLOther objectives: thorough instrumentation
and open dissemination of measurements Implementation status: embryonic
• Abilene: PSC (Penn State), UIUC, iCAIR, OARNet (Ohio State), ANL, MAGPI (U. Penn), NASA (Ames and Goddard)
• ESnet: ANL, LBNL
We hope to be a place where experience with QoS SLSs is gained and shared!
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 38
SLS Basics
An SLS is the technical component of an SLA
SLSs are bilateral and private
An SLS is not a reservation
SLSs can be arbitrarily weird; for example...• Per {sourceContinent, destContinent} RTT bounds• All VLL requests under 1 Mbps will be rejected• 100 Mbps VLL to MIT OK with 7 days notice
SLA
SLS Contractual Goop
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 39
Our Solution
Define services and reservations; punt on SLSs
Services • Globally well-known and defined• Include many parameters from draft-tequila-sls-00.txt (scope, traffic envelope, performance guarantees, etc)
Reservations• Identify a service ID• Specify service parameters
Treat SLS as black box: can this reservation be extended across the peering; yes or no?
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 40
Simple Interdomain Bandwidth Broker Signalling (SIBBS)
BB BB BB
H H
DS Domain1 DS Domain3DS Domain2
S LS1-2 S LS2-3
R RRR
Control
Data
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 41
SIBBS - Simple Interdomain Bandwidth Broker Signaling
Basic Idea• Simple protocol for one QBone domain to request of another QBone domain an increase or decrease in an aggregate reservation of a globally well-known service
Design Goals• Simple & extensible• New protocol for now; looking at mapping to COPS
Status• Protocol draft nearing completion (see: http://qbone.ctit.utwente.nl/BBroker/bboutline2.html)
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 42
BB Conceptual Model
Major Blocks• Intra-domain protocol• Inter-domain protocol• User interface• Network Management
Interface• Routing interface• Policy functions• Local resource and
configuration data
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 43
SIBBS: Basic Reservation Setup
BB
BB
BB
RTR ES RTR RTR RTR ES
RAR RAR
RAA RAA
Web interface RSVPDIAMETERCOPSRYO
CLISNMPCOPS
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 44
DRAFT SIBBS RAR
SIBBS version RAR ID
Sender ID Sender Signature
Source Prefix Destination Prefix
Ingress Router ID Start Time
Stop Time Flags
GWS ID Service Paramter TLV
Other (Optional) TLVs
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 45
Core Tunnels &Virtual Peerings
BB BB BB SIBBS SIBBS
BB BB BB
Aggregate Resevation Src RTR RTR Dest
Virtual Peering
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 46
Conclusions
Limited diff-serv implementation experiences
Even less experience with diff-serv peering
Need to specify a few internet-wide services
TEQUILA draft could be recast as a framework for specifying services
Formal specification language for SLSs is probably premature
Automated negotiation of SLSs is premature
Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 47
For More Info...
http://qbone.internet2.edu/architecture.shtml
Top Related