1 Isp Design

104
© 2010 Cisco Systems, Inc. All rights reserved. ISP Workshops 1 ISP Network Design ISP/IXP Workshops

Transcript of 1 Isp Design

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 1

ISP Network Design

ISP/IXP Workshops

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 2

ISP Network Design

PoP Topologies and Design

Backbone Design

ISP Systems Design

Addressing

Routing Protocols

Security

Out of Band Management

Operational Considerations

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 3

Point of Presence Topologies

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 4

PoP Topologies

Core routers – high speed trunk connections

Distribution routers and Access routers – high portdensity

Border routers – connections to other providers

Service routers – hosting and servers

Some functions might be handled by a single router

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 5

PoP Design

Modular Design

Aggregation Services separated according toconnection speedcustomer servicecontention ratiosecurity considerations

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 6

Modular PoP Design

Backbone linkto another PoP

Backbone linkto another PoP

Leased line customeraggregation layer

for leased line circuit deliveryChannelised circuits

NetworkOperations

Centre

Consumer

DIal Access

NetworkCore

Consumer cable, xDSL and

wireless Access

for MetroE circuit deliveryGigE fibre trunks

MetroE customeraggregation layer

ISP Services(DNS, Mail, News,

FTP, WWW)

Hosted Services &Datacentre

Other ISPsWeb Cache

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 7

Modular Routing Protocol DesignSmaller ISPs

Modular IGP implementationIGP “area” per PoPCore routers in backbone area (Area 0/L2)Aggregation/summarisation where possible into the core

Modular iBGP implementationBGP route reflector cluster per moduleCore routers are the route-reflectorsRemaining routers are clients & peer with route-reflectors only

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 8

Modular Routing Protocol DesignLarger ISPs

Modular IGP implementationIGP “area” per module (but avoid overloading core routers)Core routers in backbone area (Area 0/L2)Aggregation/summarisation where possible into the core

Modular iBGP implementationBGP route reflector cluster per moduleDedicated route-reflectors adjacent to core routersClients peer with route-reflectors only

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 9

Point of Presence Design

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 10

PoP Modules

Low Speed customer connectionsPSTN/ISDN dialupLow bandwidth needsLow revenue, large numbers

Leased line customer connectionsE1/T1 speed rangeDelivery over channelised mediaMedium bandwidth needsMedium revenue, medium numbers

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 11

PoP Modules

Broad Band customer connectionsxDSL, Cable and WirelessHigh bandwidth needsLow revenue, large numbers

MetroE & Highband customer connectionsTrunk onto GigE or 10GigE of 10Mbps and higherChannelised OC3/12 delivery of E3/T3 and higherHigh bandwidth needsHigh revenue, low numbers

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 12

PoP Modules

PoP CoreTwo dedicated routersHigh Speed interconnectBackbone Links ONLYDo not touch them!

Border NetworkDedicated border router to other ISPsThe ISP’s “front” doorTransparent web caching?Two in backbone is minimum guarantee for redundancy

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 13

PoP Modules

ISP ServicesDNS (cache, secondary)News (still relevant?)Mail (POP3, Relay, Anti-virus/anti-spam)WWW (server, proxy, cache)

Hosted Services/DataCentresVirtual Web, WWW (server, proxy, cache)Information/Content ServicesElectronic Commerce

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 14

PoP Modules

Network Operations CentreConsider primary and backup locationsNetwork monitoringStatistics and log gatheringDirect but secure access

Out of Band Management NetworkThe ISP Network “Safety Belt”

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 15

Low Speed Access Module

To Core Routers

Primary Rate T1/E1

PSTN lines tomodem bank

PSTN lines tobuilt-in modems

AS5400

2811

2800/3800

TACACS+/Radiusproxy, DNS resolver,

Content

Web Cache

Access NetworkGateway Routers

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 16

Medium Speed Access Module

To Core Routers

Channelised T1/E1

64K and nx64K circuits

Mixture of channelisedT1/E1, 56/64K and

nx64K circuits

3800/7206/7600

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 17

High Speed Access Module

To Core Routers

Metro Ethernet

Channelised T3/E3

Channelised OC3/OC12

7200/7600/ASR1000/ASR9000

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 18

Broad Band Access Module

To Core Routers

Telephone Network

The cable system

6400

SSG, DHCP, TACACS+or Radius Servers/Proxies,

DNS resolver, Content

Web Cache

Access NetworkGateway Routers

uBR7246

61xx

IP, ATM

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 19

ISP Services Module

DNScache

DNSsecondary POP3 Mail

Relay NEWS

To core routers

WWWcache

Service NetworkGateway Routers

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 20

Hosted Services Module

Customer 7Customer 3Customer 4

Customer 5Customer 6

To core routers

Hosted NetworkGateway Routers

Customer 2Customer 1

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 21

Border Module

To core routers

NetworkBorder Routers

To local IXP -NB - no default route +

local AS routing table only

ISP1 ISP2

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 22

NOC Module

Primary DNS

To core routers

Hosted NetworkGateway Routers

SYSLOGserver

TACACS+server

Network Operations Centre Staff

Out of Band

Management Network

2811/32async

NetFlowAnalyser

Firewall

Billing, Database and Accounting

Systems

Corporate LAN

Critical ServicesModule

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 23

Out of Band Network

Out of Band

Management Network

2811/32asyncTo the NOC

Out of Band Ethernet

NetFlow

Collector

NetFlow

enabled

routers

Routerconsoles

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 24

Backbone Network Design

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 25

Backbone Design

Routed Backbone

Switched BackboneVirtually obsolete

Point-to-point circuitsnx64K, T1/E1, T3/E3, OC3, OC12, GigE, OC48, 10GigE, OC192

ATM/Frame Relay service from telcoT3, OC3, OC12,… deliveryEasily upgradeable bandwidth (CIR)Almost vanished in availability now

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 26

Distributed Network Design

PoP design “standardised”operational scalability and simplicity

ISP essential services distributed around backbone

NOC and “backup” NOC

Redundant backbone links

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 27

Distributed Network Design

POP One

POP Two

POP Three

Customerconnections

Customerconnections

Customerconnections

Externalconnections

Externalconnections Operations Centre

BackupOperations Centre

ISP Services

ISP Services

ISP Services

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 28

Backbone Links

ATM/Frame RelayVirtually disappeared due to overhead, extra equipment, andshared with other customers of the telcoMPLS has replaced ATM & FR as the telco favourite

Leased Line/CircuitMost popular with backbone providersIP over Optics and Metro Ethernet very common in many partsof the world

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 29

Long Distance Backbone Links

Tend to cost more

Plan for the future (at least two years ahead) but stay inbudget

Unplanned “emergency” upgrades can be disruptive withoutredundancy

Allow sufficient capacity on alternative paths for failuresituations

Sufficient can be 20% to 50%Some businesses choose 0% – meaning they have no sparecapacity at all!!

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 30

Long Distance Links

POP One

POP Two

POP Three

Long distance link

Alternative/Backup Path

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 31

Metropolitan Area Backbone Links

Tend to be cheaperCircuit concentrationChoose from multiple suppliers

Think bigMore redundancyLess impact of upgradesLess impact of failures

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 32

Metropolitan Area Backbone Links

POP One

POP Two

POP Three

Metropolitan Links

Metropolitan Links

Traditional Point to Point Links

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 33

ISP Services

DNS, Mail, NewsDesign and Placement

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 34

ISP Services

Most ISP services such as DNS, Mail, etc are easilydeliverable on low budget hardware platforms

Single Rack Unit in height (1RU)Dual processor is “default” nowRAM is very cheap (may as well use 2Gbytes or more)Hard drives are very cheap (SCSI more reliable)Unix like operating systems (FreeBSD, Debian, Ubuntu,CentOS) are very common

In addition to commercial operating systems such as Solaris,RedHat Enterprise Linux &c

Minimal overhead, minimal OS install, plus the service required

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 35

ISP Services:DNS

Domain Name SystemProvides name and address resolutionServers need to be differentiated, properly located and specified

Primary nameserverSecondary nameserverCaching nameserver – resolver

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 36

ISP Services:DNS

Primary nameserverHolds ISP zone files

Forward zone (list of name to address mappings) for allISP’s and any customer zonesReverse zone (list of address to name mappings) for allISP’s address space

Hardware & OS: easily satisfied by simple specificationLocated in secure part of net, e.g. NOC LANUsually run as “hidden master” – secondary nameservers arethe official listed nameservers

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 37

ISP Services:DNS

Secondary nameserverHolds copies of ISP zone filesAt least two are required, more is betterHardware & OS: easily satisfied by simple specificationStrongly recommended to be geographically separate fromeach other and the primary DNS

At different PoPsOn a different continent e.g. via services offered by ISC,PCH and othersAt another ISP

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 38

$ dig apnic.net ns;; ANSWER SECTION:apnic.net. 10800 NS ns1.apnic.net.apnic.net. 10800 NS ns3.apnic.net.apnic.net. 10800 NS ns4.apnic.net.apnic.net. 10800 NS ns5.apnic.com.apnic.net. 10800 NS cumin.apnic.net.apnic.net. 10800 NS ns-sec.ripe.net.apnic.net. 10800 NS tinnie.arin.net.apnic.net. 10800 NS tinnie.apnic.net.

;; ADDITIONAL SECTION:ns1.apnic.net. 3600 A 202.12.29.25ns3.apnic.net. 3600 A 202.12.28.131ns4.apnic.net. 3600 A 202.12.31.140ns5.apnic.com. 10800 A 203.119.43.200cumin.apnic.net. 3600 A 202.12.29.59tinnie.apnic.net. 3600 A 202.12.29.60ns-sec.ripe.net. 113685 A 193.0.0.196tinnie.arin.net. 10800 A 199.212.0.53

TokyoHong KongWashington

Brisbane

ISP Services:Secondary DNS Example

Brisbane

AmsterdamWashington

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 39

ISP Services:Secondary DNS Example

apnic.net zonePrimary DNS in Brisbane (ns1.apnic.net)Secondary DNS run all over the world by APNIC:

BrisbaneHong KongTokyoWashington

Zone secondaried byRIPE NCC in AmsterdamARIN in Washington

Geographical and service provider redundancy – this is theperfect example!

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 40

ISP Services:DNS

Caching nameserverThis is the resolver – it is the DNS cacheYour customers use this as resolver, NEVER your primary orsecondary DNSProvides very fast lookupsDoes NOT secondary any zonesOne, or preferably two per PoP (redundancy)Hardware & OS: easily satisfied by simple specification

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 41

ISP Services:Caching Nameserver

To Core Routers

DIAL network

Web Cache

DNS Cache DNS Cache

Radius proxy

Switch redundancyRouter redundancyDNS Cache redundancy

DIAL users automatically given the IP addresses of DNS cacheswhen they dial in

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 42

GeekAlert

ISP Services:Anycasting the Caching Nameserver

One trick of the tradeassign two unique IP addresses to be used for the two DNS resolver systemsuse these two IP addresses in every PoProute the two /32s across your backboneeven if the two resolver systems in the local PoP aredown, the IGP will ensure that the next nearest resolverswill be reachableKnown as IP Anycast

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 43

ISP Services:DNS

Efficient and resilient designPrimary DNS – keep it secureSecondary DNS – geographical and provider redundancy

Don’t ever put them on the same LAN, switched or otherwiseDon’t put them in the same PoP

Caching DNS – one or two per PoPReduces DNS traffic across backboneMore efficient, spreads the load

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 44

ISP Services:DNS

SoftwareMake sure that the BIND distribution on the Unix system is up todate

The vendor’s distribution is rarely currentPay attention to bug reports, security issuesReboot the DNS cache on a regular (e.g. monthly) basis

Clears out the cacheReleases any lost RAMAccepted good practice by system administrators

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 45

ISP Services:DNS

ImplementationPut all your hosts, point-to-point links and loopbacks into theDNS

Under your ISP’s domain nameUse sensible/meaningful names

Put all your hosts, point-to-point links and loopbacks into theREVERSE DNS also

Don’t forget about in-addr.arpa and ip6.arpa – many ISPs doSome systems demand forward/reverse DNS mappingbefore allowing access

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 46

ISP Services:Mail

Must have at least two mail hosts (MX records) for all supporteddomains

Geographical separation helps

Dedicated POP3 serverConsumers/mobile users get mail from here

SMTP gateway dedicated to that functionConsumers/mobile users send mail via here

Mail relay open to CUSTOMERS only!Don’t let outside world use your mail relay

Block port 25 outbound for all customersInsist that outbound e-mail goes through SMTP relaySMTP relay does virus (ClamAV) and spam (Spamassassin) filtering

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 47

ISP Services:Mail Configuration

smtp.isp.net

Customer mail relay

Incoming mailfrom customer

mail.isp.net

ISP Mail Gateway

Incoming mailfrom Internet

pop3.isp.net

Customer POP3/IMAP server

Mail pulled bycustomer client

Mail out tothe Internet

SpamAssassinClamAV

SpamAssassinClamAV

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 48

$ dig cisco.com mx

;; ANSWER SECTION:cisco.com. 86400 MX 10 sj-inbound-a.cisco.com.cisco.com. 86400 MX 10 sj-inbound-b.cisco.com.cisco.com. 86400 MX 10 sj-inbound-c.cisco.com.cisco.com. 86400 MX 10 sj-inbound-d.cisco.com.cisco.com. 86400 MX 10 sj-inbound-e.cisco.com.cisco.com. 86400 MX 10 sj-inbound-f.cisco.com.cisco.com. 86400 MX 15 rtp-mx-01.cisco.com.cisco.com. 86400 MX 20 ams-inbound-a.cisco.com.cisco.com. 86400 MX 25 syd-inbound-a.cisco.com.

ISP Services:Mail Example

cisco.com mail (MX records)primary MX are 6 systems in San JoseThree backup MXes in RTP, Amsterdam and Sydneybackup MX only used if primary unavailable

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 49

IMPORTANT: Do NOT allow non-customers to use your mail system as a relay

ISP Services:Mail

SoftwareMake sure that the MAIL and POP3 distributions on the Unixsystem are up to date

The vendor distributions are rarely currentPay attention to bug reports, security issues, unsolicited junkmail complaints

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 50

ISP Services:News

News servers provide a Usenet news feed to customers

Distributed design requiredIncoming newsfeed to one large serverDistributed to feed servers in each PoPFeed servers provide news feed to customersOutgoing news goes to another serverSeparate reading news systemSeparate posting news system

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 51

ISP Services:News System Placement

POP One

POP Two

POP Three

Customerconnections

Customerconnections

Customerconnections

Externalconnections

Externalconnections News Collector

News Feeder

News Feeder

News Feeder

News Distributor

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 52

ISP Services:News System Placement

POP One

POP Two

POP Three

Customerconnections

Customerconnections

Customerconnections

Externalconnections

Externalconnections News Collector

News Feeder

News Feeder

News Feeder

News Distributor

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 53

IMPORTANT: Do NOT allow non-customers to use your news system for posting messages

ISP Services:News

SoftwareMake sure that the Internet News distribution on the Unixsystem is up to date

The vendor distributions are rarely currentPay attention to bug reports, security issues, unsolicited junkposting complaints

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 54

Addressing

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 55

Where to get IP addresses and AS numbers

Your upstream ISP

AfricaAfriNIC – http://www.afrinic.net

Asia and the PacificAPNIC – http://www.apnic.net

North AmericaARIN – http://www.arin.net

Latin America and the CaribbeanLACNIC – http://www.lacnic.net

Europe and Middle EastRIPE NCC – http://www.ripe.net/info/ncc

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 56

Internet Registry Regions

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 57

Getting IP address space

Take part of upstream ISP’s PA spaceor

Become a member of your Regional Internet Registryand get your own allocation

Require a plan for a year aheadGeneral policies are outlined in RFC2050, more specific detailsare on the individual RIR website

There is still plenty of IPv4 address spaceRegistries require high quality documentationWhen applying for IPv4 addresses, get an IPv6 allocation too!

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 58

What about RFC1918 addressing?

RFC1918 defines IP addresses reserved for privateInternets

Not to be used on Internet backboneshttp://www.ietf.org/rfc/rfc1918.txt

Commonly used within end-user networksNAT used to translate from private internal to public externaladdressingAllows the end-user network to migrate ISPs without a majorinternal renumbering exercise

Most ISPs filter RFC1918 addressing at their networkedge

http://www.cymru.com/Documents/bogon-list.html

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 59

What about RFC1918 addressing?

List of well known problems with this approach for an SP backbone:Breaks Path MTU DiscoveryPotential conflicts with usage of private addressing inside customernetworksSecurity through obscurity does not provide securityTroubleshooting outside the local network becomes very hard

Router interface addresses are only locally visibleInternet becomes invisible from the router

Troubleshooting of connectivity issues on an Internet scale becomesimpossible

Traceroutes and pings provide no informationNo distinction between “network invisible” and “network broken”

Increases operational complexity of the network infrastructure and routingconfiguration

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 60

Private versus Globally Routable IPAddressing

Infrastructure Security: not improved by using privateaddressing

Still can be attacked from inside, or from customers, or byreflection techniques from the outside

Troubleshooting: made an order of magnitude harderNo Internet view from routersOther ISPs cannot distinguish between down and broken

Performance: PMTUD breakage

Summary:ALWAYS use globally routable IP addressing for ISPInfrastructure

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 61

Addressing Plans – ISP Infrastructure

Address block for router loop-back interfaces

Address block for infrastructurePer PoP or whole backboneSummarise between sites if it makes senseAllocate according to genuine requirements, not historic classfulboundaries

Similar allocation policies should be used for IPv6 aswell

ISPs just get a substantially larger block (relatively) soassignments within the backbone are easier to make

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 62

Addressing Plans – Customer

Customers are assigned address space according toneed

Should not be reserved or assigned on a per PoP basisISP iBGP carries customer netsAggregation not required and usually not desirable

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 63

223.10.0.0/21

Customer assignments Infrastructure Loopbacks

/24223.10.6.255223.10.0.1

223.10.0.0/20

Original assignments New Assignments

/24/24223.10.0.1

223.10.5.255 223.10.15.255

Addressing Plans – ISP Infrastructure

Phase One

Phase Two

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 64

Addressing PlansPlanning

Registries will usually allocate the next block to becontiguous with the first allocation

Minimum allocation could be /21Very likely that subsequent allocation will make this up to a /20So plan accordingly

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 65

Addressing Plans (contd)

Document infrastructure allocationEases operation, debugging and management

Document customer allocationContained in iBGPEases operation, debugging and managementSubmit network object to RIR Database

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 66

Routing Protocols

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 67

Routing Protocols

IGP – Interior Gateway Protocolcarries infrastructure addresses, point-to-point linksexamples are OSPF, ISIS, EIGRP...

EGP – Exterior Gateway Protocolcarries customer prefixes and Internet routescurrent EGP is BGP version 4

No connection between IGP and EGP

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 68

Why Do We Need an IGP?

ISP backbone scalingHierarchyModular infrastructure constructionLimiting scope of failureHealing of infrastructure faults using dynamic routing with fastconvergence

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 69

Why Do We Need an EGP?

Scaling to large networkHierarchyLimit scope of failure

PolicyControl reachability to prefixesMerge separate organizationsConnect multiple IGPs

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 70

Interior versus Exterior RoutingProtocols

InteriorAutomatic neighbour discoveryGenerally trust your IGProutersPrefixes go to all IGP routersBinds routers in one AStogether

ExteriorSpecifically configured peersConnecting with outsidenetworksSet administrative boundariesBinds AS’s together

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 71

Interior versus Exterior RoutingProtocols

InteriorCarries ISP infrastructureaddresses onlyISPs aim to keep the IGPsmall for efficiency andscalability

ExteriorCarries customer prefixesCarries Internet prefixesEGPs are independent of ISPnetwork topology

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 72

BGP4

Hierarchy of Routing Protocols

BGP4and OSPF/ISIS

Other ISPs

CustomersIXP

Static/BGP4

BGP4

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 73

Routing Protocols:Choosing an IGP

Review the “Introduction to Link State Protocols”presentation

i.e. – OSPF and ISIS have very similar properties

ISP usually chooses between OSPF and ISISChoose which is appropriate for your operators’ experienceIn IOS, both OSPF and ISIS have sufficient “nerd knobs” totweak the IGP’s behaviourOSPF runs on IPISIS runs on infrastructure, alongside IP

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 74

Routing Protocols:IGP Recommendations

Keep the IGP routing table as small as possibleIf you can count the routers and the point to point links in thebackbone, that total is the number of IGP entries you shouldsee

IGP details:Should only have router loopbacks, backbone WAN point-to-point link addresses, and network addresses of any LANshaving an IGP running on themStrongly recommended to use inter-router authenticationUse inter-area summarisation if possible

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 75

Routing Protocols:More IGP recommendations

To fine tune IGP table size more, consider:Using “ip unnumbered” on customer point-to-point links – savescarrying that /30 in IGP

(If customer point-to-point /30 is required for monitoring purposes,then put this in iBGP)

Use contiguous addresses for backbone WAN links in eacharea – then summarise into backbone areaDon’t summarise router loopback addresses – as iBGP needsthose (for next-hop)Use iBGP for carrying anything which does not contribute to theIGP Routing process

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 76

Routing Protocols:iBGP Recommendations

iBGP should carry everything which doesn’t contributeto the IGP routing process

Internet routing tableCustomer assigned addressesCustomer point-to-point linksDial network pools, passive LANs, etc

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 77

Routing Protocols:More iBGP Recommendations

Scalable iBGP features:Use neighbour authenticationUse peer-groups to speed update process and for configurationefficiencyUse communities for ease of filteringUse route-reflector hierarchy

Route reflector pair per PoP (overlaid clusters)

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 78

Security

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 79

Security

ISP Infrastructure security

ISP Network security

Security is not optional!

ISPs need to:Protect themselvesHelp protect their customers from the InternetProtect the Internet from their customers

The following slides are general recommendationsDo more research on security before deploying any network

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 80

ISP Infrastructure Security

Router securityUsernames, passwords, vty filters, TACACS+Disable telnet on vtys, only use SSHvty filters should only allow NOC access, no external accessSee IOS Essentials for the recommended practices for ISPs

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 81

ISP Infrastructure Security

ISP server securityUsernames, passwords, TCP wrappers, IPTABLESProtect all servers using routers with strong filters applied

Hosted services securityProtect network from hosted servers using routers with strongfiltersProtect hosted servers from Internet using routers with strongfilters

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 82

DNScache

DNSsecondary POP3 Mail

Relay NEWS

To core routers

Service NetworkGateway Routers

Access-list examples:

Allow tcp/established to all serversICMPDNS 2ary: udp/53 and tcp/53POP3: tcp/110Mail Relay: tcp/25 and ISP address

range onlyNews: tcp/119 and ISP

address range onlyDNS Cache: udp/53Web server: tcp/80

Other necessary filters:

All servers: SSH (tcp/22) from NOC LAN only

Webserver

ISP Infrastructure SecurityISP Server Protection

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 83

Access-list examples:InboundAllow tcp/established to all serversICMPWeb server: tcp/80SSH for customer accessAny other ports for services

sold to customers

OutboundICMPAllow DNS udp/53 and

tcp/53Block all access to ISP

address rangeServer5Server1 Server2 Server3 Server4

To core routers

Service NetworkGateway Routers

Server6

ISP Infrastructure SecurityHosted Server Protection

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 84

ISP Infrastructure Security

Premises securityLocks – electronic/card key preferredSecure access – 24x7 security arrangementsEnvironment control – good aircon

Staff responsibilityPassword policy, strangers, temp staffEmployee exit procedures

RFC2196(Site Security Handbook)

RFC3871(Operational Security Requirements for Large ISP IP NetworkInfrastructure )

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 85

ISP Network Security

Denial of Service Attackseg: “smurfing”see http://www.denialinfo.com

Effective filteringNetwork borders – see Cisco ISP EssentialsCustomer connections – unicast RPF on ALL of themNetwork operation centreISP corporate network – behind firewall

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 86

ISP Network SecuritySecure external access

How to provide staff access from outsideSet up ssh gateway (Unix system with ssh daemon and nothingelse configured)Provide ssh client on all staff laptopsssh available on Unix and Windowsssh is Secure Shell – encrypted link

How not to provide access from outsidetelnet, rsh, rlogin – these are all insecureOpen host – insecure, can be compromised

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 87

Ingress & Egress Route Filtering

Your customers should not besending any IP packets out to the

Internet with a source addressother then the address you have

allocated to them!

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 88

Out of Band Management

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 89

Out of Band Management

Not optional! Allows access to network equipment in times of failure

Ensures quality of service to customersMinimises downtimeMinimises repair timeEases diagnostics and debugging

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 90

Out of Band Management

OoB Example – Access server:modem attached to allow NOC dial inconsole ports of all network equipment connected to serial portsLAN and/or WAN link connects to network core, or via separatemanagement link to NOC

Full remote control access under all circumstances

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 91

Out of Band Network

Ethernetto the NOC

Router, switchand ISP server

consoles

(Optional) Out of bandWAN link to other PoPs

Modem – accessto PSTN for out of

band dialin

Equipment RackEquipment Rack

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 92

Out of Band Management

OoB Example – Statistics gathering:Routers are NetFlow and syslog enabledManagement data is congestion/failure sensitiveEnsures management data integrity in case of failure

Full remote information under all circumstances

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 93

Test Laboratory

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 94

Test Laboratory

Designed to look like a typical PoPOperated like a typical PoP

Used to trial new services or new software underrealistic conditions

Allows discovery and fixing of potential problems beforethey are introduced to the network

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 95

Test Laboratory

Some ISPs dedicate equipment to the lab

Other ISPs “purchase ahead” so that today’s labequipment becomes tomorrow’s PoP equipment

Other ISPs use lab equipment for “hot spares” in theevent of hardware failure

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 96

Test Laboratory

Can’t afford a test lab?Set aside one spare router and server to trial new servicesNever ever try out new hardware, software or services on thelive network

Every major ISP in the US and Europe has a test labIt’s a serious consideration

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 97

Operational Considerations

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 98

Operational Considerations

Why design the world’s best networkwhen you have not thought about whatoperational good practices should be

implemented?

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 99

Operational ConsiderationsMaintenance

Never work on the live network, no matter how trivialthe modification may seem

Establish maintenance periods which your customers are awareof

e.g. Tuesday 4-7am, Thursday 4-7am

Never do maintenance on a FridayUnless you want to work all weekend cleaning up

Never do maintenance on a MondayUnless you want to work all weekend preparing

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 100

Operational ConsiderationsSupport

Differentiate between customer support and theNetwork Operations Centre

Customer support fixes customer problemsNOC deals with and fixes backbone and Internet relatedproblems

Network Engineering team is last resortThey design the next generation network, improve the routingdesign, implement new services, etcThey do not and should not be doing support!

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 101

Operational ConsiderationsNOC Communications

NOC should know contact details for equivalent NOCsin upstream providers and peers

Or consider joining the INOC-DBA systemVoice over IP phone system using SIPRuns over the Internetwww.pch.net/inoc-dba for more information

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 102

ISP Network Design

Summary

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 103

ISP Design Summary

KEEP IT SIMPLE & STUPID ! (KISS)

Simple is elegant is scalable

Use Redundancy, Security, and Technology to makelife easier for yourself

Above all, ensure quality of service for your customers

© 2010 Cisco Systems, Inc. All rights reserved.ISP Workshops 104

ISP Network Design

ISP/IXP Workshops