2003 © SWITCH

SWITCHlambda - Experiences with national dark fibers @ SWITCH

Willi Huber / Urs Eppenberger

huber@switch.ch / eppenberger@switch.ch

2003 © SWITCH 2

The SWITCHlambda Project

Project goals:

• Provide fore front network services to the Swiss Universities at low recurring cost

• Network scalable to multiples of 10 Gbps with low additional cost

• Long living infrastructure, no need to change provider every couple of years

• Independence of carrier market

2003 © SWITCH 3

SWITCHlambda Project

Basic decisions:

• Replacement of the STM-1 links by GigabitEthernet connections based on dark fiber

• Long term lease of dark fiber, indefeasible right of use (IRU) for at least 10 years

• Use of DWDM on main trunk lines

• Redundant wide area links

• Simple network design (no SDH, no ATM): Use of Ethernet-Interface all over the place

2003 © SWITCH 4

SWITCHlambda Network Design

• Hierarchy of technologies – DWDM on main trunk lines (16 possible today, expandable if

later needed)– CWDM or single GE to smaller sites off the backbone

• Transparent optical path between Zurich and Geneva– No regeneration between the two main sites of SWITCH

• Optical path on main trunks designed for 10 Gbps / – Chromatic dispersion compensated– Polarization mode dispersion under control

• Bi-directional use of fibers– lower fiber costs

2003 © SWITCH 5

SWITCHlambda Network end of 2003

dark fiber based links (approx. 1200 km)

leased GE link

350 km

Zurich

Bern

Basel

Geneva

Lausanne

2003 © SWITCH 6

Bi-directional Use of Fiber

• Only one fiber used for the DWDM system

• Second fiber free to connect smaller sites inbetween Universities with cheap GE technology

P2P GigabitEthernet-connections

n * 1 GE / 10 GE implemented with DWDM

RouterRouter

One pair of fiber

DWDM node DWDM node

Router

Router

2003 © SWITCH 7

Rx Tx

GigEMux/Demux

Rx Tx

GigEMux/Demux

dispersioncompensator

add/dropcircuit

opticalamplifiers

opticalamplifiers

dispersioncompensator

directionalcoupler

directionalcoupler

to/fromsite C

to/fromsite A

localconnections

site B

Bi-directional transmission mode: i.e. a single fiber is used for signals in both directions, leaving the second fiber free for future use.

Typical SWITCHlambda node

2003 © SWITCH 8

Sorrento Gigamux

2003 © SWITCH 9

Sorrento CWDM

JumpStart-400 CWDM System

– 8 wavelengths (4 receive, 4 transmit)

– Bi-directional operation

– Optional backup path

SWITCH uses currently 1 pair of Jumpstarts for two p2p GE connections over on single fiber.

2003 © SWITCH 10

Bi-directional GigabitEthernet

Cheap solution for p2p GE connection over one single fiber with colored GBICs and passive optical devices

Splitter

OADM

Coloured GBIC

Switch /Router

Splitter

OADM

Coloured GBIC

Switch /Router

Signal reflection (e.g. bad connector)

2003 © SWITCH 11

Testing 10 Gbps/channel capability

Goals• verify equipment capability under real life conditions• verify optical design (dispersion compensation)• verify fiber quality for 10 Gbps application (PMD and

other adverse characteristics)

Realization• DWDM interface cards on loan from Sorrento (10GE

WAN-PHY)• IP equipment from Force10 (2002) and Cisco (2003)• traffic generation with Linux based workstations and

the free software “mgen”

2003 © SWITCH 12

10 Gbps tests

10 Gbps/, 360 km, bi-directional transmission

1. Test with signal regeneration optical - electrical - optical in the middle (at Bern)

2. Test with transparent optical signal Zurich - Geneva

3. Same as 2., but with additional attenuation of 3 dB of optical signal at the sender

Each test: 8 Gbps of traffic during 35 hours, 0 bit errors BER < 8 * 10-15

SWITCHlambda ready for 10 Gbps !

2003 © SWITCH 13

Pros and Cons: Technical View

• Stable, long living infrastructure

• High future potential

• This new stuff is fun

• none

2003 © SWITCH 14

Is this reliable?

• Stable infrastructure with occasionally long interruptions

– Interruptions mainly due to planned maintenance

– True fiber cuts are rare, but last for hours if not days (e.g. fiber on high voltage power lines, fiber along railway)

– Experience of a big Swiss carrier: Fiber cuts happen mostly in the local loop (city)

• Our experiences so far– Several interruptions due to planned maintenance (lasting up

to 1 day)

– Broken optical amplifier (interruption of 4 hours)

» Redundant links are mandatory for a good sleep!

2003 © SWITCH 15

No connection, what now?

• 24/7 monitoring by Ascom Netwatch– ping, SNMP– alarm SWITCH staff– basic analysis

• SWITCH NOC for analysis and action decision• University IT staff

– power problems– physical equipment check

• Onsite support by Ascom Netwatch– localize cable problems with OTDR (optical time domain

reflectometer)– Cisco hardware

• Onsite support by Deltanet for Sorrento DWDM

2003 © SWITCH 16

Pros and Cons: Operational View

• we are the boss • Big effort for negotiations with a lot of contract partners

• no single provider to blame in case of connection loss

• need a car to fix problems instead of telnet

2003 © SWITCH 17

Network Cost, general Remarks

• Cost for wide area links– Well known in advance

• Local loop cost– Vary a lot from place to place

– Good (say cheap) solutions need a lot of planning and patience

– Building own infrastructure is economically better than renting, but not possible in all cities

– In some cities there is a de facto monopole of the city’s industrial works

– The railways are an excellent partner for wide area links: low local loop cost because the schools are mostly located near railway stations

2003 © SWITCH 18

Network Cost, general Remarks (2)

• Cost for optical equipment– DWDM is expensive

– Long 10 Gbps/ DWDM trunks are very expensive (dispersion compensation, amplification)

– CWDM stuff is getting cheap

• Routers / Switches– Ethernet interfaces are cheaper by factors than SONET ones

2003 © SWITCH 19

Cost Structure

Total yearly cost: 2.05 MEURCost include: Investment years 2001-2004

10 Gbps between the major sites maintenance & support

wide area fiber

local loop fiber

DWDM equipment

CWDM equipment

maintenance & support

Depreciation period used for the calculations:Fiber optic infrastructure (wide area & local loop):

10 yearsFiber optic equipment DWDM & CWDM:

5 years 31 %

12 %

6 %

44 %7 %

2003 © SWITCH 20

Comparison with predecessor

SWITCHng SWITCHlambda

Years covered: 1999-2002 2001-2004

Technology: ATM/STM-1 dark fiber, GE & 10 GE

Sites connected: 13 universities 21 universities

Yearly cost: 2.95 MEUR 2.05 MEUR

2003 © SWITCH 21

Isn’t it cheapter to buy it?

We have not done a serious market review lately or even negotiated for capacity to get the real prices

But we have a figure we assume is the best current offer on the market for high capacity links: 100 kEUR

– limited to 300 Mbit/s

– local loop must exist

For 21 sites: 2.1 MEUR

Similar cost as SWITCHlambda,but less flexibility and performance.

2003 © SWITCH 22

Pros and Cons: Economic View

• Low recurring cost

• Low marginal cost for additional bandwidth

• High investment

• Uncertainity about development of bandwidth prices