Operations & AutomationUpdateMore-IP USA

Kostas (Konstantinos) KoutalisKonstantinos.koutalis@ams-ix.net

June 17th 2016

Incoming tickets

l 580 Provision & Support

l approx.100 less than last year

00.5

11.5

22.5

33.5

2015-6 2015-10 2016-2

Messages per day

Average per day

Outside NL office hours (including weekends)US (17:00 – 01:00, GMT+2)

0%

20%

40%

60%

80%

100%

2015-6 2015-10 2016-2

Timezone distribution

Europe (9:00 – 17:00, GMT+2)

Asia/pacific (01:00 – 09:00, GMT+2)

US (17:00 – 01:00, GMT+2)

AMS-IX US NOC Announced Tickets

Planned maintenances 9

Non service affecting incident 3

Total Tickets 12

AMS-IX Tickets to Tech-L

AMS-IX US NOC & Operations

l AMS-IX NOC is a team of network engineers based in NL

o Standby duty

o Operation / Troubleshooting

o Customer provisioning

o Software upgrades

o Platform expansion and evolution

l Experience in simple configurations

AMS-IX US NOC & Operations

l & more complex configurations

AMS-IX US NOC & Operations

10GEPE router

EQX AM3PE router

NIKHEFPE router

Global SwitchPE router

Interxion AM3PE router

TC AMS4PE router

EvoswitchPE router

EQX AM1PE router

TC AMS2PE router

SARAPE router

TC AMS5PE router

PXC

CE

CECE

P Router EUNP Router

PXC

CE

PXC

CE

PXC

CE

PXC

CE

PXC

CE

PXC

CE

PXC

CE

PXC

CE

PXC

CE

PXC

CE

PXC

CE

TC AMS2PE router

10GEPE router

10GEPE router

10GEPE router

10GEPE router

10GEPE router

EQX AM1PE router

10GEPE router

TC AMS5PE router

TC AMS2PE router

CECE CE CECE CECE CE

10GEPE routerEunetworks

PE router

GSP Router

TC5 P Router

EQX AM1P Router

Global SwitchPE router

NIKHEFPE router

EQX AM3PE router

EvoSwitchPE router

Interxion AMS3PE router

TC4PE routerTC AMS4

PE routerEQX AM1PE router

TC AMS2PE router

TC AMS2PE router

SARAPE router

TC AMS5PE router

GSP Router

EUNP Router

l Administering and provisioning in these scales is not easy

l Automate … & set-up a 24/7/365 First Line NOC!

• The (very) old days:

• Manual labor

• Could take hours for 1 customer

• The (not so) old days:

• CLI scripts, one at a time

• Some functions not there (e.g. testing)

AMS-IX US NOC & Operations

AMS-IX US Automationl Provisioning an AMS-IX customer

AMS-IX US Automationl In more detail

AMS-IX US Automationl Even more details

Request

Customer portal

Control Server

REST API

XMLProvision

scriptsProvision

queueSQL

Network Equipment

AMS-IX US Automation

Under the hood:

• Ruby on rails, xml & PostgreSQL database

• SNMPo Query port nameo Port statuso Port counterso etc.

• CLI, Perl scriptso Perform configuration taskso A lot of checks based on “expect” scripts

AMS-IX US AutomationWhere are we now?

Available tools:– Perform all standard provisioning tasks (NOC & Customer Services)

• Reserve, assign a port• Test a link• Bring it to production

Tools available to customers / resellers (self provisioning):• Change mac address• Change port from untagged to tagged (qtag id)• Enable/disable v4/v6 BGP peering with admin ASN & Route Servers• Resellers: move virtual links from one reseller link to another

AMS-IX US AutomationWhere are we now?

– 24x7 NOC & AMS-IX US NOC debug information• Configured services (VLANs)• Switch configuration• Port status, optic light levels• Status of BGP sessions with route servers & AMS-IX routers• 10/100GE customers, see connected PE (primary / secondary)• See L2 violations & link flap logs

– 24x7 NOC & AMS-IX US NOC basic operations• Move connections from primary PE to secondary (10/100GE specific)• Unblock BGP sessions with route servers• ASN change• Change BGP prefix limits on route servers• Move a link from production to quarantine and vice versa (WIP)• Disable, Enable a link (WIP)

AMS-IX US AutomationAre we there yet?

• Continuous improvement

• Megaport - AMS-IX partnership

– More automations needed!

– IXP customers è Megaport Services

AMS-IX US Automation• Ordering a Megaport service via AMS-IX

AMS-IX US Automation• Megaport Customers è AMS-IX Services

AMS-IX AutomationHow good is all this?

– It works… for AMS-IX– We need an abstract version of “My AMS-IX”– Project kicked off late 2015

User Request

Customer portal

Control Server

REST API

XMLProvision

scriptsProvision

queueSQL

Network Equipment

AMS-IX AutomationHow good is all this?

– Remove hard coded values– Simplify provisioning process– Remove XML dependencies– Vendor independent

Setting up CHN-IX & conclusions– Different vendor being used added complications– Hacking here and there to make it work– XML partially removed in this implementation– Need to rethink our data model– Fix abstraction boundaries– Make use of Brocade Workflow Composer (Stackstorm)

Stay tuned, more updates to come!

AMS-IX USAutomation Project Team

Solutions Architect Steven Bakker

Web Devs Martin Lyden, Petros Papadopoulos, Peter Brouwer

Soft Devs Gloria Garcia Dosil, László Bogárdi, Peter Corlett

NOC Engineer Eric Nghia Nguyen Duy

Customer Services Mariya Andonova

Finance Elisabetta Ghermandi

User Feedback AMS-IX US NOC, Customer Services and AMS-IX US portal users

Wait! No memes or funny catpictures?

Questions?