Operations & Automation Update
21
Operations & Automation Update More-IP USA Kostas (Konstantinos) Koutalis [email protected] June 17 th 2016
Transcript of Operations & Automation Update
Incoming tickets
l 580 Provision & Support
l approx.100 less than last year
00.5
11.5
22.5
33.5
2015-6 2015-10 2016-2
Messages per day
Average per day
Outside NL office hours (including weekends)US (17:00 – 01:00, GMT+2)
0%
20%
40%
60%
80%
100%
2015-6 2015-10 2016-2
Timezone distribution
Europe (9:00 – 17:00, GMT+2)
Asia/pacific (01:00 – 09:00, GMT+2)
US (17:00 – 01:00, GMT+2)
AMS-IX US NOC Announced Tickets
Planned maintenances 9
Non service affecting incident 3
Total Tickets 12
AMS-IX Tickets to Tech-L
AMS-IX US NOC & Operations
l AMS-IX NOC is a team of network engineers based in NL
o Standby duty
o Operation / Troubleshooting
o Customer provisioning
o Software upgrades
o Platform expansion and evolution
l Experience in simple configurations
AMS-IX US NOC & Operations
l & more complex configurations
AMS-IX US NOC & Operations
10GEPE router
EQX AM3PE router
NIKHEFPE router
Global SwitchPE router
Interxion AM3PE router
TC AMS4PE router
EvoswitchPE router
EQX AM1PE router
TC AMS2PE router
SARAPE router
TC AMS5PE router
PXC
CE
CECE
P Router EUNP Router
PXC
CE
PXC
CE
PXC
CE
PXC
CE
PXC
CE
PXC
CE
PXC
CE
PXC
CE
PXC
CE
PXC
CE
PXC
CE
TC AMS2PE router
10GEPE router
10GEPE router
10GEPE router
10GEPE router
10GEPE router
EQX AM1PE router
10GEPE router
TC AMS5PE router
TC AMS2PE router
CECE CE CECE CECE CE
10GEPE routerEunetworks
PE router
GSP Router
TC5 P Router
EQX AM1P Router
Global SwitchPE router
NIKHEFPE router
EQX AM3PE router
EvoSwitchPE router
Interxion AMS3PE router
TC4PE routerTC AMS4
PE routerEQX AM1PE router
TC AMS2PE router
TC AMS2PE router
SARAPE router
TC AMS5PE router
GSP Router
EUNP Router
l Administering and provisioning in these scales is not easy
l Automate … & set-up a 24/7/365 First Line NOC!
• The (very) old days:
• Manual labor
• Could take hours for 1 customer
• The (not so) old days:
• CLI scripts, one at a time
• Some functions not there (e.g. testing)
AMS-IX US NOC & Operations
AMS-IX US Automationl Provisioning an AMS-IX customer
AMS-IX US Automationl In more detail
AMS-IX US Automationl Even more details
Request
Customer portal
Control Server
REST API
XMLProvision
scriptsProvision
queueSQL
Network Equipment
AMS-IX US Automation
Under the hood:
• Ruby on rails, xml & PostgreSQL database
• SNMPo Query port nameo Port statuso Port counterso etc.
• CLI, Perl scriptso Perform configuration taskso A lot of checks based on “expect” scripts
AMS-IX US AutomationWhere are we now?
Available tools:– Perform all standard provisioning tasks (NOC & Customer Services)
• Reserve, assign a port• Test a link• Bring it to production
Tools available to customers / resellers (self provisioning):• Change mac address• Change port from untagged to tagged (qtag id)• Enable/disable v4/v6 BGP peering with admin ASN & Route Servers• Resellers: move virtual links from one reseller link to another
AMS-IX US AutomationWhere are we now?
– 24x7 NOC & AMS-IX US NOC debug information• Configured services (VLANs)• Switch configuration• Port status, optic light levels• Status of BGP sessions with route servers & AMS-IX routers• 10/100GE customers, see connected PE (primary / secondary)• See L2 violations & link flap logs
– 24x7 NOC & AMS-IX US NOC basic operations• Move connections from primary PE to secondary (10/100GE specific)• Unblock BGP sessions with route servers• ASN change• Change BGP prefix limits on route servers• Move a link from production to quarantine and vice versa (WIP)• Disable, Enable a link (WIP)
AMS-IX US AutomationAre we there yet?
• Continuous improvement
• Megaport - AMS-IX partnership
– More automations needed!
– IXP customers è Megaport Services
AMS-IX US Automation• Ordering a Megaport service via AMS-IX
AMS-IX US Automation• Megaport Customers è AMS-IX Services
AMS-IX AutomationHow good is all this?
– It works… for AMS-IX– We need an abstract version of “My AMS-IX”– Project kicked off late 2015
User Request
Customer portal
Control Server
REST API
XMLProvision
scriptsProvision
queueSQL
Network Equipment
AMS-IX AutomationHow good is all this?
– Remove hard coded values– Simplify provisioning process– Remove XML dependencies– Vendor independent
Setting up CHN-IX & conclusions– Different vendor being used added complications– Hacking here and there to make it work– XML partially removed in this implementation– Need to rethink our data model– Fix abstraction boundaries– Make use of Brocade Workflow Composer (Stackstorm)
Stay tuned, more updates to come!
AMS-IX USAutomation Project Team
Solutions Architect Steven Bakker
Web Devs Martin Lyden, Petros Papadopoulos, Peter Brouwer
Soft Devs Gloria Garcia Dosil, László Bogárdi, Peter Corlett
NOC Engineer Eric Nghia Nguyen Duy
Customer Services Mariya Andonova
Finance Elisabetta Ghermandi
User Feedback AMS-IX US NOC, Customer Services and AMS-IX US portal users
Wait! No memes or funny catpictures?
Questions?
