© 2019 Juniper Networks Juniper Business Use Only...© 2019 Juniper Networks Juniper Business Use...
Transcript of © 2019 Juniper Networks Juniper Business Use Only...© 2019 Juniper Networks Juniper Business Use...
© 2019 Juniper Networks Juniper Business Use Only
© 2019 Juniper Networks Juniper Business Use Only
CONFIDENTIALITY AND LEGAL NOTICE
This material contains information that is confidential and proprietary to Juniper Networks, Inc. Recipient may not
distribute, copy, or repeat information in the document without a signed non-disclosure agreement (NDA).
Any statements of product direction contained in this presentation sets forth Juniper Networks’ current intention and is
subject to change at any time without notice. No purchases are contingent upon Juniper Networks delivering any
feature or functionality depicted in this presentation.
Copyright 2019 Juniper Networks, Inc. All rights reserved. Juniper Networks, the Juniper Networks logo,
Juniper, Junos, and NXTWORK are registered trademarks of Juniper Networks, Inc. in the United States and other
countries. All other trademarks, service marks, registered marks, or registered service marks are the property of
their respective owners. Juniper Networks assumes no responsibility for any inaccuracies in this document. Juniper
Networks reserves the right to change, modify, transfer, or otherwise revise this publication without notice.
© 2019 Juniper Networks Juniper Business Use Only
James Kelly, Juniper Networks
Paul Arsenault, Blackberry
NETWORK AUTOMATION
Getting from Automatable to Automated
© 2019 Juniper Networks Juniper Business Use Only
EFFECTIVE
RELIABLE
VELOCITY, AGILITY,SCALE, REACH
EFFICIENT
DIGITAL OPS,DEVOPS,
PUBLIC CLOUD, IoT
SECURITY,SRE
EXPERIMENTATION,BIG DATA, AI
TCO,OPEN SOURCE
Hierarchy of needs
© 2019 Juniper Networks Juniper Business Use Only
STATE OF
NETWORK AUTOMATION
© 2019 Juniper Networks Juniper Business Use Only
NETWORK AUTOMATION
AUTOMATED NETWORKS
© 2019 Juniper Networks Juniper Business Use Only
FROM making better networks TO making networking better
NETWORKS(noun)
NETWORKING(verb)
success
Automation is an operational paradigm
© 2019 Juniper Networks Juniper Business Use Only
TWO SIDES TO AUTOMATED NETWORKS
DEV OPS
Automated NetOpsVendor’s Goal Customer’s Goal
EVOLVING TOOLS & ARCHITECTURE • Quality products• Autonomous (SDN)• Abstraction (intent)• Analytics• APIs and tool integrations
EVOLVING OPS & NRE• Codified workflows• Infrastructure as code (GitOps)• Automated building and testing (CI)• Automated deployments (CD)• Automated response (CR)
VENDOR CUSTOMER
Simpler and Automatable(building the tools)
Engineering Reliable Services(using the tools)
© 2019 Juniper Networks Juniper Business Use Only
EVOLUTION OF NETWORK ARCHITECTURE VS. NETOPS
In spite of years of focus on network automation:
• Programmability, APIs and API protocols
• SDN, NFV, virtual networks
• SDKs and toolkits
• Integrations with tooling like Ansible, Puppet, Chef, Salt, etc.
Today’s average NetOps work looks like the 1980s
• Still manual rote tasks and toil
• Manual work maybe from CLI to GUI, but not yet to APIs
• Midnight maintenance windows are normal
• Change is the main cause of outage
• People are CLI jockeys / CCIE technicians more than engineers
FROMSCALE-UP
TO SCALE-OUT
SOME THINGS HAVE CHANGED … AND SOME THINGS HAVE NOT
The 80s called and…
They want their CLI back
© 2019 Juniper Networks Juniper Business Use Only
EVOLUTION OF NETWORK ARCHITECTURE VS. NETOPS
In spite of years of focus on network automation:
• Programmability, APIs and API protocols
• SDN, NFV, virtual networks
• SDKs and toolkits
• Integrations with tooling like Ansible, Puppet, Chef, Salt, etc.
Today’s average NetOps work looks like the 1980s
• Still manual rote tasks and toil
• Manual work maybe from CLI to GUI, but not yet to APIs
• Midnight maintenance windows are normal
• Change is the main cause of outage
• People are CLI jockeys / CCIE technicians more than engineers
FROMSCALE-UP
SOME THINGS HAVE CHANGED … AND SOME THINGS HAVE NOT
TO SCALE-OUT
© 2019 Juniper Networks Juniper Business Use Only
CONSUMPTION IS THE CONTEMPORARY CHALLENGE
of data center NetOps are still manual
82%
Mistakes of the past• Forcing engineers’ rebirth as a “developers”• Over-sensationalizing of tech has left behind
transformation of people & processes
Ironically…humans are the heroesin the journey to automated NetOps
BRIGHT SPOT:
NRENetwork reliability engineer(ing)
© 2019 Juniper Networks Juniper Business Use Only
WHAT DOES SUCCESS
LOOK LIKE?
© 2019 Juniper Networks Juniper Business Use Only
NRE: INSPIRATION FOR NETWORKERS
Workbook now FREE
2019
DevOps is Coined
2009
Google publishes SRE
2016
DevOps Handbook
2015
Treats Ops as a software engineering problem… More engineering rigor and detail
© 2019 Juniper Networks Juniper Business Use Only
WHAT IS NETWORK RELIABILITY ENGINEERING
Nre
Core networking
fundamentals still matter.
It’s right in the name.
nRe
Represents a better way
of doing things.
Emphasizes the true goal
of automation
https://juniper.net/us/en/products-services/what-is/nre/
nrE
You can’t buy engineering
- you DO it. Sidesteps the
“productization” of
automation
Codify Automate Test Monitor Measure
Juniper Business Use Only
BLACKBERRY’SAUTOMATION PROGRAM
Juniper Business Use Only © 2017 BlackBerry. All Rights Reserved. 19
19
Table of Contents• Why Automate
• What Makes Automating Hard
• What We Did
• How We Did It
• How The Business Benefits
• What’s Next
Juniper Business Use Only
Why Automate?
Juniper Business Use Only © 2017 BlackBerry. All Rights Reserved. 21
21
Security
▪ Cylance Acquisition – BlackBerry is now branded as a Cyber Security company
Juniper Business Use Only © 2017 BlackBerry. All Rights Reserved. 22
22
Security
▪ Cylance Acquisition – BlackBerry is now branded as a Cyber Security company
▪Over the last couple of years, the number of catalogued vulnerabilities has almost doubled
▪Over 70% of organizations take up to 90 days to patch
▪Once a threat is made public the clock starts ticking
▪ It takes attackers time to develop tools to exploit new vulnerabilities
▪ If you can test and deploy a fix faster than the attacker can develop an exploit you get to live
another day
Juniper Business Use Only 23© 2017 BlackBerry. All Rights Reserved.
Need to Patch Faster
▪ Traditional way of certifying new code for production deployment is
resource intensive and slow
▪ Human resources
▪ Lab resources
Juniper Business Use Only 24© 2017 BlackBerry. All Rights Reserved.
Balancing Act
STABLE
NETWORK
Juniper Business Use Only 25© 2017 BlackBerry. All Rights Reserved.
Balancing Act
STABLE
NETWORK
CYBER
THREATS
Juniper Business Use Only 26© 2017 BlackBerry. All Rights Reserved.
Balancing Act
STABLE
NETWORK
CYBER
THREATSAUTOMATIO
N
Juniper Business Use Only 27© 2017 BlackBerry. All Rights Reserved.
Balancing Act
STABLE
NETWORK
CYBER
THREATSAUTOMATIO
N
Enterprises Need to Balance the
Requirement to Patch Security
Vulnerabilities Against the Need to
Provide a Stable Network
Infrastructure
✓ Automated Testing
✓ Regression as well as New Features
✓ 24x7
✓ Infrastructure as Code
✓ Standards-based Design
✓ Vendor Agnostic
✓ Open Source Tools
✓ Automated Deployment
Juniper Business Use Only © 2017 BlackBerry. All Rights Reserved. 28
28
Other Benefits
▪ Get more done in less time
▪ Employees focus on higher-value work
▪ Improved employee engagement
▪ Simplified operations
▪ Better test coverage
Juniper Business Use Only
What Makes Automating Hard?
Juniper Business Use Only 30© 2017 BlackBerry. All Rights Reserved.
Complicated Landscape
▪ Enterprises typically have a heterogeneous network environment
▪Multi-vendor
▪ Vendor-specific, monolithic operating systems
▪ Different feature sets with customer-specific use cases and
configurations
Juniper Business Use Only 31© 2017 BlackBerry. All Rights Reserved.
Complicated Landscape
▪ Enterprises typically have a heterogeneous network environment
▪Multi-vendor
▪ Vendor-specific, monolithic operating systems
▪ Different feature sets with customer-specific use cases and
configurations
▪ Talent with the required skills is at a premium
▪ Staff that have the networking and programming skills and
experience are very hard to find
Juniper Business Use Only 32© 2017 BlackBerry. All Rights Reserved.
Complicated Landscape
▪ Enterprises typically have a heterogeneous network environment
▪Multi-vendor
▪ Vendor-specific, monolithic operating systems
▪ Different feature sets with customer-specific use cases and
configurations
▪ Talent with the required skills is at a premium
▪ Staff that have the networking and programming skills and
experience are very hard to find
▪ Different vendors have different maturity levels with respect to their
automation capabilities
Juniper Business Use Only
BlackBerry Solution
Juniper Business Use Only © 2017 BlackBerry. All Rights Reserved. 34
34
What We Did
▪ Built a vendor-agnostic automation framework based on open-source tools that support automated
testing, deployment, and configuration management
Juniper Business Use Only © 2017 BlackBerry. All Rights Reserved. 35
35
What We Did
▪ Built a vendor-agnostic automation framework based on open-source tools that support automated
testing, deployment, and configuration management
▪ Git
▪ Repository (code, variable files, templates, playbooks, test reports)
▪ YAML and JSON describe specific instances
▪ JINJA2 templates describe configuration standards
Juniper Business Use Only © 2017 BlackBerry. All Rights Reserved. 36
36
What We Did
▪ Built a vendor-agnostic automation framework based on open-source tools that support automated
testing, deployment, and configuration management
▪ Git
▪ Repository (code, variable files, templates, playbooks, test reports)
▪ YAML and JSON describe specific instances
▪ JINJA2 templates describe configuration standards
▪ Ansible
▪ Does the heavy-lifting of interacting with devices
▪ YAML Playbooks define the operations we want to perform via Ansible
▪ Templates and variable files with Ansible modules generate
device-specific configurations
Juniper Business Use Only © 2017 BlackBerry. All Rights Reserved. 37
37
What We Did
▪ Built a vendor-agnostic automation framework based on open-source tools that support automated
testing, deployment, and configuration management
▪ Git
▪ Repository (code, variable files, templates, playbooks, test reports)
▪ YAML and JSON describe specific instances
▪ JINJA2 templates describe configuration standards
▪ Ansible
▪ Does the heavy-lifting of interacting with devices
▪ YAML Playbooks define the operations we want to perform via Ansible
▪ Templates and variable files with Ansible modules generate
device-specific configurations
▪ Jenkins
▪ Orchestrates our actions using pipelines
custom-built to match our workflows
▪ Integrated with our ticketing system
Juniper Business Use Only © 2017 BlackBerry. All Rights Reserved. 38
38
What We Did
▪ Built a vendor-agnostic automation framework based on open-source tools that support automated
testing, deployment, and configuration management
▪ Git
▪ Repository (code, variable files, templates, playbooks, test reports)
▪ YAML and JSON describe specific instances
▪ JINJA2 templates describe configuration standards
▪ Ansible
▪ Does the heavy-lifting of interacting with devices
▪ YAML Playbooks define the operations we want to perform via Ansible
▪ Templates and variable files with Ansible modules generate
device-specific configurations
▪ Jenkins
▪ Orchestrates our actions using pipelines
custom-built to match our workflows
▪ Integrated with our ticketing system
▪ Python Scripts
Juniper Business Use Only 39© 2017 BlackBerry. All Rights Reserved.
Deployment
▪ To mitigate the vulnerabilities we need to update the devices to the
new certified code
Juniper Business Use Only 40© 2017 BlackBerry. All Rights Reserved.
Deployment
▪ To mitigate the vulnerabilities we need to update the devices to the
new certified code
▪ Built deployment tools built on top of our automation framework
▪Orchestrated by Jenkins
▪ Use Ansible where possible
▪ Custom Python where required
▪ Incorporates traffic shifting for routers and HA for switching
▪ Pre and post-checks
Juniper Business Use Only 41© 2017 BlackBerry. All Rights Reserved.
What It Looks Like
▪ Automation Framework using open source, vendor-agnostic
tools
▪ Regression test suite that covers our own specific uses
cases and configurations
▪ Dynamic generated and easy-to-read test reports
▪ A library of test cases
▪ Deployment framework to push code and manage
configuration
▪ Eco-system built around the automation framework
Juniper Business Use Only 42© 2017 BlackBerry. All Rights Reserved.
Automation Building Blocks
Git
Repository
Network
EngineerLab Jenkins
Production
Jenkins
Production
AnsibleUpgrade Tools
Production
Devices
Test Traffic
Lab Ansible
Upgrade Tools
Regression
Test Suite
Test Bed
Juniper Business Use Only 43© 2017 BlackBerry. All Rights Reserved.
Code Certification Testing Process
Git
Repository
Review Test
Results
Updated
Vendor Code
ProductionTest ResultsRegression
Test Suite
Test Bed
Passed?Yes
No
Juniper Business Use Only
How We Did It
Juniper Business Use Only 45© 2017 BlackBerry. All Rights Reserved.
How We Did It
Created a new culture over a period of 2 years
Juniper Business Use Only 46© 2017 BlackBerry. All Rights Reserved.
How We Did It
Created a new culture over a period of 2 years
▪ Identified innovators and leaders within our staff
Juniper Business Use Only 47© 2017 BlackBerry. All Rights Reserved.
How We Did It
Created a new culture over a period of 2 years
▪ Identified innovators and leaders within our staff
▪ Empowered staff to be creative – permission to fail
▪ Encouraged and enabled peer-to-peer collaboration and support
Juniper Business Use Only 48© 2017 BlackBerry. All Rights Reserved.
How We Did It
Created a new culture over a period of 2 years
▪ Identified innovators and leaders within our staff
▪ Empowered staff to be creative – permission to fail
▪ Encouraged and enabled peer-to-peer collaboration and support
▪ Provided baseline formal training and detailed knowledge transfers
for all technical staff
▪Mandated that all new platforms require automated testing and
deployment before being released to production
▪ Set corporate goals and objectives around automation
Juniper Business Use Only 49© 2017 BlackBerry. All Rights Reserved.
How We Did It
Created a new culture over a period of 2 years
▪ Identified innovators and leaders within our staff
▪ Empowered staff to be creative – permission to fail
▪ Encouraged and enabled peer-to-peer collaboration and support
▪ Provided baseline formal training and detailed knowledge transfers
for all technical staff
▪Mandated that all new platforms require automated testing and
deployment before being released to production
▪ Set corporate goals and objectives around automation
▪ Set cut-off dates for manual changes and measured using KPIs that
are shared with the team
▪ Included automation capabilities as part of network equipment
vendor selection
Juniper Business Use Only
Business Benefits
Juniper Business Use Only © 2017 BlackBerry. All Rights Reserved. 51
51
More Secure…Better Quality…Faster
▪ New code with security vulnerabilities patched delivered to production at a much faster rate
▪90% reduction in certification time – 10 weeks down to 1 week
▪80% reduction in deployment time for new code – 6 months down to 5 weeks
▪ Improved test coverage
▪ Better code quality from vendors
Juniper Business Use Only 52© 2017 BlackBerry. All Rights Reserved.
Simplified Operations
Automation Drives Standardization
▪ Defined standards and templates enforced via the automation
framework and the eco-system
▪ Test cases that are reusable by operations as sanity checks for
troubleshooting as well as pre and post-change
▪Measurable KPIs to gauge improvements
▪ Current and standards-based network operating system
deployments means more consistency and improved feature
availability
▪ Standardized and well-known environment reduces the risk of
unexpected behaviors
Juniper Business Use Only © 2017 BlackBerry. All Rights Reserved. 53
53
Improved Employee Engagement
▪ Employees can spend their time on higher-level and more interesting work
▪Less time writing and approving MOPs
▪ Improved service delivery
▪ Challenging work that demands new skills coupled with on-the-job training keeps employees
engaged
▪ Smaller and more agile team can achieve more in less time
Juniper Business Use Only
What’s Next?
Juniper Business Use Only 55© 2017 BlackBerry. All Rights Reserved.
It’s a Journey, Not a Destination
▪ Automation framework has a lifecycle of its own
▪ Content requires frequent updating
▪ New test cases developed when corner cases are
discovered and issues are encountered in production
▪ Updates to test cases as the environment changes
▪ New platforms require automation to be written
Automation is never complete
Juniper Business Use Only 56© 2017 BlackBerry. All Rights Reserved.
What’s Next?
▪ Software-defined lab
▪ Use automation to orchestrate building the lab topology dynamically on-demand
▪ Leverage NFV to cost-effectively expand the capabilities of the lab
▪ Coordinated deployments using automation
▪ Use Jenkins jobs to call other Jenkins jobs to coordinate deployments across platforms
▪ This includes multi-cloud deployments
▪ Terraform deploys public cloud environments and calls our automation to do the on-prem portion
▪ Continuous Integration Continuous Delivery (CICD)
▪ Integration with Slack or an equivalent
▪ Minor changes coordinated and approved in real-time
▪ Change notifications sent via channels to which stakeholders (eg. NOC) subscribe
▪ Pre-deployment change validation and testing in lab
Juniper Business Use Only
Thank You
© 2019 Juniper Networks Juniper Business Use Only
5-STEP JOURNEY
© 2019 Juniper Networks Juniper Business Use Only
Manual Ops
• NetOps at the device or system UI
• Engineers are more technicians than technologists
Continuous Processes,Continuous Pipeline
• Dev/test environment
• CI-CD-CR DevNetOps pipeline for changes to intent and code
• Fast fail feedback, micro changes, staging and canary deployments
• Toolchain and code to automate analytics for regulation / remediation
As-code, Test-driven
• Connect actions to triggers
• Rethink troubleshooting as testing
• Everything is code (even configs) to be tested
• Security integrated from the start
Ad Hoc Automated Workflows
• Automate the design of NetOps workflows
• Focus on frequent troubleshooting or read-only tasks before config management
Engineering Outcomes
• Manage, don’t maximize, reliability by higher-order metrics
• NRE outcomes with service-level objectives, indicators and agreements (SLO/SLI/SLA)
• Use error budgets, toil budgets
• Chaos engineering
• Manage dependencies, separation of concerns
AUTOMATED NETOPS 5-STEP
DESTINATIONPeople:
Network ReliabilityEngineers (NRE)
Process:
“DevNetOps”And NRE’ing
Technology:
Abstracted, automated and autonomous NetOps
© 2019 Juniper Networks Juniper Business Use Only
THANK YOU