Intelligent Information Networkpalo/Rozne/cisco-expo/2006/Operating_your_MPLS... · VPN, Metro...

1© 2005 Cisco Systems, Inc. All rights reserved.IP VPN [email protected] Cisco Public

Intelligent Information Network


Bruno KlauserConsulting Engineer NMS/[email protected]

Operating your MPLS Core and MPLS-basedServices


Session Abstract

• This is an Overview Session of state-of the art approaches and best practices for MPLS Core and IP VPN Management

• Concepts are illustrated with implementation examples and some outlook into upcoming development is provided

• Basic understanding of MPLS and IP VPN as well as of network and service management is a prerequisite

• Target Audience: Network Management engineers and architects who administer MPLS-based networks and services

• Finally, please turn off all pagers and cell phones


1) Increasing Revenues and OpportunityReacting in real time to customer and market demandsDriving innovative products and services to market faster

Time to market for new services

Greater process efficiency, monitoring & reporting on activityReduce the escalating costs of IT, achieving ROI expectations

Enterprise: reduce OPEX; SP: generate revenue

4) Increasing Productivity & Efficiency

3) Improving Customer Relationships Building long-term business partner relationships

Management of SLC / SLA

2) Increasing Business Resiliency and AgilityGreater flexibility to use resources where and when needed

Activate network resources on demand

Intersection of Technology & BusinessAddressing the Business Objectives


We‘ve come a long way ...Mid ‘90 Late ‘90

2000 TodayAbuse a CCIE Find a way to scale

Find a way to integrate Evolve the OSS Architecture


Agenda – Focus Today: Changes & Diagnosis

IP OSS – Introduction & Overview

Manageability

Element Management

IP VPN Fulfillment

IP VPN Assurance

Putting Things Together

Summary, Q&A


Agenda


Manageability

Element Management

IP VPN Fulfillment

IP VPN Assurance


Summary, Q&A

Industry Trends- Trends Impacting NMS/OSS

- NMS/OSS Evolutions

- Paradigms & Standards

Cisco NMS/OSS Direction- Architecture

- Service Provider / Managed Services

- Enterprise / Managed Campus


IP OSS – Introduction & Overview Architectural Paradigms being adopted

• Common Sense rather than Religious Belief

• Goals & Feasibility dictate use of paradigms

• No way to survive without a paradigm

A typical real-life NGN blend is:- TMForum‘s eTOM for the bigger picture

- enriched by ITIL practices(typically for Config and Release Management)

- Access Technology specifics for the local loop(TR69 / DOCSIS / ... )

- overlayed by Quality, Legal and Security related models(TL9000, ...)


Agenda


Manageability

Element Management

IP VPN Fulfillment

IP VPN Assurance


Summary, Q&A

Industry Trends- Trends Impacting NMS/OSS

- NMS/OSS Evolutions

- Paradigms & Standards

Cisco NMS/OSS Direction- Architecture

- Service Provider / Managed Services

- Enterprise / Managed Campus


• Provide excellence in device manageability:– Granular monitoring & control– Device programmability

• Deliver state-of-the-art, scalable platform forMulti-Vendor network device management

• Deliver value add management applications in provisioning, service activation & control, traffic engineering, fault management, and diagnostics

• Provide extensible open, standard interfaces and developer kit for

IntegrationApplication freedom-of-choice

Service Provider Cisco’s Strategy for NMS/OSS


Cisco’s Strategy to Support Multi-Vendor

• Data driven model allows addition of new device and/or feature support by adding new registry entries (and code where required)

• Cisco will provide a development environment(IDE) which will enable third parties to produce new device VNEs (agents)

• The VNE IDE will be initially opened to customers and select Cisco partners and then to the SI developer community and other vendors

• Cisco will work with 3rd party vendors on VNE compliance testing program and a Northbound API compliance testing program.


Agenda


Manageability

Element Management

IP VPN Fulfillment

IP VPN Assurance


Summary, Q&A

Device Instrumentation- Fault

- Configuration

- Accounting

- Performance

- Security

Mediation at Network Level- Fault

- Configuration

- Accounting

- Performance

- Security


• Enhanced CLI / Enhanced Programmatic Interface– Cisco Enhanced Device Interface (E-DI)

• Innovative Provisioning Workflows – Zero-Touch-Deployment (ZTD)

• Device-Capability-Model & Device-Driver Concept– Partially adopted by Applications

(such as Cisco Active Network Abstraction, IP Solution Center, ...)

– Internal to the Applications today

– Standardizing & Opening up device-driver APIs is stillfuture-talk / work-in-progress

Mediation at Network Level – Configuration Mediation for Configuration – Focus Areas


• More Configuration Items due to changes rather than new roll-out

• Which do you prefer for configuring your network ?

Source: Cisco Web Seminar 522 attendees

Mediation at Network Level – Configuration The Impact of Maintenance

62% Command Line Interface 33% NMS Tools 5%

Source: Several Major MPLS Operators


Network Operations Failure Types Percent of respondents that ranked type as most frequent source of network

operations failure

Configuration errors 39%

Upgrade errors 27%

Data entry errors 10%

Maintenance errors 10%

Errors in monitoring the network 7%

Version control errors 7%

75%

IP OSS Overview What if at first you don‘t succeed ...

48% Human Config Error 52% Other

Yankee Group

• Network Operations Failures by Cause

• Network Outages Related to Human Configuration ErrorsSage

• Provisioning Efficiency of L3VPN Operators varies by a factor 15+ in terms of orders per HeadCount over time Cisco IBSG


• Improve Productivity for- device configuration- maintenance & troubleshooting- CatOS to IOS upgrades (planned)

• Single point of access to device configuration

• Unified interface acrossplatforms & releases

- enhanced CLI- Perl integration- XML API (IETF NETCONF draft 5 compliant)

• Complementary to EMS and NMS

• Support existing and new cisco devices

Engineer Perl Script NMS Application

• IOS / CatOS Today

• others later

Enhanced Device Interface – E-DI

E - CLI Perl API Netconf XML

Configuration

Exec Commands

File System

Events / Status

SW Image

Grouping

Scheduling

Authentication

Admin

Inventory

Cisco Enhanced Device Interface – E-DI 2.1Why E-DI ?


Engineer Perl Script NMS Application

Enhanced Device Interface – E-DI

E - CLI Perl API Netconf XML

Configuration

Exec Commands

File System

Events / Status

SW Image

Grouping

Scheduling

Authentication

Admin

Inventory

Access PointAccess Point

SwitchSwitch

RouterRouter

Eth-0/1

Radio-0RAM Eth-0/0

(A)

(B)

(C)Flash0

Eth-0/1Eth-0/0

Eth-0

Radio-0Radio-1

RAMFlash0

RAMFlash0

RAMA B C

FLASH

A0 B0 C0

Radio-A0Radio-C0Eth-A0Eth-B0/0Eth-B0/1Eth-C0/0Eth-C0/1

Interfaces

Cisco Enhanced Device Interface – E-DI 2.1Example: Network Virtualization


• OS Parser Emulation– CLI syntax checking & command context validation

– eliminates human errors

• Device Grouping– Definition of administrative domains

– Access Control per administrative domain level

– Simplifies Configuration and Administrative tasks.

• Group least common denominator CLI – Perform group operations without risk of generating unsupported command

• Context – Based CLI– Simultaneously apply changes to one or more devices by selecting the context

– Single point for network configuration

Cisco Enhanced Device Interface – E-DI 2.1Main Features


• Real-time syntaxvalidation & visualfeedback

Cisco Enhanced Device Interface – E-DI 2.1IOS-like Command Line Interface


Cisco Enhanced Device Interface – E-DI 2.1Visual Config Editor and Translator


• Reduced Network Operating Costs.

• Increased Operational Efficiency and Productivity.

• Reduced Training Costs – Less Additional Learning foreach new Technology or Platform.

• Extendable and Customizable.

• Complementary to Existing to Solutions.

• Quality Improvement over Working with device CLI

Cisco Enhanced Device Interface (E-DI 2.1) Customer Benefits


Agenda


Manageability

Element Management

IP VPN Fulfillment

IP VPN Assurance


Summary, Q&A

The Provisioning Challenge

IP Solution Center Overview

ISC / MPLS VPN Provisioning

ISC / Layer 2 VPN Provisioning

ISC / QoS

ISC / Traffic Engineering Management

Zero-Touch CPE Deployment and ISC

Deployment / Upgrade


Cisco ISC 4.x Overview IP Solutions Center

• IP Solutions Center is a Family of Management Applications for Planning, Provisioning, and Troubleshooting MPLS and Metro Ethernet networks.

ISC:MPLS simplifies, integrates and automates the management of IP VPN and QoS.

ISC:L2VPN and ISC:Metro Ethernet simplify, integrate and automate the management of IP VPN, Metro Ethernet, ATM, FR, and QoS.

ISC:TEM leverages MPLS TE to enable network convergence and offers advanced MPLS-TE management functionality, including network optimization and bandwidth protection.

ISC:MDE MPLS Diagnostic Expert improves Service availability and MTTR through advanced Diagnostics MPLS Core, Aggregation and Access networks.

Policy Based QoS

XML/SOAP API

L3VPN

CsCSOO

Mcastmvrf

L2VPNVPLSERSEWSEMSFR

ATM

L2TPv3

MPLSTEM

MPLSMDE

new


Policy Based QoS

XML/SOAP API

L3VPN

CsCSOO

Mcastmvrf

L2VPNVPLSERSEWSEMSFR

ATM

L2TPv3

MPLSTEM

MPLSMDE

Cisco ISC 4.x Overview IP Solutions Center – Key Characteristics

• 4-tier distributable architecture

• Comprehensive IP services provisioning

• Web based GUI

• Role Based Access Control user model support (RBAC)

• Support for 200+ simultaneous user access and 1M managed nodes

• CNM views

• Integration with Cisco Config Engine for Zero-Touch CPE Deployment

• RDBMS (bundled with Sybase, option to use Oracle)

soon


VPNSC: MPLS VPNSC: MPLS (Eureka 1.0)(Eureka 1.0)

Oct. ’99

VPNSC: MPLSVPNSC: MPLS(Eureka 1.0.2)(Eureka 1.0.2)

Nov. ’99

VPNSC: MPLSVPNSC: MPLS(Eureka 1.1)(Eureka 1.1)

Mar. ’00

VPNSC: MPLS VPNSC: MPLS (Eureka 1.2.1)(Eureka 1.2.1)

Nov. ’00

Cisco ISC 4.x Overview Track Record

VPNSC 2.0VPNSC 2.0MPLS & IPSecMPLS & IPSec

Mar. ’01


Nov. ’01


Jun. ’02

ISC 3.0ISC 3.0Layer 2 VPNLayer 2 VPNLayer 3 VPNLayer 3 VPN

QoSQoSApr. ’03


QoSQoSNov. ’03


QoSQoSApr. ’04


QoSQoSTE MgmtTE Mgmt

Dec. ’04

ISC 4.2Layer 2 VPNLayer 3 VPN

QoSTE Mgmt

DiagnosticsQ4 CY’06


QoSQoSTE MgmtTE Mgmt

DiagnosticsDiagnosticsDec. ’05


Agenda


Manageability

Element Management

IP VPN Fulfillment

IP VPN Assurance


Summary, Q&A

The Assurance Challenge

Correlating Faults

MPLS Diagnostic Expert


1) A PE Interface goes down and trap is sent to fault management …

IP VPN Assurance – Correlating Faults A Simple Failure Scenario – 1/7

2)

… root-cause vs. symptom ?… in service / out of service ?… which customer is affected ?… which VPN / Site ?… neighbouring devices ?… SLA breaches approaching ?

???


Incoming Interface Down Alarm



CiC events are enriched with topology information from Precision IP, including “connected device”, and “VRF name”



CiC ISC Policy Manager (based on IMPACT) automatically extracts service information from the Cisco ISC MPLS Provisioning Product.

This enables CiC to automatically identify which customers and Sites will be impacted by a particular PE failure.



Security Events

Database Events

Web Events

Web Events

Load B. Events

Network Events

Network Events

Network Events

Network Events

End-user Events

Identifying and monitoring the components that support the Service enables end-2-end Service Assurance

Identify service elements

Monitor service elements



Security Events

Network Events

Network Events

Database Events

Web Events

Network Events

Load B. Events

Web Events

Network Events

End-user Events

Consolidate Service Events > Event Consolidation provides visibility of all service-impacting events> Consolidation engine should reduce service event “noise”

> Deduplication> Suppression> Auto-clear events that resolve themselves

IP VPN Assurance – Correlating FaultsA Simple Failure Scenario – 6/7


Netw

ork

Load

Bal

ance

r

Web

Data

base

Secu

ritySecurity

EventsNetwork Events

Network Events

Database Events

Web Events

Network Events

Load B. Events

Web Events

Network Events

End-user Events

Events mapped to Service Model

Service Model

OLAsDefined

Consolidate Service EventsOverall

SLA

IP VPN Assurance – Correlating FaultsA Simple Failure Scenario – 7/7


CiC IOS’ MPLS Troubleshooting Tools:

CIC MPLS Tools leverage Precision IP topology data e.g. “VRF Ping Connected Device” uses Precision data to auto populate the “connected device field”

IP VPN Assurance – Correlate UPAnd NOW start Drill-DOWN / Troubleshoot


Agenda


Manageability

Element Management

IP VPN Fulfillment

IP VPN Assurance


Summary, Q&A

The Assurance Challenge

Correlating Faults

MPLS Diagnostic Expert


IP VPN Assurance – MPLS Diagnostic Expert Detection of Faults

Everything that can go wrong in an MPLS Network will produce a TRAP

• And even if it would:• Was it raised by the problem or by a symptom if it ?

• Can you drill-down from the trap ?

• Did you configure it (the TRAP or the Audit raising the TRAP) ?

• Do you always capture it ?

• Do you always understand it‘s impact ?

• Do you always assign the right priority ?


IP VPN Assurance – MPLS Diagnostic Expert Detection of Faults

1. Network Reported• Alarms (Traps, Syslog)

2. Proactive Audit Results• Polling, Auditing, Thresholds, IPSLA

3. Customer Reported• Missed by 1) and 2)

ISSUE


IP VPN Assurance – MPLS Diagnostic Expert Customer Reported Faults (CRF)

• Often (too-) late to prevent SLA violation

• Unprecise problem description

• Cannot be planned for

⇒Minimize # of Customer Reported Faults

⇒Reproduce & Isolate CRF

⇒MPLS OAM can help


IP VPN Assurance – MPLS Diagnostic Expert MPLS OAM in Assurance

Support the Problem Resolution Cycle

• Detect

• Locate

• Isolate

• Fix

• Test

• Deploy

aka ‚Troubleshoot‘


Customer reported fault -Smoking exhaust …

TroubleshootingBasic alarm -

Oil lamp

Root Cause Analysis –from Engine Management System”

Alarm Management

IP VPN Assurance – MPLS Diagnostic Expert MPLS Troubleshooting


LSPLSP

1) Fault on line card ASIC, traffic disappears in “black hole”. No alarm generated.

3) Start locating the problem on this nodeusing CLI, MIBs, …

2) Customer Reported Fault

4) … then this one …

5) … then this one … 70) … then this one …

71) … HOURS later … after checking 70 P and PE nodes the fault is located.

=> now troubleshooting starts

IP VPN Assurance – MPLS Diagnostic Expert MPLS OaM manual Troubleshooting


LSPLSP

newIP VPN Assurance – MPLS Diagnostic Expert MPLS OaM Tool Support

MPLS Diagnostic ExpertEncapsulates Cisco TAC IP Over 100+ potential failure

scenarios checked automatically

Non-Intrusive

CLI transcript & Logs

Automated drill down GUIs

Affected Node isolated directly byMPLS Diagnostic Expert & IOS instrumentation


newCisco ISC – MPLS Diagnostic Expert OAM Application – Input Screen


newCisco ISC – MPLS Diagnostic Expert OAM Application – Input Screen 2


newCisco ISC – MPLS Diagnostic Expert OAM Application – Progress Screen


newCisco ISC – MPLS Diagnostic Expert OAM Application – Result Screen


newCisco ISC – MPLS Diagnostic Expert OAM Application – Test Log


newCisco ISC – MPLS Diagnostic Expert OAM Application – CLI Transcript


• Edge – 30 Unique Scenarios …Route Target Mismatches between Ingress/Egress PEMismatches between FIB/LFIB…

• Core – 30 Unique Scenarios …RP/LC inconsistenciesLSP Blackholes…

• Access Circuit – 40 Unique Scenarios …ATM/Frame Relay/Ethernet IssuesHDLC interfaces…

newCisco ISC – MPLS Diagnostic Expert OAM Application – Test Scenarios


Type in Simple details e.g. Customer Edge IP

addresses … and press “OK” to start

• GUI locating the problem and isolatingit’s root cause & recommended action• 100+ potential failure scenarios checked • Repeatable & Transcipted & Logged

Cisco ISC – MPLS Diagnostic Expert OAM Application – Summary new


This will completely change the dynamics of our service desk –we can put advanced tools into

the hands of Tier 1 operators to run while the customer is still on the phone. We expect this to eliminate 50% of escalations to

Tier 2/3 technicians

Cisco ISC – MPLS Diagnostic Expert Some Quotes from EFT Customers new

We haven’t seen anything like this before. This is a step function in capability above our fault vendors

We want this tomorrow for our

service desk

Very Impressive. Will be very useful for our Tier 1 & 2

support teams

This is unique in the industry


Cisco ISC – MPLS Diagnostic Expert CCIE vs. MPLS Diagnostics Expert new

See: http://www.miercom.com/?url=reports/&v=16

• Free up CCIE time for more forward-looking and revenue generating activities


Agenda


Manageability

Element Management

IP VPN Fulfillment

IP VPN Assurance


Summary, Q&A


Putting Things Together Typical IP VPN Management Architecture

Programmable and Physical Network Layers

Programmable Network LayerProgrammable Network Layer

Network Devices & Device InstrumentationNetwork Devices & Device Instrumentation

Assurance

SalesSales OrderHandling

OrderHandling

ProblemResolutionProblem

ResolutionPerf/SLAReportingPerf/SLAReporting

Invoicingand RatingInvoicing

and Rating

Service Product Development and Maintenance

Network and Systems Management

NetworkPlanningNetworkPlanning

ElementManagement

ElementManagement

MaintenanceRestoration

MaintenanceRestoration

NetworkMonitoringNetwork

Monitoring

ServiceCreationServiceCreation

ServiceInventoryService

InventoryServiceQualityServiceQuality

MediationAggregationMediation

Aggregation

Customer Care

Fulfillment Billing

NetworkProvisioning

NetworkProvisioning

ServiceProvisioning

ServiceProvisioning

Programm

able Netw

ork Infrastructure

Common Information and

Services

Event MgmtEvent Mgmt

IP MgmtIP MgmtDNS, DHCPAddressingDNS, DHCPAddressing

Intelligent Agents

Intelligent Agents

Inventory andTopology

Inventory andTopology

Identity and Security

Identity and Security

Workflow,GUI and APIsWorkflow,

GUI and APIs

IAIA

CiscoCiscoInfoInfo

CenterCenter

NTP Source(s)NTP Source(s)

/ or Custom SLM Portal/ or Custom SLM Portal

Cisco Cisco NetFlowNetFlowEngineEngine

Quallaby ProvisoQuallaby ProvisoInfovistaInfovista VistaViewVistaView

CA CA eHealtheHealth

Cisco Info Center / SLAM andCisco Info Center / SLAM and

CiscoCiscoIP SolutionIP Solution

CenterCenter

PolPol MgrMgr

Cisco Cisco ConfigConfigEngineEngine

IPSLAIPSLASNMPSNMP SyslogSyslog SSHSSH NTPNTP NetFlowNetFlow

Cisco Cisco NetFlowNetFlowCollection EngineCollection Engine

CiscoSecureCiscoSecure ACSACSTACACS +TACACS +

CiscoWorksCiscoWorks LMSLMSCTMCTM

Enhanced DeviceEnhanced DeviceInterface (EInterface (E--DI)DI)


Agenda


Manageability

Element Management

IP VPN Fulfillment

IP VPN Assurance


Summary, Q&A


Service Fulfillment for IP VPNs Summary

Service Fulilment is not only about putting the correct IOS config into a node, it‘s also about ...

... being secure, reliable, efficient and scalable

... having an accurate logical inventory which allows to understand, reproduce, map, correlate, audit, modify, test and troubleshoot what has been configured

... coping with heterogenous platforms and IOS releases, multiple access technologies, unknown IP addresses, etc.

... causing minimal impact on network nodes, convergence times but also on time/skills required of network operators


Service Assurance for IP VPNs Summary

Assurance is not only about receiving alarms, it‘s also about ...

... turning alarms into information by - enriching them with technical information- correlating them to IP VPN services- mapping them against the SLA contract

... supporting an effective problem resolution cycle

... applying pro-active audits to decrease the number of customer found defects

... applying re-active audits to accelerate the troubleshooting cycle


Q & AReferences

• Cisco IP Solution Center: www.cisco.com/go/isc

• MPLS Diagnostic Expert: www.cisco.com/go/mde

• Cisco Configuration Express: www.cisco.com/go/ce

• Cisco Configuration Engine: www.cisco.com/go/ie2100

• Cisco Info Center: www.cisco.com/go/cic

• CiscoWorks LMS 2.5 : www.cisco.com/go/lms

• NetFlow: www.cisco.com/go/netflow

• IPSLA (aka SAA, aka RTR): www.cisco.com/go/ipsla

• Feature Navigator: www.cisco.com/go/fn

• MIB Locator: www.cisco.com/go/mibs

• Cisco Network Management Newsletter (email subscription possible):www.cisco.com/external/networkmanagement/nm-newsletter/May06.htm


… thank you …

[email protected]

Intelligent Information Networkpalo/Rozne/cisco-expo/2006/Operating_your_MPLS... · VPN, Metro...

Documents

Transcript of Intelligent Information Networkpalo/Rozne/cisco-expo/2006/Operating_your_MPLS... · VPN, Metro...