Resilient EtherNet/IP Networking...2020/04/04 · Catalyst 2955 Cell/Area Zone Switch Stack...

UserGroupTechnical Seminars

Our presentation will begin at 10:00 am Central

Resilient EtherNet/IP Networking:Device Level Ring (DLR)

April 2020

2020 Online Events

https://trcautomation.solutions/events/

May 21st

CIP Security

June 18th

ControlLogix Redundancy

April 22nd

Remote Assistance with Augmented Reality (AR)

April 29th

Electronic Circuit Protection –Features and Benefits

April 23rd

Legacy Migration Series Part 1: SLC-500 to CompactLogix

April 27th

Legacy Migration Series Part 2: ControlLogix/CompactLogix to ControlLogix 5580/CompactLogix 5380

Presenters

Brandon SinghNetwork Specialist

The Reynolds CompanyDallas/Fort Worth

Luis RamosSolution Architect

Rockwell AutomationHouston

Wayne WelkAutomation Specialist

The Reynolds CompanyNew Orleans

David NuteAutomation Specialist

The Reynolds CompanyHouston

Resilient Network Protocols for High-

Availability Systems

PUBLIC | Copyright ©2019 Rockwell Automation, Inc. | #ROKLive | 5

High-availability process control systems require networks that match them in reliability. With multiple choices available, this session will help you understand the commonly used options, with a focus on PRP (Parallel Redundancy Protocol).

Resilient network protocols for high-availability systems


Overview


High-availability networks help prevent the loss of production caused by media breaks an infrastructure failures.

Why do you care?

• Media breaks and infrastructure failures

• Can be intermittent

• Can be difficult to physically locate

• Can be difficult to repair


Network resiliency topics to be covered

• Redundant media

• DLR (Device Level Ring)

• PRP (Parallel Redundancy Protocol)

• Provides both media redundancy and a degree of infrastructure redundancy

• Redundant infrastructure

• 1715 redundant adapters

• 1756 redundant adapters

• REP

• Flexlinks


Redundant media and infrastructure


Network design considerationsTopology options

Redundant star Ring Linear

Cabling requirementsEase of configurationImplementation costsBandwidthRedundancy and convergenceDisruption during network upgradeReadiness for network convergenceOverall in network TCO and performance Best OK Worst

Redundantstar

HMI

CiscoCatalyst 2955

Cell/Area Zone

Switch Stack

Controllers,Drives, and Distributed I/O

Cell/Area Zone

Ring

HMI

Cell/Area Zone

Controllers

Switch Stack

Linear

Cell/Area Zone

Distributed I/O

Drives

Controllers


Application and technical drivers

Device-level topology

• Linear

• Comfort level with traditional field bus topology

• Help minimize cost of additional switches

• Simplify network cabling

• Applicable for certain applications that physically have a linear layout

• Conveyor applications

• Material handling application

• Ring

Making the linear topology into a ring provides single fault tolerance

Network still functions if there is a (single) break

Better fault tolerance over normal star topology

A resiliency protocol is needed in order to:

Keep packets from circling the ring forever

Reconfigure to linear topology in event of a fault

Detect ring restoration and reconfigure to ring mode


Networking design considerations

Network convergence

• Network convergence (healing, recovery, etc.) time – is a measure of how long it takes to detect a fault, find an alternate path, then start forwarding network traffic across that alternate path

• MAC tables must be relearned

• Multicast on uplinks must be relearned

• During the network convergence time, some portion of the traffic is dropped by the network because interconnectivity does not exist

• If the convergence time is longer than the Logix controller connection timeout, the IACS devices on the affected portion of the network may stop operating and may affect the industrial automation application


Device Level Ring protocol

Device Level Ring topology

• Layer 2 protocol

• Meaning that it’s transparent to higher-level protocols

• Single-ring topology

• Fast fault detection and ring recovery

• Ring supervisor

• Supervises the ring, one or more

• Normally a scanner or a dedicated supervisor




• ODVA - open standard enabling suppliers to develop compatible products

• Support for ring and linear topologies, fiber and copper implementations

• Network traffic is managed to helps deliver timely delivery of critical data (Quality of Service, IEEE-1588 Precision Time Protocol, Multicast Management)

• Ring is a single fault tolerant network

• Designed for 1-3 ms convergence for simple device networks




• Supervisor blocks traffic on one port

• Sends Beacon frames on both ports to detect break in the ring

• Once ring is restored, supervisor hears beacon on both ports, and transitions to normal ring mode, blocking one port

Beacon Beacon

Announce AnnounceBlocking

Forwarding

I/O I/O

Active Ring Supervisor

Controller

VFDI/O

IES




• All faults that are detectable at physical layer

• Physical layer failure detected by protocol-aware node

• Status message sent by ring node and received by ring supervisor

Link Status Link Status

Link Failure

Active Ring Supervisor

I/O I/O

Controller

VFDI/O

IES




• After failure detection, ring supervisor unblocks blocked port

• Network configuration is now a linear topology

• Fault location is readily available via diagnostics

Link Failure

Forwarding ForwardingActive Ring Supervisor

I/O I/O

Controller

VFDI/O

IES




• Once ring is restored, supervisor hears beacon on both ports, and transitions to normal ring mode, blocking one port

Beacon

Active Ring Supervisor Beacon

Announce AnnounceBlocking

Forwarding

I/O I/O

Controller

VFDI/O

IES

PRP (Parallel Redundancy Protocol)

Network design considerations

Redundant Star Ring Linear

Cabling RequirementsEase of ConfigurationImplementation CostsBandwidthRedundancy and ConvergenceDisruption During Network UpgradeReadiness for Network ConvergenceOverall in Network TCO and Performance Best OK Worst

RedundantStar

HMI

CiscoCatalyst 2955

Cell/Area Zone

Switch Stack


Cell/Area Zone

Ring

HMI

Cell/Area Zone

Controllers

Switch Stack

Linear

Cell/Area Zone

Distributed I/O

Drives

Controllers

• What is PRP?• PRP, parallel redundancy protocol, IEC standard

62439-3• Fault tolerant, fully redundant Ethernet infrastructure

at Layer 1 and 2• Same packet is sent on both LANs • Separate and different protocol than DLR or

EtherChannel• Typical applications for PRP

• Where redundant network infrastructure is desired• Process applications (eg. 24x7x365 operational

requirements)• ControlNet redundant media migration opportunities,

such as transportation tunnels


LAN A LAN B

Overview

• PRP terminology• DAN, Dually Attached Node, has PRP protocol built in and attaches to both LANs• SAN, Singly Attached Node, is a node that does not have PRP built, attaches to one LAN• RedBox, facilitates PRP connectivity for standard devices• VDAN, Virtually attached Node would be the device connected through the RedBox.


LAN ALAN B

SAN

DANs in I/O chassis

VDANs

DAN in controller chassis

RedBox – Stratix® 5400

Infrastructure Switches

IEC standard terminology

PRP (Parallel Redundancy Protocol)Plant network connectivity

• Network switches • The infrastructure switch passes the PRP-marked packets just like

any other packet. • Stratix® managed switches are recommended for Diagnostic support• Managed switches required due to larger frame sizes (1506 bytes)

• Network convergence times• Zero. No Switchover required• A transmitting DAN duplicates the packet and sends it out on both

LANs, the receiving DAN discards the duplicate• PRP end devices (e.g. 1756-EN2TP, 5094-AENTR)

• One IP address, one MAC ID per device• The same identity is used on Both LAN A and LAN B• Makes PRP seamless to the application

• Network monitoring• Network faults will be seamless to the application and therefore

detection via monitoring is critical• To achieve Monitoring accuracy, it is recommended that different

network addressing is used for Infrastructure devices on LAN A vs. LAN B

PRP (Parallel Redundancy Protocol)Details

• Stratix® 5410 • Support for up to 512 VDANs• Connectivity to 2 PRP channels or pairs

• Stratix® 5400• Support for DLR up to 3 rings for VDANS• Support for up to 512 VDANs• Connectivity to 1 PRP Channel or pair

PRP (Parallel Redundancy Protocol)Redbox details

VDANs in DLR ring (up to 3 rings) RedBox – Stratix®

5400

DAN in controller chassis

PRP (Parallel Redundancy Protocol)ControlLogix® redundancy support available as of V31.050

LAN ALAN B

SAN

DANs in I/O chassis

VDANs

RedBox –Stratix® 5400

PRP Application TechniqueENET-AT006https://literature.rockwellautomation.com/idc/groups/literature/documents/at/enet-at006_-en-p.pdf

https://literature.rockwellautomation.com/idc/groups/literature/documents/at/enet-at006_-en-p.pdf

PRP White PaperENET-WP041 https://literature.rockwellautomation.com/idc/groups/literature/documents/wp/enet-wp041_-en-p.pdf

https://literature.rockwellautomation.com/idc/groups/literature/documents/wp/enet-wp041_-en-p.pdf

PRP Design and Implementation GuideENET-TD021https://literature.rockwellautomation.com/idc/groups/literature/documents/td/enet-td021_-en-p.pdf

https://literature.rockwellautomation.com/idc/groups/literature/documents/td/enet-td021_-en-p.pdf

Redundant Infrastructure


1715 redundant I/O featuresRedundant power

supply

Two slot adapter backplane

DLR ports

Redundant input modules

Redundant output

modules

Redundant termination assemblies

• 24VDC discrete input module• 24VDC discrete output module• 4 to 20 ma analog input module• 4 to 20 ma analog output module• Redundant 24VDC power supply

connections

Three slot I/O backplanes

Redundant ethernet

adapters when on a DLR ring


1715 redundant I/O benefits

• Redundant Ethernet adapters

• Features:

• Fault-tolerant I/O

• Ability to operate on device level ring topologies

• Benefit: integrates all levels of a system on a common fault-tolerant network

• Advantages:

• Automatic switch over in the event of any fault in a module pair

• Requires no additional hardware to implement and no programming required


• Overview

• Firmware target: April 2020• Provides primary and backup

adapters for 1756 I/O chassis• Key features

• Redundant adapter functionality is a firmware update to the existing 1756-EN4TR, no hardware update required

• Customer benefits

• Allows customers to extend network redundancy through to the 1756 adapter level

• Provides additional resiliency compared to a single adapter

• Existing 1756 I/O chassis can be modified for redundant adapter -slots 0 and 1 are used for redundant adapter

1756-EN2T

1756-EN4TR pair

1756-EN4TR redundant adapter firmware enhancement


• When used with ControlLogix®

Redundancy and redundant switches, it provides for no single point of failure

1756-EN2T

1756-EN4TR pair



• DLR architecture available at release

1756-EN2TR

1756-EN4TR pair



• Single controller architecture supported at release

• Star or DLR1756-EN4TR or 1756-EN2T

1756-EN4TR pair



Details and rules • I/O modules only, no controllers in the redundant adapter chassis• Motion in redundant adapter chassis not supported• Supported in future releases of 1756-EN4TR redundant adapter:

• CIP Safety modules• PRP• CIP Security

• Redundant adapters reside in slots 0 and 1 only• Rotary switch on the redundant 1756-EN4TR modules need to be set to #7 for redundant adapter with

linear/DLR



Topology optionsNetwork design considerations

Redundant Star Ring Linear

Cabling RequirementsEase of ConfigurationImplementation CostsBandwidthRedundancy and ConvergenceDisruption During Network UpgradeReadiness for Network ConvergenceOverall in Network TCO and Performance Best OK Worst

RedundantStar

HMI

CiscoCatalyst 2955

Cell/Area Zone

Switch Stack


Cell/Area Zone

Ring

HMI

Cell/Area Zone

Controllers

Switch Stack

Linear

Cell/Area Zone

Distributed I/O

Drives

Controllers


PRP Application TechniqueENET-AT006https://literature.rockwellautomation.com/idc/groups/literature/documents/at/enet-at006_-en-p.pdf

https://literature.rockwellautomation.com/idc/groups/literature/documents/at/enet-at006_-en-p.pdf

Other Resillient Protocols

Link Aggregation Control Protocol (LACP) port aggregation – IEEE 802.3ad

Redundant Star Topology Built into Stratix 5410/5400/5700/8000/8300 Aggregates multiple physical links into one logical

link Provides resiliency between connected switches

if a connection is broken

StratixAccess

Switches

F - Forwarding

F F F F

DistributionSwitch

Catalyst 3850 Switch Stack

Spanning Tree Protocol(MSTP)

MSTP is an IEEE standard Ring and redundant star topology Built into Stratix 5410/5400/5700/8000/8300 Provides:

Loop-free network Redundancy in case of failure

Distribution is the root bridge Operates in a plug-and-play fashion Coordinate with IT before implementing

FB

F - Forwarding

F

DistributionSwitch

Catalyst 3850 Switch Stack

StratixAccess

Switches

B

B - Blocking

REP- Resilient Ethernet Protocol

The Resilient Ethernet Protocol (REP) is a Cisco proprietary protocol that provides an alternative to the Spanning Tree Protocol (STP)

A REP segment is a chain of ports connected to each other and configured with a segment ID. Each segment consists of standard (nonedge) segment ports and two user-configured edge ports. A router can have no more than two ports that belong to the same segment, and each segment port can have only one external neighbor

Because REP runs on a physical-link basis and not on a per-VLAN basis, only one hello message is required for all VLANs, thus reducing the load on the protocol.

Although REP and DLR are not compatible protocols they are frequently used together.

DLR Reference Architectures

PUBLIC | Copyright ©2019 Rockwell Automation, Inc. 47

Deploying DLR within CPwE Design Guide – Phases Review

Phase II: Mixed device/switch-Level Ring (2019)

Phase III: DLR Redundant Gateway (2020)

Deploying DLR within a CPwE Design Guide publication ENET-TD015

Phase I: Switch-Level Ring (2018)



Phase 1 – DLR Reference ArchitectureSwitch-Level Ring

Up to 24 switches Mixed Stratix 5400

and 5700 switches 100 Mbps or 1 Gbps

but not mixed Single VLAN


Single Mixed Device/Switch-Level DLR Ring at 100 Mbps (50 ring nodes)Phase 2 – DLR Reference Architecture

Results

• Traffic disruption localized to the DLR ring were 3 msor less.

• Local CIP Safety control and messaging applications reported no I/O or produce/consume connection loss.

• Local high-speed motion application with a 2 msCUR reported no motion faults.


Single Mixed Device/Switch-Level DLR Ring at 1 GbpsPhase 2 – DLR Reference Architecture

Results





Multiple Mixed Device/Switch-Level DLR Ring at 100 Mbps (All Rings at same speed)Phase 2 – DLR Reference Architecture

Results





Multiple Mixed Device/Switch-Level Ring at Mixed Ring Speeds (Ring 1 at 1 Gbps and Ring 2 and 3 at 100 Mbps)Phase 2 – DLR Reference Architecture

Results





Switch-Level DLR with Redundant GatewayPhase 3 – DLR Reference Architecture

Recommendation

Results• Traffic disruption localized to the DLR ring were 3 ms or less.

• Local CIP Safety, Standard CIP control and messaging applications reported no I/O or produce/consume connection loss.

• Local high-speed motion application with a 2 ms CUR reported no motion faults.


Single Mixed Device/Switch-Level DLR (100 Mbps) with Redundant GatewayPhase 3 – DLR Reference Architecture

Recommendation





Multiple Mixed Device/Switch-Level DLR with Redundant Gateway at Mixed Ring Speeds (Single VLAN)Phase 3 – DLR Reference Architecture

Recommendation




• It was observed and recommended not to apply safety and high-speed motion beyond the local DLR.


Multiple Mixed Device/Switch-Level DLR with Redundant Gateway at Mixed Ring Speeds (Multiple VLAN)Phase 3 – DLR Reference Architecture

Recommendation




• It was observed and recommended not to apply safety and high-speed motion beyond the local DLR.

Networking Resources

CPwE websites CPwE Solution Graphic White Papers and Design Guides Overview Documents Cisco / Rockwell Automation

Alliance Profile Prepare industrial operations for

the future with Cisco and Rockwell Automation

Top 10 Recommendations for Plant-wide EtherNet/IP Deployments

Networking ResourcesCPwE Architectures - Collection of Architected, Tested & Validated Designs

http://www.aquentstudiosra.com/11035_cpwe/

http://www.rockwellautomation.com/global/products-technologies/network-technology/architectures.page?

http://literature.rockwellautomation.com/idc/groups/literature/documents/sp/enet-sp006_-en-p.pdf

https://www.rockwellautomation.com/resources/downloads/rockwellautomation/pdf/sales-partners/RA-Cisco-PrepareIndustrialOperations-Overview.pdf

Networking ResourcesCPwE Architectures - Collection of Architected, Tested & Validated Designs

Topic Design Guide WhitepaperDesign Considerations for Securing IACS Networks N/A ENET-WP031A-EN-P

Converged Plantwide Ethernet – Baseline Document ENET-TD001E-EN-P N/A

Deploying 802.11 Wireless LAN Technology within a Converged Plantwide Ethernet Architecture ENET-TD006A-EN-P ENET-WP034A-EN-P

Deploying Identity and Mobility Services within a Converged Plantwide Ethernet Architecture ENET-TD008B-EN-P ENET-WP037C-EN-P

Securely Traversing IACS Data Across the Industrial Demilitarized Zone (IDMZ) ENET-TD009B-EN-P ENET-WP038B-EN-P

Deploying Network Address Translation within a Converged Plantwide Ethernet Architecture ENET-TD007A-EN-P ENET-WP036A-EN-P

Migrating Legacy IACS Networks to a Converged Plantwide Ethernet Architecture ENET-TD011A-EN-P ENET-WP040A-EN-P

Deploying A Resilient Converged Plantwide Ethernet Architecture ENET-TD010B-EN-P ENET-WP039D-EN-P

Deploying Industrial Firewalls within a Converged Plantwide Ethernet Architecture ENET-TD002A-EN-P ENET-WP011B-EN-P

Deploying Device Level Ring within a Converged Plantwide Ethernet Architecture ENET-TD015C-EN-P ENET-WP016D-EN-P

OEM Networking within a Converged Plantwide Ethernet Architecture ENET-TD018A-EN-P ENET-WP018A-EN-P

Cloud Connectivity to a Converged Plantwide Ethernet Architecture ENET-TD017A-EN-P ENET-WP019B-EN-P

Deploying Industrial Data Center within a Converged Plantwide Ethernet Architecture ENET-TD014A-EN-P ENET-WP013A-EN-P

Deploying Scalable Time Distribution within a Converged Plantwide Ethernet Architecture ENET-TD016A-EN-P ENET-WP017B-EN-P

Deploying Network Security within a Converged Plantwide Ethernet Architecture ENET-TD019A-EN-P ENET-WP023B-EN-P

Deploying Parallel Redundancy Protocol within a Converged Plantwide Ethernet Architecture ENET-TD021A-EN-P ENET-WP041A-EN-P

http://literature.rockwellautomation.com/idc/groups/literature/documents/wp/enet-wp031_-en-e.pdf

http://literature.rockwellautomation.com/idc/groups/literature/documents/td/enet-td001_-en-p.pdf


http://literature.rockwellautomation.com/idc/groups/literature/documents/wp/enet-wp034_-en-p.pdf


























Integrated Architecture® Builder (IAB) Updates and additions to better-reflect CPwE

structure, hierarchy and best practices Improved Switch Wizard for distribution (e.g.

Stratix® 5410) and access (e.g. Stratix® 5700 switches)

Implemented VLANs in the EtherNet/IP™

network editor Parallel Redundancy Protocol (PRP) Support CIP™ traffic is measured per segment, not just

controller scanner and adapter centric

System Configuration Drawings Updates and additions to better reflect CPwE

recent enhancements

Networking ResourcesRockwell Automation® Tools

http://www.rockwellautomation.com/global/support/configuration.page?

https://www.rockwellautomation.com/global/detail.page?pagetitle=System-Configuration-Drawings&content_type=tech_data&docid=b481b5f052775d0f7aaf7590d854c4cd

http://literature.rockwellautomation.com/idc/groups/literature/documents/qr/iasimp-qr035_-en-p.pdf

UserGroupTechnical Seminars

Thank you for attending

Resilient EtherNet/IP Networking...2020/04/04 · Catalyst 2955 Cell/Area Zone Switch Stack...

Documents

Transcript of Resilient EtherNet/IP Networking...2020/04/04 · Catalyst 2955 Cell/Area Zone Switch Stack...