Building for operations - Reinhardt Quelle
-
Upload
devopsdays -
Category
Technology
-
view
1.555 -
download
2
Transcript of Building for operations - Reinhardt Quelle
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Building Applications for Operability
Reinhardt QuelleOperations [email protected]
rquelle@*
1
V16e
© 2011 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected]) 3
Great stuff, but not that’s not what my team does…
© 2011-2012 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
We host and manage SaaS Applications
4
Millions of Meetingsfor
10s of Millions of Userstotaling
Billions of Minuteseach month
7x24x365
Cisco Social
WebEx
Jabber
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
DevOps
Damon Edwards (DTO)John Willis (now at Dell) suggest
“CAMS” as useful mnemonic.
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
DevOps > Tools
•Automation
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
DevOps > Tools
•Automation
•Metrics
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
DevOps > Tools
•Automation
•Metrics
•Sharing
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
DevOps > Tools
•Culture
•Automation
•Metrics
•Sharing
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
DevOps > Tools
•Culture / Collaboration
•Automation
•Metrics
•Sharing
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Collaboration
Requires:
• shared goals
•communication
•empathy
•
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Yay, We’re going to do DevOps!
http://www.realclearworld.com/blog/2009/01/
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
When you start, its more like this:
http://www.news.com.au/world/are-barack-obama-and-vladamir-putin-holding-highlevel-talks-or-are-they-on-an-awkward-date/story-fndir2ev-1226665803294
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
At a more personal level: remember that first date?
http://www.hercampus.com/love/dating-hooking/9-first-date-disasters-how-deal-them
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Where do we begin?
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Perspective helps (a lot)
http://www.navigonlp.com/blog/stop-look-look-again/boat-land-perspective-2/
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
I come from an Operations background
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Green = Good
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Illustrating the Ops perspective to Engineering
http://cdn.twentytwowords.com/wp-content/uploads/Rhinos-Perspective-e1369734532511-634x447.jpg
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
A quick illustration:A real, albeit deceased, product
(RIP)
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Developer’s View
Compute Node
config A
resolv.conf
ntp.conf
smb.conf
...
config BApp
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Operations’ View
Compute Node
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BApp
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Servers run in "clusters"
Compute Node
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
start/stoporchestration
config Aconfig B
App
Compute Node
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
start/stoporchestration
config Aconfig B
App
Compute Node
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
start/stoporchestration
config Aconfig B
App
Compute Node
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
start/stoporchestration
config Aconfig B
App
Compute Node
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
start/stoporchestration
config Aconfig B
App
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Which means they may have to share data:
spare
LUN
SAN
LUN LUN LUN
Compute Node
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
start/stoporchestration
config Aconfig B
App
Compute Node
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
start/stoporchestration
config Aconfig B
App
Compute Node
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
start/stoporchestration
config Aconfig B
App
Compute Node
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
start/stoporchestration
config Aconfig B
App
Compute Node
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
start/stoporchestration
config Aconfig B
App
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Which means they may have to share data:
spare
LUN
SAN
LUN LUN LUN
Compute Node
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
start/stoporchestration
config Aconfig B
App
Compute Node
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
start/stoporchestration
config Aconfig B
App
Compute Node
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
start/stoporchestration
config Aconfig B
App
Compute Node
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
start/stoporchestration
config Aconfig B
App
Compute Node
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
start/stoporchestration
config Aconfig B
App
Note: there are *much* better ways to share data!
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
spare
LUN
SAN
LUN LUN LUN
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Webmail RPC ADM
We have several front-end servers
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
...which also have to be redundant.
spare
LUN
SAN
LUN LUN LUN
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Webmail RPC ADM
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
spare
LUN
SAN
LUN LUN LUN
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Load Balancer
Webmail RPC ADM
And that means load balancers to direct traffic
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
spare
LUN
SAN
LUN LUN LUN
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Webmail RPC ADM
Load Balancer
spare
LUN
SAN
LUN LUN LUN
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Webmail RPC ADM
spare
LUN
SAN
LUN LUN LUN
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Webmail RPC ADM
There are multiple clusters
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
... and geo-redundant data centers.
spare
LUN
SAN
LUN LUN LUN
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Webmail RPC ADM
Load Balancer
spare
LUN
SAN
LUN LUN LUN
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Webmail RPC ADM
spare
LUN
SAN
LUN LUN LUN
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Webmail RPC ADM
spare
LUN
SAN
LUN LUN LUN
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Webmail RPC ADM
Load Balancer
spare
LUN
SAN
LUN LUN LUN
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Webmail RPC ADM
spare
LUN
SAN
LUN LUN LUN
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Webmail RPC ADM
GSS
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Which means data must be replicated - somehow.
spare
LUN
SAN
LUN LUN LUN
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Webmail RPC ADM
Load Balancer
spare
LUN
SAN
LUN LUN LUN
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Webmail RPC ADM
spare
LUN
SAN
LUN LUN LUN
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Webmail RPC ADM
spare
LUN
SAN
LUN LUN LUN
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Webmail RPC ADM
Load Balancer
spare
LUN
SAN
LUN LUN LUN
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Webmail RPC ADM
spare
LUN
SAN
LUN LUN LUN
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Webmail RPC ADM
GSS
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
But wait! There’s more!
spare
LUN
SAN
LUN LUN LUN
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Webmail RPC ADM
Load Balancer
spare
LUN
SAN
LUN LUN LUN
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Webmail RPC ADM
spare
LUN
SAN
LUN LUN LUN
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Webmail RPC ADM
spare
LUN
SAN
LUN LUN LUN
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Webmail RPC ADM
Load Balancer
spare
LUN
SAN
LUN LUN LUN
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Webmail RPC ADM
spare
LUN
SAN
LUN LUN LUN
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Webmail RPC ADM
GSS
Security Appliances
Monitoring Servers
Identity Servers
Billing Mgt Servers
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
But wait! There’s more!
spare
LUN
SAN
LUN LUN LUN
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Webmail RPC ADM
Load Balancer
spare
LUN
SAN
LUN LUN LUN
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Webmail RPC ADM
spare
LUN
SAN
LUN LUN LUN
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Webmail RPC ADM
spare
LUN
SAN
LUN LUN LUN
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Webmail RPC ADM
Load Balancer
spare
LUN
SAN
LUN LUN LUN
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Webmail RPC ADM
spare
LUN
SAN
LUN LUN LUN
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Postpath Unified Node (PUN)
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BPPSD
cluster manager
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Webmail RPC ADM
GSS
Security Appliances
Monitoring Servers
Identity Servers
Billing Mgt Servers
Security Appliances
Monitoring Servers
Identity Servers
Billing Mgt Servers
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
“Turtles all the way down”
eth0
hostname
/
NIC
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
Service Yconfig Bconfig A
hd
SAN
inst1
eth0 physical
physical hostname
/
NIC
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
Service Yconfig Bconfig A
/
eth0:1 cluster
eth1 mgt
/var..
inst2
eth2 iSCSI, NFS
service hostname
Even individual systems are not like your developer’s workstation:
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Deployment Environment is Complex
• Many nodes in cluster = many “points” to monitor, including systems and services
• Dependencies on external services (DNS, Active Directory, SAN)
• Dependencies within application stack
• Sequencing of operations critical
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
So, lets just use PaaS!
ContainerAppA
$ cf deploy appname
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Uh, Yeah.
DEA 1Ubuntu 12.04
DEA 1Ubuntu 12.04
DEA 1Ubuntu 12.04
DEA ContainerApp
A
Container
AppB
REST ServerUbuntu 12.04
RabbitMQCentOS 6.4 Cassandra
CentOS 6.4CassandraCentOS 6.4
CassandraCentOS 6.4
Cloud ControllerUbuntu 12.04
Ubuntu 12.04NATS
CF Router
RabbitMQCentOS 6.4
Internet
VHOSTA
CassandraCentOS 6.4
Keysp.A
LB/SSL
HTTPS
HTTP
HTTP/SData/Msg Bus
C/C: NATS/HTTP
Keysp.B
VHOSTB
UAM
HM
ContainerAppA'
Container
AppB'
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
I’m not going to discuss application architecture in detail• Designing for Always-on Services:
• We don’t ever stop serving customers (intentionally, anyway)
• Avoiding the Fallacies of Distributed Computing:
Emphasized by load and geography
• The need for loosely coupled services:
Multi-service synchronized updates are extraordinarily costly
• Embracing Failure:
At scale, something is ALWAYS breaking
• Designing for the real world
Circuit breakers, exponential fall-back, dynamic configuration
Read Web Operations (Alspaw, et al), Release It! (Nygaard)
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Running Bits in ProductionWhere do we want the bits to run? Deployment Targets
How do they get there? Packaging/Distribution
How are they supposed to behave? Configuration
How do we start and stop them? Command/Control (aka Orchestration)
Are we sure they are running (or ready to run)?
Health Checks and Verification
When will we run into a wall? Metrics
Ow! A wall! Alerts
How did we end up at that wall? Logs
1
2
3
4
5
6
7
8
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Note: “Monitoring” is multi-faceted
Where do we want the bits to run? Deployment Targets
How do they get there? Packaging/Distribution
How are they supposed to behave? Configuration
How do we start and stop them? Command/Control (aka Orchestration)
Are we sure they are running (or ready to run)?
Health Checks and Verification
When will we run into a wall? Metrics
Was that a wall we just hit? Alerts
What just happened there? Logs
1
2
3
4
5
6
7
8
Monitoring
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Where does a package run?
Service ≠ Server• Multiple servers may run same service
• A single server may run multiple services - even several of the same kind (esp. 32bit)
1
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
One Service, Multiple Hosts
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
Front-End Server
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BService X Service Y
config Bconfig A
LB
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Application Compute Node
config A
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config BApp
Instance config A'config B'
AppInstance
Multiple Services, One Host
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Deployment Target Choices
Where bits run:
Shared system
Virtual Machine
Dedicated server
May depend on:
Balancing Resources
Failover Strategies
Security or Isolation Changes over time
“Refactoring your Infrastructure”
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Packaging/Distribution
• How is software Bundled, Transported and Unpacked/Installed
• Goal is to get a specific version of bits extracted onto a system in a predictable way with its dependencies
2
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Distribution Choices
• Recommended:
• versioned and signed OS package (RPM, Deb) from controlled repository
• puppet “code” is in package, too - puppetfile is source
• Less desirable:
• direct version control repository (svn, git) checkout
• scp tarball
• Note there are different “units of deployment”
• Images, Packages, Applications
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Configuration
• How is this instance supposed to behave?
• How is this server supposed to be configured?
• May be multiple instances on a given server
• Configuration is often changed over time
• Consider “static” and “dynamic” configuration
3
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Standard OS Platform
Postpath Unified Node (PUN)
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
+ Application
Postpath Unified Node (PUN)
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config Aconfig B
PPSD
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Postpath Unified Node (PUN)
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
config Aconfig B
PPSD
Fail
uh, oh...
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Application Configuration• Unique per application instance
• Does not overwrite system/environment
• Application switches:
Select Configapache2 -fnginx -cpostfix -c
Verify Configapachectl configtestnginx -tpostfix check
Reload Changed Configapachectl gracefulnginx reloadpostfix reload
Dump Config postconf
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Various classes of configuration
• Static (ish)
• Environments: Dev, QA, Staging, Production
• Data Centers: SJC, DFW, NRT, AMS
• Clusters: ISJ1, AC
• Use Puppet/Chef/cfengine
• Dynamic
• Cluster membership
• Service hosts: DB, Logs, Metrics, Identity
• Use Zookeeper/Noah/Eureka/DNS - MX, GSS/Anycast
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Common Practice:Configuration Management
Compute Node
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
configurationdaemon
config Aconfig B
App
Compute Node
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
configurationdaemon
config Aconfig B
App
configurationmaster+
Environmental Data - Puppet Hiera - Chef Databags
Cluster Spec - * - TOSCA - CloudFormation - Heat
Machine Spec - Puppet Modules - Chef Cookbooks
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Need a new Dev/QA test environment?
Compute Node
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
configurationdaemon
config Aconfig B
App
Compute Node
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
configurationdaemon
config Aconfig B
App
mstr A
+Environmental Data - Puppet Hiera - Chef Databags
Cluster Spec - * - TOSCA - CloudFormation - Heat
Machine Spec - Puppet Modules - Chef Cookbooks
Compute Node
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
configurationdaemon
config Aconfig B
App
Compute Node
configsshd
configrsyslog
configinetd /etc/sysconfig/*
/etc/logrotate/*
/etc/snmp/*
/etc/passwd
...LVM
/etc/*
....
resolv.conf
ntp.conf
smb.conf
...
...DR:BD
configurationdaemon
config Aconfig B
App
mstr B
Environmental Data - Puppet Hiera - Chef Databags
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Make applications configuration-friendly
Select Configapache2 -f /path/to/confignginx -c /path/to/configpostfix -c /path/to/config
Verify Configapachectl configtestnginx -tpostfix check
Reload Changed Configapachectl gracefulnginx reloadpostfix reload
Dump Config postconf
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Make configuration cooperative and configuration management friendly:
logrotate include /etc/logrotate.d
monit include /etc/monit.d
sudo #include /etc/sudoers.d
Sensu“Deep Merge” of JSON config
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Command and Control(aka “orchestration”)
• Starting and stopping services usually requires specific sequence
• By definition, cluster has multiple systems on which processes must be controlled
• Sequencing must be enforced, repeatable
• Dependencies must be managed
• Ad-hoc processes may be required on demand
4
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Many tools available
• Fabric
• Capistrano
• MCollective
• RunDeck
• And many, many more
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
So now we’re running...
Sure about that?
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Are We Alive?
• Health Checks provide specific status
• Often variable depth:
• shallow/frequent, deep/less frequent
• Many Consumers
• Cluster managers, load balancers, process monitors
• Personal favorite - “Pre-Flight checks”
• Call on demand
5
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Now you can run, but...
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Where are we headed?• Metrics help us understand trends and predict future...
• or give us historical context for some event.
• Collected by central tools for every device in environment
• Operations team can deal with server, environment
OS
App
Network Equipment
Monitoring
?
6
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Application Metrics
•Metrics used in development for profiling often continue to be interesting in production
•Highly specific to a specific service:
number of open connections quota usage
message duration message size distribution
AD Lookup response time, uptimemessage distribution across users/folders
cache hit/miss rate MAPI operation count
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Example Application Metrics
Apache $ apache2ctl status
memcached
$ echo -ne "stats\r\n" | nc -i1 localhost 11211 ...STAT curr_connections 2STAT total_connections 71072406STAT connection_structures 11STAT cmd_get 58419832STAT cmd_set 4127175STAT get_hits 53793350STAT get_misses 4626482...
squid
[calh@photocache3 ~] $ /usr/sbin/squidclient -p 80 cache_object://localhost/5min...client_http.all_median_svc_time = 0.000911 secondsclient_http.miss_median_svc_time = 0.042766 secondsclient_http.nm_median_svc_time = 0.000000 secondsclient_http.nh_median_svc_time = 0.024508 secondsclient_http.hit_median_svc_time = 0.000911 seconds...
mysql
mysql> SHOW SLAVE STATUS \G... Connect_retry: 60 Master_Log_File: db1-bin.439 Read_Master_Log_Pos: 374103193 Relay_Log_File: dbslave-relay.125 Relay_Log_Pos: 854427197 Replicate_ignore_db: Last_errno: 0 Last_error: Skip_counter: 0 Exec_master_log_pos: 587702364 Relay_log_space: 854431912...
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Again, many tools available
• JMX
• NewRelic
• statsd (extremely light-weight and easy to use)
• snmp (highly standard, but high barrier to use and configuration for application layer)
• collectd + graphite
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Alerts
• Actionable
• May be derived from other sources, or raised directly.
• Have to be routed to be useful - thank you, Sean!
•The application teams will get these!
7
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
What have you been up to?
• Logging key for debugging, tracing, compliance, etc.
• Logs should be collected off-box for multiple constituents, including:
Operations, Support, InfoSec/Compliance, Engineering
• Maximum utility when levels manageable at runtime without application restart
• Logstash, Splunk... no excuses
8
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Logs vs. Stats
• Logs can be fall-back facility for monitoring of health and metrics
But this is less than desirable
- Shifts development to ops, usually late in the game
- Can be brittle
- Usually less efficient
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Logging in Production
• Clustered, parallel processes require host and process IDs
• Activity flows across services - how do we correlate?
• Audit logs important - who did what when?
• Keep privacy in mind - if logs have private data, we can’t easily take them out of production.
• Passwords must never be logged
© 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
“Wow, that’s a lot we have to build!”
© 2013 Cisco and/or its affiliates. All rights reserved. Reinhardt Quelle ([email protected])
Or not.
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
If applications provide standard management
interfaces...
App
Logging
Metrics
Health
Process Mgt
Config
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
OSLogging
Metrics
Health
Process Mgt
Config
Config
Config
Log HostLogging
Stats HostMetrics
Alert ManagerHealth
Cluster/Orchestration ManagerCM
Configuration ManagerConfig
VCS
Operations
...they can “plug in” to existing infrastructure
App
Logging
Metrics
Health
Process Mgt
Config
Engineering
Reinhardt Quelle ([email protected])© 2013 Cisco and/or its affiliates. All rights reserved.
Operation’s Asks
• Make Operability an integral part of product requirements and designs
• Engage your operations team as soon as possible to let us help you leverage our platform and services
• Talk with us about HA architectures: we don’t have a magic wand!