Conf2013 bchristensen thebig_t


Infrastructure Operations


Monitoring for the big T.


About Me


Roy Christensen: 35 years3C > HoneyWell > Digital Equipment CorpMike Christensen: 40 yearsDEC > Compaq> Hewlett Packard

Beau Christensen: 17 yearsAstraZeneca > Send.com > DirecTV > Ping Identity

3rd Generation Computer Geek

92 years family experience (hah)


• We believe secure professional and personal identities underlie human progress in a connected world. Our purpose is to enable and protect identity, defend privacy and secure the Internet.

• Over 1,000 companies, including over half of the Fortune 100, rely on our award-winning products to make the digital world a better experience for hundreds of millions of people.

• Denver, Colorado. Est. 2003

About Ping Identity


Site Reliability Engineering (SRE)Production Web OperationsOps to On-Demand Dev

Configuration Engineering (CFE)Automation, Deployment, LabsOps to On-Premise Dev

Infrastructure Engineering (IFE)Iron, Network, SecurityOps to Support & Helpdesk

Infrastructure Operations @Ping


(Quickie) Architecture

• Hybrid Cloud Application• VMware/AWS• SOA (lots of little services)• Red/Black Deployment• Cassandra & Galera MySQL


Splunk Systems

• 7 Indexers • Distro search across 3 regional

data centers• ~ 40Gib/day• No clustering (yet)• Universal FWD installed in all

templates• Moving all Splunk to clustered

architecture in VPCs.


Trust(the big T)


Because uptime is dead.

http://www.kitchensoap.com/2013/01/03/availability-nuance-as-a-service/


What Builds Trust?


Lots of Things…

Honesty

Integrity

ReliabilityDependabilitySecurity

TransparencyMaturity


We rolled those up into two.

Reliability & Transparency


Reliability:Keep Shit Running


Keeping Shit Running• Architecture

– SOA (duh!!!!)– Highly Automated Deployments– Active/Active, quick failover, multi-region

• Tools– We use a ton of tools– We constantly question the tools we use– We are always looking for new tools

• Security– Eyes always on– Scanners running continuously– Constant Remediation


Transparency:Talk About Shit


Talking About Shit• Talk Publicly

– Talk about decisions we make– Talk about architecture– Talk to vendors– Talk to customers

• When Stuff Breaks, Talk More– Quick & dirty status updates– Verbose post mortems– Be honest

• Metrics– Use 3rd Parties– Expose as much monitoring as you can– Make it relevant to customers


T = r + t

Trust = reliability + transparency


Trust is a Zero Sum Game

Trust

Time


Trust

Time

T = +2


Trust

Time

T = -2


Trust

TimeT = 0


Monitoring is the foundation

of Trust.


is the backbone of our monitoring

strategy.


<3 MonitoringThe right tool for the right job = Lots of tools.


Lots of tools = Best of Breed, orStack Monitoring


Let’s build the stack.


Transport & Network Layer.

Fundamental interactions between services. Extremely important to understand, forms the basis of troubleshooting efforts.

“Can you ping it? Tracert? Tcpdump?”


Instance Resource MonitoringHistorical resource consumption, collected and correlated with incidents and system events. Scalability intelligence.

“Traditional” systems monitoring. Nagios, Zenoss, Zabbix, etc.


Machine Data, Logs.

Extraordinary versatility: events, alerting, reporting, security, business metrics, and OI. Allows the creation of dashboards and event types relative to your own environment.

Events, logs, traditional “syslog stuff.”


Application Performance Monitoring

Detailed view of how the applications are running on top of the rest of the stack. Identify bottlenecks, architecture issues, and accurately model user experience with RUM.

“Page loads are slow.” “Why are there SQL queries that return 30,000 rows?”


Availability Monitoring

External agents monitor services from multiple locations around the globe. Use application heartbeats, not ICMP or TCP. Provides near real-time alerting and is immediately visible to the customer.

“The site is down.”


Splunk is the system’s mortar.

Binding systems together, filling gaps, creating stability.


Security

Quick visual impact. Easy to identify location, frequency, and network address of aggressor. Quick drill down into detailed Splunk searches.


Traffic

Real-time traffic maps are immediately recognizable to everyone in the organization.


Operations

Operational dashboards provide a quick overview of system health by displaying error correlation with traffic levels and event type heat maps.


Global Load Balance

Shows traffic distribution between production data centers.


Maximize Machine Data Layer VersatilityBuild your own dashboards (it’s easy).

Maintain proper naming conventions:na-den-www-13-8-17-0ec5e8da-128-193

Design for wetware.

Ask for feedback.

Maintain event types.


Having tons of tools is great. Each layer of the stack is matched well with the system’s requirements…

Aggregation of metrics is the new challenge. How do you see it all?


Stack Integration


eventtype=critical.*

Application integration allows speedier access to event timeline and context of the alarm using markers. Allows SRE to gather relevant information quicker, using the same tools with less screens.

Stack Integration = MTTR Speed


Splunk has a TON of integrations.


See? -->


Always Need More.


Remember…

T = r + t


Keep Shit Running.


Talk About Shit.


Use Splunk.


Questions?


Thanks!

[email protected]@beauchristensenwww.pingidentity.com/blogshttp://status.pingidentity.com

https://twitter.com/beauchristensen

https://twitter.com/beauchristensen

http://www.pingidentity.com/blogs

http://www.pingidentity.com/blogs

http://status.pingidentity.com/

http://status.pingidentity.com/

Conf2013 bchristensen thebig_t

Technology

Transcript of Conf2013 bchristensen thebig_t