Stage 2 Management Systems

Building a foundation for flexible service delivery

Introduction Kelcey Damage

Infrastructure Systems Architect – Cloud Computing

Backbone Technology

The Concept Empower infrastructure to perform its own administration

Free up IT resources from maintaining delivered projects

Automate the post-mortem

Intelligence assisted managed services

Closing The Loop Open Loop Processing

- The main purpose behind Intelligence Assisted Systems/Operations is to leverage the theory of closed feedback loop processing.

- When heating your house with an open loop control system, the parts look like the below diagram:

Closing The Loop Intelligence Assisted Systems/Operations

- To take this a step further and enter into intelligence assisted systems/operations we would make a slight tweak to our open loop system:

Prototyping Intelligence Assisted Systems/Operations

Orange Systems:

Controller

Sensor

Mass Orchestrator

Control Logic

Healing Logic

Broker

Notifications

Sync/Async Tasks

Green Systems:

Report API

Service/Health Scheduler API

Solving Problems – Running in production Intelligence Assisted Systems/Operations

- Initial test runs on production systems have been quite promising

- Most current Heal-Ops are completed in under 2 seconds

- Iterative algorithms allow for state update after each operation pass to mitigate the chance of attempting a Heal-Op on a non-existent error.

- Have been able to heal CRM cluster failures before Nagios can scream

- Can provide data streams for system health and usage

- Fully agent-less design

Why Stage 2? Stage 2 Management Systems – Automated Troubleshooting

- Not re-inventing the wheel, instead managing the failure of common service management tools

- Programmed to heal with IT methodologies

- Not just automating the system, but the system administration

- Works in conjunction with most current infrastructure management tools

How Much Further Can We Go? Stage 3 Management Systems – Predictive Analytics

- Working closely with Metafor Software as an upstream analytics engine

- Providing system resource streams

- State and drift measurement over large clusters

- Pro-active healing actions

- Predict component failure

- Isolate causes of system turbulence or bad cluster cells

- Provide scale and de-scale analysis

- Predict optimum load resource commitments

IASO and IaaS Stage 2/3 Management Systems – Cloud Enhancements

- Application infrastructure auto-scaling

- Using mass-orchestration and CFL processing data

- Predictive scaling for fault and load scenarios

- Currently supporting Apache CloudStack

- Automated guest administration

- Working on agent-less guest tools beyond auto-scaling

- Contemplating adding the code for self user management

Managed Production System Intelligence Assisted Systems/Operations

Example of a multi-tiered application infrastructure running on the cloud and managed by closed feedback loop systems

Application Server

Application Server

Load-Balancer{Nginx}

Load-Balancer{Nginx}

Master Database{MySQL}

Firewall

Memcache: DataMemcache: SessionsMySQL Read

Replication: Slaves on D-RAAS

Inbound Web

Shared Storage

{NetApp}

.

.

Caching Server

{Varnish}

Backbone Technology | Distributed RAAS Cluster v5 rev2

Caching Server

{Memcache}

Caching Server

{Memcache}

Clustered Pool

Firewall

Backup Services GW

Logging{Graylog}

External Monitoring

Internal Monitoring | Self healingAML

Services

Thank You Contact Details

- Kelcey Damage

- kdamage@apache.org

- irc.freenode.net #cloudstack id:kdamage