Predictive Analytics & Visualization - Towards Data Driven Insights for Openstack Clouds

16
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Cisco Confidential © 2013 Cisco and/or its affiliates. All rights reserved. Predictive Analytics & Visualization Towards Data Driven Insights for Openstack Clouds

description

Slightly edited deck presented by myself and my awesome team at the Openstack Summit in May 2014 at Atlanta.

Transcript of Predictive Analytics & Visualization - Towards Data Driven Insights for Openstack Clouds

Page 1: Predictive Analytics & Visualization - Towards Data Driven Insights for Openstack Clouds

© 2013 Cisco and/or its affiliates. All rights reserved. Cisco

Confidential Cisco Confidential © 2013 Cisco and/or its affiliates. All rights reserved.

Predictive Analytics & Visualization Towards Data Driven Insights for Openstack Clouds

Page 2: Predictive Analytics & Visualization - Towards Data Driven Insights for Openstack Clouds

© 2013 Cisco and/or its affiliates. All rights reserved. Cisco

Confidential Cisco Confidential Cisco Confidential © 2013 Cisco and/or its affiliates. All rights reserved.

Cloud/Big-data Winnovation Team

Alex Holden Damian Van Vuuren Ailing Zhang

Marc Solanas Matt Weeks Pengfei Zhang

Kai Zhang

Xinyuan Huang

Debo Dutta

Yathiraj Udupi

Page 3: Predictive Analytics & Visualization - Towards Data Driven Insights for Openstack Clouds

© 2013 Cisco and/or its affiliates. All rights reserved. Cisco

Confidential

Visualization in OpenStack today

Page 4: Predictive Analytics & Visualization - Towards Data Driven Insights for Openstack Clouds

© 2013 Cisco and/or its affiliates. All rights reserved. Cisco

Confidential

Tomorrow?

Sources: 2001 A Space Oddesy, MGM. Minority Report, Twentieth Century Fox Film Corporation. Mad Men, AMC

OpenStack, deploy me an

app

Page 5: Predictive Analytics & Visualization - Towards Data Driven Insights for Openstack Clouds

© 2013 Cisco and/or its affiliates. All rights reserved. Cisco

Confidential

Our Approach: Predictive Analytics & Awesome Visualisation Why Predictive? ●  OpenStack has lots of moving parts ●  Rule/Model/Policy based system hard without insight

Why Predictive? ●  Ops folks want to know right now!

Why Visualisation? ●  Easy to observe insights by humans ●  East way to consume lots of information from loads oflogs/metrics ●  Improve user experience

Page 6: Predictive Analytics & Visualization - Towards Data Driven Insights for Openstack Clouds

© 2013 Cisco and/or its affiliates. All rights reserved. Cisco

Confidential Cisco Confidential © 2013 Cisco and/or its affiliates. All rights reserved.

Digital Exhaust = Metrics/Logs What information can we get from our cloud? Where can we get it?

●  Ceilometer ●  OpenStack Syslog ●  Project APIs (Nova, Neutron, Cinder etc)

What can we do with it? ●  Extract Insights ●  Figure out what failed and why ●  Move workloads around ●  Detect attacks on the cloud

●  Physical Servers ○  CPU ○  Hardware Faults ○  OpenStack Logs ○  OpenStack Messaging Queue

●  Tenant/Project Data ○  Quotas

●  Virtual Machines ○  Config ○  CPU ○  Disk ○  Network Flows ○  Performance ○  Latency

●  Application Level ○  Application Logs ○  Application Requirements ○  Application Performance

Page 7: Predictive Analytics & Visualization - Towards Data Driven Insights for Openstack Clouds

© 2013 Cisco and/or its affiliates. All rights reserved. Cisco

Confidential Cisco Confidential Cisco Confidential © 2013 Cisco and/or its affiliates. All rights reserved.

AVOS Dashboard

Page 8: Predictive Analytics & Visualization - Towards Data Driven Insights for Openstack Clouds

© 2013 Cisco and/or its affiliates. All rights reserved. Cisco

Confidential

AVOS: Summary ●  Analytics and Visualization on OpenStack ●  Stateless: easy deployment ●  Client side architecture:

○  Polls some OpenStack APIs ○  Listens to the messaging bus ○  Minimum overhead

●  Provides users and service providers full insight into cloud

○  Real time info on all instances/networks etc by project and as admin ○  Historical data on performance ○  Stack-search: Search for all components in the cloud, quickly. ○  Multiple select (See combined data)

Page 9: Predictive Analytics & Visualization - Towards Data Driven Insights for Openstack Clouds

© 2013 Cisco and/or its affiliates. All rights reserved. Cisco

Confidential

https://www.youtube.com/watch?v=vyb-Cd4VEDg

AVOS Demo

Page 10: Predictive Analytics & Visualization - Towards Data Driven Insights for Openstack Clouds

© 2013 Cisco and/or its affiliates. All rights reserved. Cisco

Confidential

Bottlenecks

Use Cases: Why use AVOS?

Attacks

Page 11: Predictive Analytics & Visualization - Towards Data Driven Insights for Openstack Clouds

© 2013 Cisco and/or its affiliates. All rights reserved. Cisco

Confidential

Use Case: Bottlenecks

Page 12: Predictive Analytics & Visualization - Towards Data Driven Insights for Openstack Clouds

© 2013 Cisco and/or its affiliates. All rights reserved. Cisco

Confidential

Use Case: Attack Detection ●  Extract Metrics ●  Analyse Patterns ●  We can differentiate

between: ○  Hadoop Traffic ○  Network Faults ○  DDoS Attack

Page 13: Predictive Analytics & Visualization - Towards Data Driven Insights for Openstack Clouds

© 2013 Cisco and/or its affiliates. All rights reserved. Cisco

Confidential

Future Plans •  Advanced Search Primitives •  Reducing the volume of logs •  Real-time error analytics •  Add to/edit the cluster •  Ability to preview changes

Curvature:

https://github.com/CiscoSystems/curvature

Page 14: Predictive Analytics & Visualization - Towards Data Driven Insights for Openstack Clouds

© 2013 Cisco and/or its affiliates. All rights reserved. Cisco

Confidential

Notes: OpenStack changes •  Ceilometer

Better API queries, optimizing return structure Custom meters: VM to VM traffic flows

•  Other suggestions/Ideas

Standardization of the Python clients (several inconsistencies between method calls) Integration into horizon

Page 15: Predictive Analytics & Visualization - Towards Data Driven Insights for Openstack Clouds

© 2013 Cisco and/or its affiliates. All rights reserved. Cisco

Confidential

Conclusion

Openstack Predictive Analytics/Visibility = Reduce Ops pains We must 1) gather data 2) extract insights 3) present to ops folks with awesome UX

Page 16: Predictive Analytics & Visualization - Towards Data Driven Insights for Openstack Clouds

© 2013 Cisco and/or its affiliates. All rights reserved. Cisco

Confidential

Thank you. Please take our survey: http://goo.gl/CMAZje

Coming Talk: Storage Visibility and Optimization for OpenStack:

Wednesday, May 14 • 3:30pm - 4:10pm