Faster, Higher, Stronger - OPNFV
Transcript of Faster, Higher, Stronger - OPNFV
Faster, Higher, StrongerAccelerating Fault Management to Next Level
Speakers
Carlos Gonçalves
Software Specialist on the 5G Networks team at NEC Laboratories Europe in Heidelberg, Germany.
He works in the areas of Network Functions Virtualization and Carrier-Cloud Operation & Management
Yujun Zhang
NFV System Engineer from ZTE Corporation
He is current PTL of QTIP in OPNFV, and creator of MitmStack in OpenStack
His main interest focuses on performance testing, analysis and tuning
Doctor project introduction
Doctor is fault management and maintenance project to develop and realize the consequent implementation for the OPNFV reference platform.
● Goals○ build fault management and maintenance framework
■ high availability of Network Services■ immediate notification of unavailability
○ requirement survey○ development of missing feature
● Scope: NFVI, VIM
Role of QTIP in the collaboration
QTIP is the project for "Platform Performance Benchmarking"
● Reveal details behind a simple indicator● Benchmarking of various testing environment and condition
Expected to learn
● How you can enable fast fault mitigation from a rich set of monitoring data sources
● How to fasten NFVI failure event to user● How to leverage performance profiler to find the
bottleneck
A “Strong” unbreakable mobile call
Faster! How?
Notification strategies: conservative
Notification strategies: shortcut
Notification strategies: pros and cons
Conservative+ Cloud resource states are
always up-to-date- Takes longer to report the
alarm out to consumers
Shortcut+ Faster notification to consumer- Cloud resource states could
still be out-of-sync by the time consumer processes the alarm notification
Consumer: User-side Manager; consumer of the interfaces produced by the VIM; VFNM, NFV-O or Orchestrator in ETSI NFV terminology
Notification times comparison (1/3) OpenStack Ocata (DevStack out of the box); 1x Controller, 1x Compute
Notification times comparison (2/3)Same deployment + Congress w/ notification capabilities (draft) & parallel execution driver support (cherry-picked from master)
Notification times comparison (3/3)- Sample outperforms Congress out of the box- Congress is much feature richer supporting dynamic user-defined policies and execution actions on most OpenStack cloud resources.
Issues and challenges
● Passed on Pod-A, but poor result on Pod-B○ Why such difference?
● Performance degradation when scaling up to more servers○ What is the bottleneck?
● Distributed services○ How to collect data from different nodes
Performance Profiler
Get more details behind Pass/Failure
Simple diagnostic
Want more details?
● check log files● check debugging messages ● ...
High technology equipments are helping Doctors in real world
Example in Chrome DevTool: Measuring Resource Loading Times
What does a profiler do
Craft a PoC
Profiler Poc
Still not enough
Inspiration fromOpenStack Summitpresentation
Now we know why
What’s behind `nova reset-state`
How osprofiler works
The implementation is quite simple. Profiler has one stack that contains ids of all trace points. E.g.:
profiler.start("parent_point") # trace_stack.push(<new_uuid>)
# send to collector -> trace_stack[-2:]
profiler.start("parent_point") # trace_stack.push(<new_uuid>)
# send to collector -> trace_stack[-2:]
profiler.stop() # send to collector -> trace_stack[-2:]
# trace_stack.pop()
profiler.stop() # send to collector -> trace_stack[-2:]
# trace_stack.pop()
Supported vs Needed
osprofiler doctor
CINDER
HEAT
KEYSTONE
NOVA
NEUTRON
GLANCE
TROVE
SENLIN
MAGNUM
CEILOMETER
VITRAGE
CONGRESS
AODH
Recommended to track by default
All HTTP calls - helps to get information about: what HTTP requests were done, duration of calls (latency of service), information about projects involved in request.
All RPC calls - helps to understand duration of parts of request related to different services in one project. This information is essential to understand which service produce the bottleneck.
All DB API calls - in some cases slow DB query can produce bottleneck. So it’s quite useful to track how much time request spend in DB layer.
All driver calls - in case of nova, cinder and others we have vendor drivers. Duration
ALL SQL requests (turned off by default, because it produce a lot of traffic)
Challenges in doctor use case
Doctor use case
● Composed by several consecutive steps
● Relies on events for fast notification
● Starts on monitor and ends in consumer
● Multi threaded in inspector
ASYNCHRONOUS
OSProfiler limitation
● Designed for profiling ONE request
● Event notification not tracked
● Must start and end in same thread
● Multi thread is not supported
SYNCHRONOUS
Gaps identified in upstream
osprofiler feature
[ ] multiple thread supporting
No support for osprofiler in Openstack services
[ ] alarming: aodh[ ] inspector: vitrage[ ] inspector: congress
Next in Euphrates
Roadmap: Doctor-QTIP collaboration
● [doctor] Integration of osprofiler in CI jobs● [doctor] Propose changes to upstream to fill gaps
○ Osprofiler enhancement○ Aodh supporting○ Congress supporting○ Vitrage supporting
● [qtip] Benchmarking of notification performance○ Collector backend for profiler data○ Dashboard for performance profile of last build
Questions?https://wiki.opnfv.org/display/doctor
https://wiki.opnfv.org/display/qtip