2nd Jun, 20061 Measurement and Evaluation Yang Qiu Networking Laboratory Helsinki University of...

2nd Jun, 2006 1

Measurement and Evaluation S38.4030

Measurement and Evaluation

Yang Qiu

Networking Laboratory

Helsinki University of Technology

{yangqiu}@cc.hut.fi

mailto:%7Byangqiu%[email protected]

2nd Jun, 2006 2


Autonomic Computing or Communications

Autonomic Computing or Autonomic Communications is a concept that brings together many fields of computing with the purpose of creating computing systems that are reflective and self-adaptive

When reviewing the current autonomic systems, the concept of self management usually groups into having four basic properties:– Self-configuration– Self-optimization– Self-healing– Self-protection

2nd Jun, 2006 3


Software architectures for autonomic computing

The autonomic research activities in software systems can broadly be categorized into four areas:– monitoring of components– interpretation of monitored data– creation of a repair plan (i.e. an adaptation of the system)– execution of a repair plan

And we can group the approaches to autonomic computing systems into categories:– intelligent multi-agent systems– architecture design-based autonomic systems– Hot swapping components

2nd Jun, 2006 4


researches and the domain of Autonomic computing

Main characteristics of Multi-agent systems Domain Group

Communication middleware based on CORBA for monitoring and cooperation

Framework for multi-agent systems

Kuo-Ming, James, Norman

Heartbeat or pulse monitor for monitoring Autonomic components

Sterritt, Bustard

Self-organizing components with a global view expressed as architecture description

Architectural constraints for self-organizing components

Georgiadis, Magee, Kramer

Broker agents used as to provide fault tolerance to overlying problem-solving agents.

Adaptive Agent Architecture

Kumar, Cohen

Framework for building multi-agent systems. Working on including autonomic agents.

ABLE agent toolkit Bigus et al.

2nd Jun, 2006 5



Main characteristics of Architecture design-based autonomic systems

Domain Group

Probes, gauges for monitoring running system, architecture manager implements adaptive behavior, based on architecture-model of system.

Architecture model based adaptation for autonomic systems

Garlan, Schmerl

Components considered as black-boxes. Architecture for fault tolerance in adaptive systems

de Lemos, Fiadeiro

xADL 2.0 architecture description language, c2.fw development framework.

Framework for architecture-based adaptive systems

Dashofy, van der Hoek, Taylor

Autonomic behavior as a distributed multi-agent

infrastructure called Work flakes.

Adding autonomic behavior to existing systems

Valetto, Kaiser

2nd Jun, 2006 6



Main characteristics of Hot swapping components

Domain Group

BARK tool as an extension to EJB to support component replacement.

Reconfiguration in EJB (Enterprise JavaBeans) model

Rutherford et al.

Adaptivity through replacement of bindings between operations and invoked code blocks.

Model for reconfigurable software

Whisnant, Kalbarczyk, Iyer

High-performance hot-swapping of fine-grained

components in K42 OS.

Hot-swapping at OS level

Appavoo et al.

DynamicTAO, a middleware for dynamically reconfigurable software.

Reflective middleware

Kon, Campbell et al.

OpenORB, reflective middleware for self-healing systems.

Reflective middleware

G. S. Blair et al.

2nd Jun, 2006 7


Metrics and evaluation Quality of Service (QoS)

– QoS is the top-level means to compare modern systems – it should reflect the degree to which the system is reaching its primary goal.

– It is typically composed of a number of metrics:• Bandwidth • Delay• Jitter …

Cost– Currently most performance studies of architecture-based autonomic systems have

measured its ability to reach its goal– agent-based systems typically compare the amount of communication, actions performed,

and cost of actions required to reach the goal– For many commercial systems the cost cannot be measured immediately

Granularity/Flexibility– There is a chance of overhead in terms of the global system

• The actions of adaptation are tolerable in a thick-grained component based architecture where the overheads can be hidden in the system’s overall operation and potentially change is not that regular.

• The actions of adaptation are potentially too much for a finer-grained architectures, such as an Hot Swapping OS or Ubiquitous computing.

2nd Jun, 2006 8


Metrics and evaluation (continue)

Failure avoidance (Robustness)– Predictability of failure is an aspect in evaluation– The ability to cope with predicted failure

• using a mean time before failure metric of hardware and others to cope with unpredicted environments

– The ability to cope with unpredicted failure Degree of Autonomy

– failure avoidance– AI and agent-based autonomic systems primarily as their autonomic process

• Decreasing the degree of predictability in the environment and seeing how the system copes could measure this.

• Lower predictability could even reach it having to cope with things it was not designed to. – A degree of proactivity

Adaptivity– Adaptivity is something simple as a parameter of begin changed in a system

• some systems are designed to continue execution reconfiguring, while others cannot. So the adaptation ( for example hot swapping ) of a node impacts the performance

2nd Jun, 2006 9


Metrics and evaluation (continue)

Time to adapt and Reaction Time– The time to adapt is the measurement of the time a system takes to adapt to a change in

the environment– Reaction time is the time between when an environmental element has changed and the

system recognizes that change, decides on what reconfiguration is necessary to react to the environmental change and get the system ready to adapt

Sensitivity– This is a measurement of how well the self-adaptive system fits with the environment it is

sitting in• If a system is highly sensitive to its environment potentially it can cause the system to be constantly

oscillating. Stabilisation

– related to sensitivity– For open autonomic systems the sensitivity is learning how to best reconfigure the system– For closed autonomic systems the sensitivity would be a product of the static rule/constraint

base and the stability of the underlying environment the system must adapt to Benchmarking

– Benchmarking is to bring above metrics together to form some sort of benchmarking tool.

2nd Jun, 2006 10


Study case 1: Kendra audio server

Kendra is a relatively simple system with closed self-adaptation– set the top-level QoS goal to be that the audio quality was as

high as possible while avoiding periods of silence– Also measured the general quality levels, unnecessary

adaptation, missed opportunities to adapt, and sensitivity to environment

– the autonomic intelligence does not grow performance statistics were of a large volume and

difficult to interpret especially in terms of relating behaviour to varying the many tuning parameters and differing environment (networking) conditions

2nd Jun, 2006 11


Study case 2: Measuring the Effectiveness of Self-Healing

It’s possible to quantify the self-healing capability with two metrics: – a measure of how effectively the System Under Test heals itself

in response to the injected disturbances– a measure of how autonomic that healing response is

2nd Jun, 2006 12



Issues and discussion– Quantifying autonomic maturity

• the benchmark’s maturity score to differentiate systems• hope to ultimately map the quantitative measurement of maturity into benchmark index.

– Quantifying healing effectiveness• Only the throughput of correctly-handled requests• extended to capture broader impacts of disturbances

– Accounting for incomplete healing• A complete self-healing cycle includes:

– bypassing the component affected by a disturbance, – repairing the component, – reintegrating them into the system Under Test

• A system Under Test might complete only some of these stages of healing in response to a disturbance,

• A system Under Test might simply tolerate the disturbance without any active healing process

2nd Jun, 2006 13



Issues and discussion (continue)– Accounting for healing-specific resources

• A System Under Test may include resources dedicated to self-healing capability. To prevent situations where a system is intentionally over-provisioned in order to get a good self-healing benchmark score, the cost of these extra resources must be factored into the benchmark result

– Monitors– hot standby cluster nodes– spare disks

– Unified metrics• Ultimately, the results of the self-healing benchmark being reported

on a 2-D space:– one axis measuring effectiveness of self-healing as discussed above, – one axis measuring the cost of the healing capability

2nd Jun, 2006 14


Study case 3: QoS controllerWorkload intensity variation vs. QoS Controller

PerformancePredictability of the workloadIt is more important to correctly compare the

QoS-wise– two points in the search space rather than knowing

their absolute QoS values– AI is need to analysis and to generate a report or a

Benchmark

2nd Jun, 2006 15


Study case 3: QoS controller

Picture above is the work load incoming. Picture Right-upper is the evaluation of Average

QoS.– No Controller– Beam Search– Hill climbing

Picture Right-Lower is the evaluation of Average QoS on different algorithm.

Average QoS will be higher if the predict algorithm is accurate.

2nd Jun, 2006 16


ConclusionsAutonomic computing is an engineering

concept that has found its way in a myriad of computing fields

How to compose the benchmark is still a question

2nd Jun, 20061 Measurement and Evaluation Yang Qiu Networking Laboratory Helsinki University of...

Documents

Transcript of 2nd Jun, 20061 Measurement and Evaluation Yang Qiu Networking Laboratory Helsinki University of...