www.s-cube-network.eu

S-Cube Learning Package

Run-time Verification for Preventive Adaptation

University of Duisburg-Essen (UniDue), Center for

Scientific and Technological Research (FBK), Politecnico

di Milano (POLIMI)

Eric Schmieders, UniDue

© S-Cube

Learning Package Categorization

S-Cube

Quality Definition, Negotiation

and Assurance

Quality Assurance and Quality Prediction

Run-time Verification

for Preventive Adaptation

Learning Package Overview

Problem Description

Preventive Adaptation based on Runtime-Verification

Discussion

Conclusions

© S-Cube

Service-oriented Systems Need for Adaptation

Highly dynamic changes due to

– 3rd party services, multitude of service providers, …

– evolution of requirements, user types, …

– change in end-user devices, network connectivity, …

Difference from traditional

software systems

– Unprecedented level of change

– No guarantee that 3rd party

service fulfils its contract (SLA)

– Hard to assess behaviour

of infrastructure at design time

© S-Cube

Let’s Consider a Scenario (1)

Assume you are a provider of a composite service

Commercial customers order products via a Web service

interface

Your process checks an internal stock, orders some missing

parts, and assembles the product

© S-Cube

Let’s Consider a Scenario (2)

(Of course) there are contractual obligations

– Delivery in time

– Availability of your service

– Product quality

These can be formulated using Service Level Agreements

© S-Cube

– Note: in this context not necessarily

WSLA style machine processable

SLAs – can also be just regular

contracts

Let’s Consider a Scenario (3)

As a provider, you want to receive timely notifications if these

obligations are likely to be violated

– Usually there are penalties to be paid if contractual obligations are

(repeatedly) violated

– Even if this is not the case, customer satisfaction will suffer and

existing customers may terminate their contracts

© S-Cube

Based on this notifications …

– … countermeasures can be taken (i.e.,

adaptations can be triggered)

– … customers can be notified

Learning Package Overview

Problem Description

Preventive Adaptation based on Runtime-Verification

Discussion

Conclusions

© S-Cube

Reactive Adaptation

– Repair/compensate for external failure visible

to the end-user

Preventive Adaptation

– An internal failure/deviation occurs

Will it lead to an external failure?

– If “yes”: Repair/compensate internal

failure/deviation to prevent external failure

Proactive Adaptation

Is internal failure /deviation imminent

(but did not occur)?

– If “yes”: Modify system before internal

failure/deviation actually occurs

Types of Adaptation

Failure? Failure!

Failure?

Failure!

For Preventive Adaptation

– Data Mining: Extract knowledge from “historical” data - TUW & USTUTT: “Prediction and Prevention of SLA Violations in Composite Services”

- SZTAKI: “Historical Data Based Predictions for Resource Allocation”

–Run-time Verification: Formally ascertain that properties hold - Paluno, FBK, Polimi: “Assumption-based Run-time Verification”

For Proactive Adaptation

– Simulation: Execute dynamic models to simulate behavior - UPM & TUW: “Simulation of Provision Resources”

- INRIA: “QoS Assurance Using Discrete-Event Simulation”

– Static Analysis: Fomally infer properties (without execution) - UCBL & UPM: “Soft Constraints for QoS-Aware Service Selection”

- UPM: “Data-aware Resource Analysis for Service Orchestrations”

– Online Testing: Actively execute services in parallel to normal use - Paluno & UPC: “Augmenting Monitoring with Online Testing”

- SEERC & Paluno : “Just-in-time Testing of Conversational Services”

Types of Adaptation

[Deliverable JRA-1.3.5, http://www.s-cube-network.eu/qp/]

© S-Cube

Background: Runtime Verification

© S-Cube

Runtime Verification is a system analysis approach based on extracting

information from a running system extracted information is used to detect and

possibly react to observed behaviors satisfying or violating certain properties.

Properties can be functional (certain functionality) or non-functional (e.g. required

response time)

Runtime verification specifications are typically expressed in regular expressions,

context-free patterns, linear temporal logics, etc.

Based on formal requirements specifications, monitors are created and applied

during the systems runtime

Runtime verification can be used for many purposes: security or safety policy

monitoring, debugging, testing, verification, validation, profiling, fault protection,

etc…

[BianculliEtAl2008, GehlertEtAl2010, JRA-1.3.2,

http://en.wikipedia.org/wiki/Runtime_verification]

SPADE An Example for Runtime Verification

© S-Cube

Background:

S-Cube Service Life-Cycle

A life cycle model is a process model that covers the activities related to

the entire life cycle of a service, a service-based application, or a software

component or system. [Gu2007, http://www.s-cube-

network.eu/km/terms/l/life-cycle-model]

The SPADE approach uses runtime verification for preventive adaptation

ON THE NEXT SLIDES: SPADE will be used to illustrate preventive

adaptation by applying run-time verification

Furthermore

– (A) an abstract example scenario and

– (B) the S-Cube Service Life-Cycle

will be used to illustrate the design time and run time activitives of

SPADE

Runtime Verification with SPADE

SPADE (Specification and Assumption based Detection of Adaptation

Needs) equips SBAs with adaptation capabilities, empowering them to

adapt preventively [MetzgerEtAl2010, SchmiedersEtAl2011]

TO ACHIEVE THIS: SPADE uses

– run-time verification techniques

– execution data of the monitored instances

– assumptions concerning the SBAs' context, e.g. the response time of

constituent services (derived from SLAs: http://www.s-cube-

network.eu/km/terms/s/service-level-agreement)

– formalized requirements of the SBA

This data is used for performance prediction and preventive adaptation

which is able to avoid requirement violations of running SBAs

© S-Cube

Action 5

:FastWeather

:CurrencyConverter

:HyperlinkExtractor

Action 7

:WSAmazonBox

Action 2 :Google

Action 1

Action 4

:GetJoke Action 6

Action 3

5900 ms

8400 ms

6400 ms

14000 ms

2700 ms

4000 ms

Abstract Example

© S-Cube

Action 5

:FastWeather

:CurrencyConverter

:HyperlinkExtractor

Action 7

:WSAmazonBox

Action 2 :Google

Action 1

Action 4

:GetJoke Action 6

Action 3

5900 ms

8400 ms

6400 ms

14000 ms

2700 ms

4000 ms

Abstract Example

Activity

Diagram

of Workflow

3rd Party

Service

Assumed

response time

Service Binding

© S-Cube

S-Cube Service Life-Cycle

Evolution

Deployment &

Provisioning

Identify

Adaptation

Need

Identify

Adaptation

Strategy

Enact

Adaptation

Adaptation

Operation &

Management

Design

Requirements

Engineering

Realization

Requirements

R := {r1, r2 …}

Workflow

Specification S,

Service

Candidates C

SLA Negotiation

A:= {a1, a2, …} Deployment

Check: S, A |= R

Monitoring

m1 |= a1

(vgl. SM)

Check: S, A‘, M |= R

[MetzgerEtAl2010, SchmiedersEtAl2011]

Solve:

C, S |= R

Create:

Plan p Enact p

© S-Cube

Requirements Engineering

In the requirements engineering phase, the functional

and quality requirements for the SBA are elicited and

documented!

GOAL: enable check, whether application

will from its requirements during operation?

functional and non-functional requirements

are formally expressed as input to SPADE

SPADE uses the specification language ALBERT to formalize the SBA

requirements

© S-Cube

Requirements

Engineering

Requirements

R := {r1, r2 …}

For more information on ALBERT:

Bianculli, D., Ghezzi, C., Spoletini, P., Baresi, L., Guinea, S.: A guided tour through

SAVVY-WS: A methodology for specifying and validating web service compositions.

In: Borger, E., Cisternino, A. (eds.) Advances in Software Engineering. LNCS, vol.

5316. Springer (2008)

Requirements Engineering

Example:

– Formalize Requirement r: end-to-end response time is at most 55

seconds

– ALBERT is used to formalize r:

r := onEvent(start, "Action1") -> Within( onEvent( end,

"Action7"), 55000)

– onEvent operator evaluates to true if the activity specified in its

second argument performs the state change denoted in its first

argument.

– Within operator evaluates to true if its first argument evaluates to true

within the amount of milliseconds specified in its second argument

© S-Cube

Design

During the design phase, the activities and

the control flow of the application are

specified!

Definition of workflow in e.g. BPEL

Candidate services are identified that can

provide the required functionality/quality

SPADE extends idea of Bianculli et al. [BianculliEtAl2008] by using a

model checker during run-time

ALBERT expressions can be executed by BOGOR: formalize the workow

using BIR, the BOGOR Input Representation.

© S-Cube

Design

Workflow

Specification S,

Service

Candidates C

Design

Example:

BIR file (excerpt) as input for the BOGOR Model Checker:

© S-Cube

…

Background: Model Checking

© S-Cube

With model checking it is automatically testable, whether a model meets a given

specification.

Concept is general and applies to all kinds of logics and suitable structures

Example: verifying whether a given formula in the propositional logic is satisfied by

a given structure

Typically hardware or software systems are checked

The checked specification contains safety requirements, e.g. the absence of

deadlocks, critical states that can cause the system to crash, …

The model of the system and the specification are formulated in precise

mathematical language (e.g. LTL, CTL…)

Common model checker: http://nusmv.fbk.eu/, http://spinroot.com/spin/whatispin.html,

http://www.uppaal.org/, and http://babelfish.arc.nasa.gov/trac/jpf

[GehlertEtAl2011, Bianculli2008, http://de.wikipedia.org/wiki/Model_Checking]

Realization

To achieve the desired end-to-end quality of

SBAs, contracts between service providers

and service consumers on quality aspects

of services are established!

For each candidate service: negotiate best quality

level for the available budget & stipulated in an SLA

SPADE: the quality levels are formalized, that have been negotiated and

agreed upon with the service providers

Quality levels are used as assumptions A about the SBA's context

ALBERT is used to formalize A

© S-Cube

Realization

SLA Negotiation

A:= {a1, a2, …}

For more information on negotiating SLAs:

Comuzzi, M., Pernici, B.: A framework for qos-based web service contracting. ACM

Transactions on web 3(3) (2009)

Realization

Example:

– For our example we use the assumed response time given in the

example SBA.

– The assumption for the aFastWeather service bound to Action 1, is formalized

as follows1:

aFastWeather := onEvent (start, "Action 1")

Within( onEvent(end, "Action 1" ), 5900)

© S-Cube

Background:

Assumptions

In the case of service-based applications, assumptions may characterize the

constituent services (e. g., their interfaces, QoS parameters, etc.) and/or the

context (e. g., infrastructure, business, context, user, etc.).

A violation of those assumptions may lead to a situation, in which the software

system does not provide the expected quality anymore and, therefore, deviates

from its requirements.

[GehlertEtAl2010]

Deployment

The deployment and provisioning phase

comprises all the activities needed to make

the SBA available to its users!

SPADE uses BOGOR to check whether the

workflow specification (S), under the given

assumptions (A), satisfies the requirements (R),

i.e. whether:

S, A |= R

If requirements are not satisfied phases of the evolution loop are re-

executed, e.g., in order to bind faster services.

Example:

– The specification of the abstract service composition evaluates to true, i.e. S

and A satisfy R. Thus, the SBA is deployed.

© S-Cube

Deployment &

Provisioning

Deployment

Check: S, A |= R

Operation and Management

This phase comprises the execution of

the SBA and the monitoring of its

constituent services using service

monitoring techniques!

To identify assumption violations

during the operation of the SBA monitoring

mechanisms are employed

SPADE uses the monitoring facilities of the runtime environment to

CHECK: m |= a

Example:

– Assume: mFastWeather and mGoogle satisfy their related assumptions

– BUT: WSAmazonBox too late mGoogle |≠ aGoogle.

– CONSEQUENCE: Due to the deviation the next phase is entered

immediately; a performance violation could be indicated!

© S-Cube

Operation &

Management

Monitoring

m1 |= a1

(vgl. SM)

Identify Adaptation Needs

In this phase the need for the adaptation

is detected. To this end a runtime

verification against the SBAs requirements

is performed!

SPADE checks: are requirements still satisfied?

For all invoked services:

– SPADE replaces the assumption (A) values by concrete monitoring data (M)

– only for the not invoked services, assumptions (A’) are used

SPADE thus uses a subset A’ (of A), together with the set of monitored

data M to check whether: S,M,A’ |= R

SPADE utilizes BOGOR to perform this verification during run-time

If there is one future path which violates r adaptation must be

triggered

ELSE the workflow execution is continued

© S-Cube

Identify

Adaptation

Need

Check: S, A‘, M |= R

Identify Adaptation Needs

Example:

– Check whether the requirement is still met (S,M,A’ |= R):

- workflow specification S

- the monitoring data (mFastWeather, mGoogle and mWSAmazonBox)

- assumptions of the to be invoked services (aHylinkExtractor, aCurrencyConverter,

aGetJoke)

checked against the requirement r.

– Predicted end-to-end duration: 56238 ms exceeds the 55 seconds

demanded by r.

– CONSEQUENCE: the workflow has to be adapted, otherwise a

performance violation seems to be in all probability

© S-Cube

Action 5

:FastWeather

:CurrencyConverter

:HyperlinkExtractor

Action 7

:WSAmazonBox

Action 2 :Google

Action 1

Action 4

5735 ms

:GetJoke Action 6

Action 3

5900 ms

8400 ms

6400 ms

14000 ms

2700 ms

4320 ms

4000 ms

15083 ms

Identify Adaptation Needs

Re

sp

on

se

Tim

es

6400 ms

14000ms

2700 ms

Action 5

:FastWeather

:CurrencyConverter

:HyperlinkExtractor

Action 7

:WSAmazonBox

Action 2 :Google

Action 1

Action 4

5735 ms

:GetJoke Action 6

Action 3

5900 ms

8400 ms

6400 ms

14000 ms

2700 ms

4320 ms

4000 ms

15083 ms

FAILURE

Identify Adaptation Needs

Re

sp

on

se

Tim

es

A

ss

um

pti

on

s

∑ > e2e_requirement(55 ms)

e2e is predicted to be violated!

Adaptation necessary!

Identify Adaptation Strategy

In this phase an adaptation strategy is created.

The strategy is supposed to avert the menacing

SLA violation successfully when enacted!

SPADE is equipped with service substitution

capabilities

SPADE exploits the CHOCO constraint solver

to determine which not-yet-invoked services

have to be substituted (http://www.emn.fr/z-info/choco-solver/)

Find a strategy with CHOCO for every future violating path (necessary, as

it’s unclear which path will be taken in the further execution)

Example:

– Two paths are violating, i.e. p1 = (Action5, Action7) and p2 = (Action5,

Action6, Action7) paths will violate r and therefore have to be adapted.

– Based on the output of CHOCO: services for Action5 and Action6 are chosen

to be substituted by faster services

© S-Cube

Identify

Adaptation

Strategy

Solve:

C, S |= R

and create

Plan p

Enact Adaptation

During this last adaptation phase the adaptation

strategy is executed. For this purpose, the

instructions comprised in the adaptation strategy

are dispatched!

Dispatching an adaptation strategy usually utilizes

– the facilities provided by the chosen run-time environment

– involves additional adaptation mechanisms

SPADE:

– uses the interception-mechanisms provided by the runtime

environment to reroute messages to the new target service

– switchs the target service of the invocation to the service identified

during the previous phase.

© S-Cube

Enact

Adaptation

Enact p

Enact Adaptation

Example:

– In our example, the two service invocations Action5 and Action6 are

redirected to the substituting services.

– the message routing table comprising the message destinations is

manipulated.

Consequently, the execution of the instance is resumed..

© S-Cube

Learning Package Overview

Problem Description

Preventive Adaptation based on Runtime-Verification

Discussion

Conclusions

© S-Cube

Is Run-time Verification effective when adapting preventively? SPADE can be considered as one representative for Run-time

Verification techniques

We conducted an experiment to access the effectiveness of

SPADE

Our experimental measurement of SPADE's effectiveness is

twofold

– false positives: unnecessary adaptations are measured

– Impossible adaptations: amount of situations in which SPADE is

unable to perform an adaptation is counted

© S-Cube

Is Run-time Verification useful for adapting preventively? Determining the Degree of False Positives

1. Calculated Response Time:

– Calculate the end-to-end response time for each SBA instance execution

– Add the monitored response times of the invoked services along the SBA

instance's path

Based on this determine false positive adaptation triggers:

2. SPADE triggered an adaptation

3. Check: calculated SBA instance response time violates the SBAs end-to-end

requirements?

False Positive, if: 1, 2, and 3 reveal that the requirement would not have

been violated in case the SBA execution continues without adaptation

© S-Cube

Is Run-time Verification useful for adapting preventively? Determining the Degree of Impossible Adaptations

Situations during experimentation were observed:

– service invocations deviated from their stipulated response time, such that the

performance requirement is violated

– too late to apply SPADE as the performance requirement is already violated

determine the percentage of these situations:

relate the number of the service invocations which lead to these SLA

violations to the amount of all service invocations

© S-Cube

Is Run-time Verification useful for adapting preventively? Experimental Results:

SPADE has been applied to 5884 SBA instances (cf. row (a) in Table below).

629 of those SBA instances have been executed without any assumption violations

(b), in 5255 SBA instances assumptions have been violated (c), for those

instances, SPADE has identified 604 preventive adaptation triggers (d)

72 of the adaptation triggers were false positives (e) just 1.2% of the workflow

instances would have been unnecessarily adapted.

With respect to the challenging time constraint of 55 seconds, the very low

percentage of false positives can be considered as extremely promising

© S-Cube

Is Run-time Verification useful for adapting preventively? Amount of situations in which SPADE cannot adapt is very low as well

Row (c) a total of 825 out of 28624 service invocations, i.e. 2.5%, lead to

situations where an adaptation is not possible.

BUT: this could still mean a threat for applications which are required to

meet the requirements in nearly 100%, e.g. in an emergency scenario

© S-Cube

Learning Package Overview

Problem Description

Preventive Adaptation based on Runtime-Verification

Discussion

Conclusions

© S-Cube

Conclusion

SPADE is an automated technique to determine adaptation

needs to trigger preventive adaptations of SBAs

To achieve this high degree of automation SPADE exploits

runtime verification techniques

The experiments reveal situations in which SPADE cannot

perform a preventive adaptation, that again: SPADE was able

to avoid SLA violations by a high percentage

This indicates the need for further improvement of

SPADE, but also generally demonstrates the effectiveness

of applying runtime verification techniques for preventive

adaptation

© S-Cube

References

© S-Cube

[MetzgerEtAl2010] Metzger, A.; Schmieders, E.; Cappiello, C.; Di Nitto, E.; Kazhamiakin, R.; Pernici, B. & Pistore, M.: Towards Proactive Adaptation: A Journey along the S-Cube Service Life-Cycle. MESOA: 4th International Workshop on Maintenance and Evolution of Service-Oriented Systems, 2010

[SchmiedersEtAl2011] Schmieders, E. & Metzger, A.: Preventing Performance Violations of Service Compositions using Assumption-based Run-time Verification. ServiceWave, 2011

[Gu2007] Q. Gu and P. Lago, A stakeholder-driven service life cycle model for SOA. In 2nd international Workshop on Service Oriented Software Engineering: in Conjunction with the 6th ESEC/FSE Joint Meeting (Dubrovnik, Croatia, September 03 - 03, 2007). IW-SOSWE '07. ACM, New York, NY, 1-7.

[BianculliEtAl2008] Bianculli, D.; Ghezzi, C.; Spoletini, P.; Baresi, L. & Guinea, S.: A Guided Tour through SAVVY-WS: A Methodology for Specifying and Validating Web Service Compositions, Springer-Verlag, 2008, 131-160

[GehlertEtAl2010] Gehlert, A.; Bucchiarone, A.; Kazhamiakin, R.; Metzger, A.; Pistore, M. & Pohl, K.: Exploiting assumption-based verification for the adaptation of service-based applications. SAC '10: Proceedings of the 2010 ACM Symposium on Applied Computing, ACM, 2010, 2430-2437

[ComuzziEtAl2009] Comuzzi, M., Pernici, B.: A framework for qos-based web service contracting. ACM Transactions on web 3(3) (2009)

[JRA-1.3.2] http://www.s-cube-network.eu/working-area/activities-and-workpackages/jra-1/WP-JRA-1.3/deliverables/cd-jra-1.3.2/CD-JRA-1.3.2_Quality%20Reference%20Model%20for%20SBA.pdf/view

Further S-Cube Reading

© S-Cube

Leitner, Wetzstein, Rosenberg, Michlmayr, Dustdar, and Leymann. Runtime Prediction of Service Level Agreement Violations for Composite Services. In Proceedings of the 2009 International conference on Service-Oriented Computing (ICSOC/ServiceWave'09), Springer-Verlag, Berlin, Heidelberg, 176-186.

Leitner, Michlmayr, Rosenberg, and Dustdar. Monitoring, Prediction and Prevention of SLA Violations in Composite Services. In Proceedings of the 2010 IEEE International Conference on Web Services (ICWS '10). IEEE Computer Society, Washington, DC, USA, 369-376.

Wetzstein, Leitner, Rosenberg, Brandic, Dustdar, and Leymann. Monitoring and Analyzing Influential Factors of Business Process Performance. In Proceedings of the 13th IEEE international conference on Enterprise Distributed Object Computing (EDOC'09). IEEE Press, Piscataway, NJ, USA, 118-127.

Acknowledgements

The research leading to these results has

received funding from the European

Community’s Seventh Framework

Programme [FP7/2007-2013] under grant

agreement 215483 (S-Cube).

© S-Cube