An Integrated Self-Testing Framework for Autonomic Computing...

An Integrated Self-Testing Framework forAutonomic Computing Systems

Tariq M. King, Alain E. Ramirez, Rodolfo Cruz, and Peter J. ClarkeSchool of Computing and Information Sciences, Florida International University, Miami, USA

Email: {tking003, aeste010, rcruz002, clarkep}@cis.fiu.edu

Abstract— As the technologies of autonomic computing be-come more prevalent, it is essential to develop methodolo-gies for testing their dynamic self-management operations.Self-management features in autonomic systems inducestructural and behavioral changes to the system duringits execution, which need to be validated to avoid costlysystem failures. The high level of automation in autonomicsystems also means that human errors such as incorrectgoal specification could yield potentially disastrous effectson the components being managed; further emphasizingthe need for runtime testing. In this paper we propose aself-testing framework for autonomic computing systemsto dynamically validate change requests. Our frameworkextends the current architecture of autonomic systems toinclude self-testing as an implicit characteristic, regardlessof the self-management features being implemented. Wevalidate our framework by creating a prototype of anautonomic system that incorporates the ability to self-test.

Index Terms— autonomic computing, testing, validation.

I. INTRODUCTION

Continuous technological advances have led to anunprecedented growth in the size and complexity ofcomputing systems within the last two decades [1]. Thehuman and economic implications of this growth arecompounded by the need for large-scale system integra-tion, in order to facilitate collaboration among multiplebusiness enterprises. Managing complex, large-scale, in-terconnected computing systems requires a team of highlyskilled IT professionals to provide ongoing technical sup-port services. Therefore, as technology-driven businessescontinue to expand, they incur increasingly high coststo maintain their computing infrastructure. In addition,some computing infrastructures are now so complex that ithas become almost impossible to manage them manually.Major industrial players such as IBM have thereforerecognized the need to shift the burden of support taskssuch as configuration, maintenance and fault managementfrom people to technology [2].

Autonomic computing is a movement towards self-management computing technology that aims to reducethe difficulties faced by administrators when managingcomplex systems. The paradigm emphasizes computing

This paper is based on “Towards Self-Testing in Autonomic Comput-ing Systems,” by T.M. King, D. Babich, J. Alava, R. Stevens, and P.J.Clarke, which appeared in the Proceedings of the 8th IEEE InternationalSymposium on Autonomous Decentralized Systems (ISADS), Sedona,USA, March 2007. c© 2007 IEEE.

This work was supported in part by the National Science Foundationunder grants IIS-0552555 and HRD-0317692.

systems that automatically configure, optimize, heal, andprotect themselves [3], [4], in accordance with adminis-trator objectives. IBM has successfully attracted membersfrom both the IT industry and the academic communityto the area of autonomic computing through variousmanifestos, research papers, and technical reports [1]–[5].The initiative continues to stimulate great interest in thescientific community, and has led to the development ofnumerous projects based on autonomic computing [6].

Although research is advancing in many areas of auto-nomic computing, there is a lack of development in thearea of testing autonomic systems at runtime. Runtimetesting is necessary in autonomic systems because self-management features induce structural and behavioralchanges to the system during its execution. Furthermore,the high level of automation in autonomic systems meansthat incorrect goal specification could yield potentiallydisastrous, and even irreversible effects on the compo-nents being managed. This research seeks to address twofundamental questions associated with runtime changes inautonomic computing systems. These questions are: (1)After a change, how can we be sure that new errors havenot been introduced into previously tested components?and (2) If a change introduces a new component, howcan we be sure that the component will actually behaveas intended?

In this paper we propose a self-testing framework forautonomic computing systems that dynamically validateschange requests through regression testing, and the ex-ecution of additional tests which exercise the behaviorof newly added components. Our framework integratestesting into the current workflow of autonomic managersthrough communications to new autonomic managersdesigned for testing. We apply concepts of autonomicmanagers, knowledge sources, and manual managementfacilities [3] to testing activities for autonomic systems.Our testing methodology is based on two general vali-dation strategies – safe adaptation with validation, andreplication with validation.

We extend the previous work by King et. al [7] toprovide a more comprehensive self-testing frameworkwith additional components that enhance test coordinationand facilitate manual test management. We describe theduties of these new components and present examples oftheir use within the integrated framework. In addition, weprovide descriptions of the interactions between these newcomponents and the existing framework components, and

JOURNAL OF COMPUTERS, VOL. 2, NO. 9, NOVEMBER 2007 37

© 2007 ACADEMY PUBLISHER

discuss the role played by the human administrator whenmanaging the testing process. We also present an extendedimplementation of a prototype developed to show thefeasibility of our testing methodology.

This paper is organized as follows: the next sectiondescribes the autonomic computing paradigm and relatedsoftware testing approaches. Section III presents the archi-tectural model of the self-testing framework, and outlinesthe steps of our testing methodology. Section IV discussesthe component interactions of the test managers of theframework. Section V presents the implementation detailsof the prototype. Section VI presents the related work, andin Section VII we give concluding remarks.

II. AUTONOMIC COMPUTING AND TESTING

In this section we provide a brief overview of the auto-nomic computing paradigm, and describe IBM’s architec-tural blueprint for building autonomic systems. We alsosummarize the main approaches to software testing, andoutline the major steps and challenges of test automation.Techniques to support runtime testing of evolving andadaptive systems are also discussed in this section.

A. Overview of Autonomic Computing

Autonomic computing (AC) is IBM’s proposed solutionto the problems associated with the increasing complexityof computing systems, and the evolving nature of soft-ware. The AC initiative was launched in October 2001and portrayed a vision of computing systems [4] thatmanage themselves according to high-level objectives.The paradigm seeks to alleviate the burden of integratingand managing highly complex systems through increasedautomation and goal specification.

The term autonomic is derived from the human au-tonomic nervous system, which regulates vital bodilyfunctions such as homeostasis without the need for con-scious human involvement [1]. AC extends this biologicalconcept to computing systems by embedding additionalinfrastructure within the system to carry out low-leveldecisions and tasks, while administrators specify overallsystem behavior as high-level policies. The core capabil-ities that support self-management in autonomic comput-ing systems [4], [6] include:

Self-configuration. Automatically configuring or re-configuring existing system components, and seamlesslyintegrating new components.

Self-optimization. Automatically tuning resources andbalancing workloads to improve operational efficiency.

Self-healing. Proactively discovering, diagnosing and re-pairing problems resulting from failures in hardware orsoftware.

Self-protection. Proactively safeguarding the systemagainst malicious attacks, and preventing damage fromuncorrected cascading failures.

Manual Manager

Orchestrating AMs

Touchpoint AMs

Touchpoints

Managed Resources

Know

ledg

e So

urce

s

Figure 1. Layered architecture of Autonomic Computing.

In order to accomplish the aforementioned self-management tasks, autonomic systems must be able toobserve their own structure and behavior (introspection);adapt or modify their own structure and behavior (in-tercession) [8]; and be aware of their environmentaland operational contexts. Self-awareness is therefore aninherent characteristic of AC systems regardless of thespecific autonomic capabilities being implemented.

B. Architecture for Autonomic Computing

The architectural blueprint for AC [3] defines a com-mon layered approach for developing self-managing sys-tems as shown in Figure 1. The horizontal layers includemanaged resources, touchpoints, touchpoint autonomicmanagers, orchestrating autonomic managers, and a man-ual manager. A vertical layer of knowledge sources (top-left of Figure 1) spans the top three horizontal layers tofacilitate the exchange, retrieval, and archival of manage-ment information among these layers.

The managed resource layer consists of the hardware orsoftware entities for which self-management services arebeing provided. Directly above the managed resources aremanageability interfaces called touchpoints. The touch-points implement the sensor and effector behaviors [3],[4] necessary to automate low-level management tasks.Sensors provide introspection mechanisms for gatheringdetails on the current structure or behavior of managed re-sources, while effectors provide intercession mechanismsto facilitate structural or behavioral modifications.

A higher level of management is provided by auto-nomic managers (AMs). There are two categories of AMs,namely Touchpoint AMs and Orchestrating AMs [3].Touchpoint AMs work directly with managed resourcesthrough their touchpoint interfaces. Orchestrating AMsmanage pools of resources or optimize the TouchpointAMs for individual resources. The topmost layer is animplementation of a management console, called themanual manager, which facilitates the human adminis-trator activity. The vertical layer of knowledge sourcesimplements registries or repositories that can be usedto extend the capabilities of AMs, and may be directlyaccessed via the manual manager.

Autonomic systems are characterized by intelligentclosed loops of control, in which sensed changes to man-aged resources result in the invocation of the appropriateset of actions required to maintain some desired systemstate. In autonomic systems, these closed control loops are

38 JOURNAL OF COMPUTERS, VOL. 2, NO. 9, NOVEMBER 2007


Monitor

Analyze Plan

Execute

Managed Resource

Knowledge

Sensor Effector

Sensor Effector

Updated Policy(from Knowledge Source)

Symptom

Change Request

Change Plan

Autonomic Manager

Figure 2. MAPE structure of autonomic managers.

typically implemented as the monitor, analyze, plan, andexecute (MAPE) functions of AMs as shown in Figure 2.The monitor collects state information from the managedresource and correlates them into symptoms for analysis.If analysis determines that a change is needed, a changerequest is generated and a change plan is then formulatedfor execution on the managed resource. AMs containa knowledge component which allows access to datashared by the MAPE functions. This built-in knowledgecomponent generally contains information relating to self-management policies, and interacts with the knowledgesources shown in Figure 1 to provide an extensible self-management framework. For example, as shown in Figure2, extensibility in autonomic systems can be achieved byloading a new policy from a knowledge source into thebuilt-in knowledge of the AM to update its capabilities.

C. Software Testing Approaches

The two main categories of software testingare specification-based and program-based testing.Specification-based testing is based soley on thedefinition of what the program under test is supposedto do and employs no internal information about thestructure of the program [9]. The specification drivesthe testing process by providing a means of checkingthe correctness of output (test oracles); selecting testdata; and measuring test adequacy. On the other hand,program-based testing defines testing requirementsstrictly in terms of whether or not the program undertest has been thoroughly exercised [9]. This categoryof testing is usually associated with some form oftest adequacy criteria that is based on coverage ofparticular elements of the program, e.g., all statements,all branches, or all paths [10].

Understanding the differences between specification-based and program-based testing allows the softwaretester to select appropriate testing techniques under givencircumstances. Although both approaches are consideredto be equally important, the tester must be able to identifywhat kind of testing is required so that valuable time and

resurces are not wasted performing too much or too littletesting on different aspects of the software.

The notion of test data adequacy is central to any testingmethodology because it can be viewed as a measurementof test quality, and as a guideline for the generation oftest cases. If P is a set of programs, and S is a set ofspecifications, and T is a set of test cases, we can formallydefine a test data adequacy criterion C as follows [10]:

Measurements. C : P × S × T → [0, 1]. C(p, s, t) mapsto a real number representing the degree of test adequacy;and the greater the value, the more adequate the testing.

Generators. C : P × S → 2T . C(p, s) maps to a powerset containing all test cases that satisfy the criterion, andhence any element of this set is adequate for testing.

Test data adequacy criteria can also be viewed as astopping rule that determines whether or not enoughtesting has been done [10]. However, as a stopping rule,a test data adequacy criterion C is a special case ofmeasurements with the range {0, 1}, i.e., false or true.

The effectiveness of a test set with respect to itsability to reveal faults can be assessed using a techniqueknown as mutation testing [9], [10]. Mutation testinginvolves generating a set of programs or specifications,called mutants, that differ from the original program orspecification in some way. The test set is then executedusing the mutants and the results are compared with thoseproduced from using the original program or specification.For each mutant, if the test results differ from the originalresults on at least one test case, the mutant is said to havebeen killed; otherwise the mutant is still alive [10]. Amutation score can then be used to measure test adequacyby calculating the ratio of the number of dead mutantsover the total number of mutants that are not equivalentto the original program or specification.

D. Test Automation

Test automation calls for careful planning during thesoftware development process, and is impossible withouttool support. Automating the testing process involvesdesigning test cases; creating test scripts; and setting upa test harness for automatically executing tests, loggingresults, and evaluating test logs [11]. If the post-testevaluation passes then the test harness should automat-ically terminate, otherwise additional test cases shouldbe selected and fed through the harness with the aim ofimproving the testing effort. There are several categoriesof tools that support test automation. These include testdesign tools, dynamic analysis tools, GUI test drivers,capture/playback mechanisms, and test evaluation tools[11].

Most testing tool vendors boast that cost savings andreduction in defects are some of the benefits to begained from using their products [12]. However, testautomation still presents several challenges to the testengineers. For example, the capture/playback method of



automation which is prevalent in many vendor packagesdoes not adequately address the needs of complex, multi-environment systems [12]; thereby leaving test engineersto manually implement supporting programs to achievethe desired level of automation. In addition, some testingtools only support a specific granularity of testing suchas unit, integration, or system testing. Successful test au-tomation therefore requires the selection of an appropriatecombination of testing tools, and a skilled test automationteam to develop support programs and perform traditionaltesting tasks.

E. Testing Evolving and Adaptive Systems

As previously mentioned in Section II-A, autonomiccomputing addresses the software complexity and evo-lution problems through the use of a highly adaptiveself-management infrastructure. Software evolution is in-evitable due to the emergence of new requirements,changes in the operating environment, and the need tofix newly discovered errors. As a result, software sys-tems typically require perfective, adaptive, or correctivemaintenance [13] after initial deployment. Regressiontesting determines whether modifications to software dueto maintenance have introduced new errors into previouslytested code [9]. This may involve re-running the entiretest suite (retest-all), or selecting a subset of the initialtest suite for execution (selective regression testing) [14].Techniques for regression test selection include dataflow,random, safe and test minimization [14].

Testing dynamically adaptive systems is extremelychallenging because both the structure and behavior ofthe system may change during its execution. Existing testcases may no longer be applicable due to changes inprogram structure, thereby requiring the generation of newtest cases. Modern programming languages such as Javaand C# support introspection of the structure of an objectat runtime, and hence provide mechanisms for extractinguseful information for dynamic test case generation.

A disciplined approach to adaptation can also be usedto support runtime testing of adaptive systems. Safe adap-tation ensures that the integrity of system components ismaintained during adaptation [8]. The safe adaptation pro-cess is comprised of the three phases: analysis, detectionand setup, and realization. During analysis, developersprepare a data structure for holding information such ascomponent configurations, dependency relationships, andadaptive actions. The detection and setup phase occurs atruntime and involves generating safe adaption paths forperforming adaptive actions on system components. Theactual adaptation is then performed during the realizationphase in the following steps:

Step 1. Move the system into a partial operation modein which some functionalities of the component(s) to beadapted are disabled.

Step 2. Hold the system in a safe state while adaptiveactions are performed.

Step 3. Resume the system’s partial operation once alladaptive actions are complete.

Step 4. Perform a local-post action to return the systemto a fully-operational running state.

The safe adaptation process will not violate any depen-dency relationships or interrupt critical communicationbetween components, even in the presence of failures[8]. If a failure occurs at any point before Step 3, theadaptation manager can retry the failed actions or rollbackto the source configuration. Rollbacks are not allowedafter any system process has been resumed. Therefore,safe adaptation is atomic in the sense that either no sideeffects are produced before a rollback or the adaptationprocess runs to completion.

III. AUTONOMIC SELF-TESTING FRAMEWORK

The adaptive and evolving nature of AC systems re-quires that testing be an integral part of these systems.We affirm that, similar to the inherent self-awarenesscharacteristic of AC systems discussed in Section II-A,self-testing is an implicit characteristic of autonomic sys-tems regardless of the specific self-management featuresbeing implemented. Self-testing in autonomic systemsis necessary for dynamically: (1) performing regressiontesting after changes have been implemented to ensurethat new errors have not been introduced into previouslytested components; and (2) validating the actual behaviorof newly added or adapted components prior to their usein the system.

We propose an integrated self-testing framework forautonomic computing systems to perform the aforemen-tioned tasks, thereby addressing the research questionsposed in Section I. Our self-testing framework augmentsthe dynamic high-level test model for autonomic com-puting systems in [7] with components that enhance testcoordination, and provide detailed descriptions of their in-teractions. In order to be consistent with the grand visionof autonomic computing, our self-testing framework fullyautomates testing activities wherever possible, and seeksto minimize the administrator’s effort in those testingactivities which require manual intervention.

As shown in Figure 3, our approach incorporates testingactivities into autonomic computing systems by providingvalidation services to autonomic managers (AMs) and themanual manager implementation via test interfaces. Thelefthand portion of Figure 3 shows an autonomic com-puting system and the righthand portion shows the self-testing framework. The self-testing framework consists oftest managers, test knowledge sources and auxiliary testservices that collaborate to provide validation services forthe autonomic system.

Our testing methodology is based on two general strate-gies – safe adaptation with validation and replication withvalidation, which differ with respect to system overheadcost and feasibility of use. Depending on the strategyused, the autonomic system may be required to maintain



Orchestrating Autonomic Manager

Touchpoint Autonomic Manager

2. Change Request

Detected

3a. Use SelectedValidation Strategy

3b. Setup Validation

1. Ready for Self‐

Management

4. Implement Change

7. Log Test Results

3c. Load Validation Policy

8. Result of Post‐

Test Evaluation

9. Validation Response

Autonomic Computing System Self‐Testing Framework

Orchestrating Test Manager

Touchpoint Test Manager

5. Ready for Self‐Test

6. Execute Test Cases

Manual Manager (Console) Test Knowledge Sources

Know

ledg

e So

urce

s

Auxiliary Test Services

ManagedResource

Copy of ManagedResource

Figure 3. High-level architectural model for an integrated self-testing framework for autonomic computing systems.

copies of managed resources for testing purposes as illus-trated by the dashed box (bottom-left of Figure 3). Theself-testing framework also assumes that the autonomicsystem provides mechanisms for implementing changerequests on the copy of the managed resource; transferringthe contents of the copy to the actual managed resourceafter validation; and performing safe adaptation [15].

In this section we present the proposed autonomic self-testing framework and describe its major components.Subsections III-B and III-C provide the detailed stepsof our testing methodology, in the context of the twovalidation strategies, through descriptions of the workflowof Figure 3. We also provide the rationale and guidelinesfor applying each validation strategy, and discuss howwe plan to address some of the key challenges of testingautonomic systems at runtime.

A. Test Managers

Test managers (TMs) extend the concept of auto-nomic managers to testing activities. Like autonomicmanagers, TMs may also be Orchestrating or Touchpoint.Orchestrating TMs coordinate high-level testing activitiesand manage Touchpoint TMs, while Touchpoint TMsperform low-level testing tasks on managed resources.During system execution, Orchestrating AMs will notifyOrchestrating TMs that some change request requiresvalidation. Orchestrating TMs will then coordinate thetesting activities necessary to validate that change requestusing the managed resource or a copy. TMs will beresponsible for:

Test coordination. Directing the end-to-end validationprocess by interfacing with AMs; checking for new val-idation policies in test knowledge sources; coordinatingthe tasks of Touchpoint TMs; and invoking auxiliary testservices.

Test planning and execution. Scheduling and performingregression testing and, if necessary, executing newly gen-

erated test cases to validate change requests for managedresources.

Test suite management. Generating new test cases dy-namically, and discarding test cases that may no longerbe applicable due to changes in system structure.

Pre- and post-test setup. Setting up the test environment,and preparing a test log to be used during the post-testevaluation.

Post-test evaluation. Analyzing and evaluating test re-sults and test coverage provided in the test log againstthe high-level validation policy.

Storage of test artifacts. Maintaining a repository tostore test cases, test logs, test histories, and validationpolicies.

To perform the aforementioned duties, TMs will con-tain components that are consistent with the MAPEstructure [3] of autonomic managers. A test monitor willbe responsible for polling the managed resource, and/orvarious components within the self-testing framework, tocollect any information relevant to the testing process. Atest analyzer will then determine whether or not sometesting-related activity needs to be performed such as set-ting up the test environment, generating new test cases, orconducting a post-test evaluation. A test planner will thengenerate a plan for the testing activity, and a test executercomponent will perform the required testing tasks. A testknowledge component will serve as a central repositoryfor test artifacts, and coordinate the interactions betweenthe aforementioned components. Detailed descriptions ofthe component interactions within Orchestrating TMs andTouchpoint TMs are provided in Subsections IV-A andIV-B, respectively.

B. Safe Adaptation with Validation

The safe adaptation with validation strategy validateschanges resulting from self-management as part of asafe adaptation process [8], and occurs directly on the



managed resource during system execution. The steps ofthis approach as corresponds to the workflow of Figure 3,starting from its bottom lefthand portion, are as follows:

1. Ready for self-management. A Touchpoint AM gath-ers information from a managed resource and correlatesit into a symptom that warrants self-management.

2. Change request detected. A change request is gener-ated by the Touchpoint AM and this event is detected viathe sensor of an Orchestrating AM.

3. Safe adaptation with validation. The OrchestratingAM initiates safe adapation of the managed resource(3a) and concurrently requests that an Orchestrating TMset up validation (3b). An appropriate validation policy(3c) is then loaded into the test knowledge componentof a Touchpoint TM.

4. Implement change plan on managed resource. TheTouchpoint AM proceeds with safe adaptation until themanaged resource is fully-adapted (i.e., up to Step 2 of thesafe adaptation process outlined in Subsection II-E) butkeeps the resource blocked until validation is performed.

5. Ready for self-test. The Touchpoint TM detects thatthe resource is in a fully-adapted safe state, and istherefore ready to be tested.

6. Execute test cases. After analysis and planning, theTouchpoint TM executes test cases on the resource.

7. Log test results. The test results and test coverageare coalesced into a log file, which is analyzed by theTouchpoint TM with respect to the validation policy.

8. Result of post-test evaluation. Message indicatingwhether validation passed or failed, or if there wasinadequate test coverage, is sent to the Orchestrating TM.

9. Validation response. Orchestrating AM is notified asto whether it should accept or reject the change request.If the change request is accepted, the Touchpoint AMimplements the change on the actual managed resource;otherwise the change request is discarded.

The safe adaptation with validation strategy should beused if it is too expensive, impractical, or impossible toduplicate managed resources. Recall that in autonomicsystems, managed resources may be either hardware orsoftware entities. From a business perspective, it maynot be economically viable to maintain duplicates ofsome hardware or software components soley for testingpurposes. In addition, instantiating and executing copiesof software components may affect system performancenegatively and hence may not be practically feasible.Safe adaptation with validation addresses these issues byallowing testing to be performed directly on managedresources. However, this approach has the disadvantagethat some application services may have to be temporarilysuspended while the validation process completes.

C. Replication with Validation

The replication with validation strategy requires thesystem to create and/or maintain copies of the managedresource for validation purposes (recall dashed box atbottom-left of Figure 3). Change requests for managedresources are first implemented on copies and validatedprior to execution on the actual managed resources. Usingthis strategy, the steps of our testing methodology ascorresponds to Figure 3 differ as follows:

3. Replication with validation. The Orchestrating AMinitiates the replication with validation strategy (3a.)by ensuring that the managed resource and the copy areidentical with respect to their structure and behavior. Arequest is sent to the Orchestrating TM to set up thevalidation process (3b.), which loads the validationpolicy (3c.) into a Touchpoint TM.

4. Implement change plan on copy. The Touchpoint AMimplements the self-management change plan on a readilyavailable copy of the managed resource. This event wouldbe a signal to the Touchpoint TM that self-testing can nowbe performed using the changed copy.

The replication with validation strategy should be usedwhen managed resources can be easily replicated. It hasthe advantage that the regular service of the system doesnot have to be suspended for the entire time required toperform testing. Furthermore, if the component under testis hot swappable, the system may be able to provide unin-terrupted service in the presence of validation. Replicationwith validation also allows testing services to be deployedon a separate node, thereby removing any computationaland storage overhead from the autonomic system.

D. Auxiliary Test Services and Test Knowledge Sources

High-level test coordination within the self-testingframework is supported by auxiliary test services (ATS)and test knowledge sources, shown at the top-right ofFigure 3. The ATS component provides OrchestratingTMs with access to external or third-party automated testsupport tools; and implements facilities for manual testmanagement. Orchestrating TMs interact with the ATS toconfigure test support tools such as code coverage pro-filers, performance analyzers, and automated test drivers.The ATS component supplements the services offered bythese support tools and tailors them to the specific testingneeds of the autonomic system.

The manual manager implementation of the autonomicsystem interfaces with the ATS to provide administratorswith direct access to artifacts stored in test knowledgesources. The ATS will enable administrative functionssuch as updating validation policies, viewing test logs,managing test cases, and collating test histories throughthe management console of the autonomic computingsystem. In addition, administrators will be able to choosefor certain testing tasks to require human intervention. Forexample, upon determining that testing is required for a



particular managed resource, the Orchestrating TM couldsend a message to the management console requestingthe administrator to manually formulate the test plan forthis resource. Other interactive administration servicesprovided through the ATS may include defect trackingand test scenario walkthroughs.

E. Addressing Challenges

To address the many challenges of testing autonomicsystems at runtime, we plan to use the proposed self-testing framework as a focal point for the investigationof various research directions that support quality assur-ance in these systems. One such research direction isthe use of formal specifications to dynamically generatetest sequences, test oracles, and test data for systemcomponents [16]. Embedding formal specifications withinautonomic system components would provide a means forextracting precise descriptions of useful test informationsuch as preconditions, postconditions and invariants. Fur-thermore, executable formal specifications facilitate thedevelopment of visualization systems [17], which hasbeen identified as one of the major research challengesof autonomic computing [5]. Such visualization systemscould be used to interactively guide the administratorthrough test scenarios and during system debugging.

Regarding the problem of testing autonomic compo-nents in the presence of unforseen circumstances, mech-anisms for policy-based risk analysis and trust [18] couldbe incorporated into the self-testing framework. Testingrequirements could then be based on measurements of theestimated risk of interactions with unknown entities, orunder unexpected environmental conditions. Furthermore,a risk-based strategy to regression test selection [19]could be adopted to identify test cases which would beappropriate for testing high-risk components.

IV. TEST MANAGERS: COMPONENT INTERACTIONS

Similar to the components in autonomic managers, theMAPE components in TMs implement intelligent closedloops of control. However, the intelligent control loops inTMs focus on self-testing rather than self-management.In this section, we provide descriptions of the interactionsbetween the components of our self-testing framework inthe context of these intelligent control loops. Componentinteractions are described at two levels of granularity. Atthe higher level, we discuss how the internal componentsof Orchestrating TMs facilitate the coordination of test-ing activities among multiple framework components. Atthe lower level, we present step-by-step details on howthe internal components of Touchpoint TMs interact toaccomplish low-level testing tasks on managed resources.A high-level algorithm for validating change requests isalso provided in this section.

A. Interactions within Orchestrating TMs

Figure 4 illustrates how the intelligent control loops ofOrchestrating TMs implement high-level test coordination

OrchestratingAM

Sensor

OTM1(a)

Effector

TouchpointTM

Test KnowledgeSouces

OTM1(b)

OTM3

Auxilliary Test Services

TestMonitor

TestPlanner

TestExecuter

Orchestrating Test Manager

OTM2

OTM6

OTM5

Test Knowledge

OTM4

TestAnalyzer

Figure 4. Intelligent closed loops of control in Orchestrating TMs.

activities. Test coordination involves complex interactionsbetween Orchestrating TMs and multiple components ofthe integrated framework as shown at the bottom of Figure4. Orchestrating TMs can invoke the sensor and effec-tor mechanisms of Orchestrating AMs, test knowledgesources, Touchpoint TMs, and auxiliary test services.Information gathered from these framework componentsis then passed through the MAPE functions of the Or-chestrating TM to achieve various testing objectives; asindicated by the lines labeled OTM that enter and exitthrough the sensors and effectors in Figure 4.

Orchestrating TMs will continuously poll the sensorsof Orchestrating AMs (OTM 1a) to detect when theautonomic system requires validation services. The testknowledge sources component may also be monitored(OTM 1b) to detect whenever a human administratormanually updates a validation policy. Either of theseevents can be handled by an intelligent control loop whichuploads the appropriate validation policy into TouchpointTMs (OTM 2) to instantiate the new or requested testingfunctionality. During testing, Touchpoint TMs may bemonitored to detect when they require support tools suchas code coverage profilers to be configured (OTM 3).Configuration of the support tools could then be initiatedby invoking the effector of the auxiliary test servicescomponent (OTM 4). Once the support tool configurationcompletes, this event would be detected (OTM 5) by theOrchestrating TM so that a notification message could besent to the Touchpoint TM (OTM 6).

Other orchestrating interactions, not shown in Figure 4,include receiving notifications from Touchpoint TMs asto whether or not validation passed (OTM 7a) or if therewas inadequate test coverage (OTM 7b); and respondingto those notifications by either terminating the testingprocess (OTM 8a) or continuing testing (OTM 8b) in anattempt to improve test coverage.

B. Interactions within Touchpoint TMs

Figures 5(a) and 5(b) show two intelligent loops ofcontrol within Touchpoint TMs. The first loop traces the



TestMonitor

Test TestPlanner

TestExecuter

Managed Resource

Test Knowledge

Sensor Effector

Sensor Effector

Analyzer

OTM 2:Upload Validation Policy

OTM 3: Test Plan Detected,

OTM 6:Profiler Setup Complete

1.3.21.0

1.2

1.3

1.4

1.3.1

Setup Profiler

TouchpointTest Manager

TestMonitor

Test TestPlanner

TestExecuter

Managed Resource

Test Knowledge

Sensor Effector

Sensor Effector

Analyzer

OTM 8a:Completed(Terminated)

OTM 7a: Passed(Failed)

OTM 8b:ContinueTesting

2.5

2.2

2.3

2.4

2.1

2.2.1(a)

OTM 7b: Inadequate Test Coverage

2.2.1(b)

2.2.2(a)

2.2.2(b)

TouchpointTest Manager

1.51.1

Figure 5. Component interactions through two intelligent closed control loops (a) and (b) in Touchpoint TMs.

arc labeled with artifacts 1.1 through 1.5 in Figure 5(a),and the second loop traces the arc labeled with artifacts2.1 through 2.5 in Figure 5(b). As previously stated,the operations of Touchpoint TMs may be coordinatedby an Orchestrating TM; as indicated by the lines labeledwith the prefix OTM which enter and exit through the topsensors and effectors, respectively. Figure 6 shows a high-level algorithm which corresponds to how the intelligentcontrol loops of Touchpoint TMs and Orchestrating TMswork together to validate changes to managed resources.The discussion for the rest of this subsection providesdetails for the relationships between Figures 4, 5, and 6.

The Orchestrating TM first sends a validation policyfor the managed resource to a Touchpoint TM, whichloads it into the test knowledge component (1.0). TheTouchpoint TM automatically invokes the test monitor toretrieve the new structure of the managed resource (1.1)when it senses that the resource is ready to be validated.The test monitor then sends the information about thenew structure to the test analyzer (1.2), which uses itin conjuction with the previous structure and a baselinetest suite to prepare a new test suite. The test analyzernotifies the test planner that a new test suite is ready andrequests that a test plan be created (1.3). At this point,the Orchestrating TM detects that a request for a new testplan (1.3.1) has occurred and intercepts the closed loopto set up a code coverage profiling tool. Once the codeprofiler has been set to instrument the managed resourcefor test coverage, the test plan is finalized and control isreturned to the closed loop (1.3.2). The test plan is thenpassed to the test executer component (1.4), which thenstarts running test cases on the managed resource (1.5)and initiates the second closed control loop.

The second intelligent control loop commences with theretrieval of the test results and test coverage information(2.1) from the managed resource. The test monitorcorrelates this information into a test log and passes it

(OTM 2)

// External Monitored Event

(OTM 3)

// Corresponding Actuated Event

(OTM 6)

// External Monitored Events

(OTM 7a)

(OTM 7b)

// Corresponding Actuated Events

(OTM 8a)(OTM 8a)

Figure 6. Change request validation algorithm for managed resources.

to the test analyzer (2.2). The test analyzer evaluatesthe test log against the validation policy and determineswhether validation passed or failed (2.2.1a), or if thetest suite was inadequate (2.2.1b) with respect to thetest coverage criteria. The results of the evaluation are sentto the Orchestrating TM, which determines whether to endthe validation process (2.2.2a), or continue testing afterre-analyzing the current test suite (2.2.2b). If a decisionis made to re-analyze the test suite, then the behavior ofthe second loop from 2.3 to 2.5 in Figure 5(b) is thesame as the first loop from 1.3 to 1.5 in Figure 5(a).



V. PROTOTYPE: AN AUTONOMIC CONTAINER

Due to the lack of freely available autonomic systems,we were unable to locate an application that wouldallow the seamless integration of a self-testing frame-work. Therefore, to show the feasibility of our approach,we developed a prototype of an application with self-management capabilities. Our prototype is based on theconcept of an autonomic container [20], which is a datastructure with self-configuration and self-testing capabil-ities. However, we extend the work presented in [20] byimplementing the container as a distributed system withthe added feature of self-protection. The current versionof the autonomic container therefore provides data storageservices to remote users, while performing dynamic self-configuration, self-protection, and self-testing.

The prototype was developed in Java using the EclipseSDK, along with the necessary plugins and librariesfor the tools to support testing. Two tools were usedto support the testing effort: JUnit, a Java unit testingtool from the xUnit family of testing frameworks [21];and Cobertura, a Java code coverage analysis tool thatcalculates the percentage of source code exercised by unittests [22]. In this section we describe the main features ofthe autonomic container and present its top-level design.We also provide implementation details on the self-testingsubsystem, and describe our test procedures and experi-ment setup. We then discuss the findings of this work inthe context of the uses and limitations of the prototype.

A. Main Features and Top-Level Design

The managed element of the autonomic container pro-vides data storage services to general users through itspublic remote interface, while self-management featuresare only available to autonomic managers via its touch-point interface. The underlying data structure for theprototype is implemented as a stack, and the followingservices are defined in its remote interface: login,logout, push, pop, isFull, and isEmpty.

The self-configuration and self-protection features ofthe prototype dynamically execute changes on the stackbased on predefined symptoms. Self-configuration occurswhen the stack reaches or exceeds 80% of its full capacity,and involves increasing the stack capacity to anticipatedepletion of available storage space. The self-protectionfeature of the prototype safeguards the stack from failuresthat may be caused by repeated overflow or underflowconditions. Self-protection occurs when the number ofstack exceptions thrown by a particular user during asession exceeds 3, and results in the associated useraccount being disabled.

The design of the prototype is based on the replicationwith validation strategy, and therefore a copy of the stackis maintained by the autonomic system exclusively fortesting purposes. The top-level package for the self-testingautonomic container, labeled edu.fiu.strg.STAC inFigure 7, contains all the nested packages and librariesused to develop the prototype. The four main packagesare as follows: (1) mResources – provides access to

edu.fiu.strg.STAC

.aManagers

<<subsystem>>selfConfig

<<subsystem>>selfProtect

.tManagers

<<subsystem>>replicaTest

.mResources

<<subsystem>>container

<<library>>java.rmi.*

<<subsystem>>copyContainer

.auxTest

<<library>>junit.*

<<library>>cobertura.*

Figure 7. Top-level design of self-testing autonomic container.

the actual stack resource being managed and its copy; (2)aManagers – implements autonomic managers that per-form dynamic self-configuration and self-protection of thestack; (3) tManagers – implements test managers thatvalidate self-configuration and self-protection changesusing the stack copy; and (4) auxTest – provides aninterface to a test support tool that analyzes program codefor test coverage. The packages described in (1) and (2)represent the autonomic system, while those described in(3) and (4) comprise the self-testing framework.

The aforementioned packages are composed of subsys-tems which provide the core functionality of the proto-type. Within the mResources package, access to thesubsystems for the remote stack container, and itsreplica copyContainer, is achieved by using the JavaRemote Method Invocation [23] system (indicated by thedependencies on java.rmi in Figure 7). The packageaManagers contains two subsystems, selfConfigand selfProtect, which implement intelligent closedloops of control for self-configuration and self-protectionrespectively. These two subsystems are responsible forexecuting change plans on the copy of the stack priorto validation, and on the actual stack if validation issuccessful. The package tManagers consists of a singlesubsystem replicaTest that coordinates the end-to-end testing activity, including the execution of parameter-ized test cases developed using the junit library. Lastly,the auxTest package contains classes that implementcustomized test services for code profiling, and automatethe analysis of test coverage reports generated by thecobertura library.

B. Test Procedures and Setup Environment

To simulate the behavior of the architectural modelof the integrated framework depicted in Figure 3, theautonomic system and self-testing framework were im-plemented as separate threads within a remote serverapplication. Client programs were then developed to au-tomatically invoke the public interface of the autonomiccontainer, thereby emulating the actions of remote users.Both the server and client application programs wererun on the Eclipse 3.2 platform using the Java RuntimeEnvironment (JRE) 5.0 Update 12.

The client programs were designed to operate in amanner that would induce the self-management features



of the autonomic container. For example, a client wasdesigned to login to the server and push random integersonto the stack until it exceeded 80% of its full capacity,thereby triggering dynamic self-configuration of the stack.We then observed as the self-testing framework validatedthe self-configuration change request, and examined theresults produced during this pass of the simulation. Asimilar technique was used to setup a testing scenario forthe self-protection feature of the prototype.

The initial test suite for the autonomic container con-sisted of 24 test cases written in JUnit 3.8.1. The test caseswere developed using a combination of test strategiesincluding: boundary, random, and equivalence partitioning[10]. A parameterization technique was used in test casesthat addressed variable properties of the container such asstack capacity and user accounts. The validation policyfor the Touchpoint TM required 100% pass rate forall test cases executed, and at least 75% branch andstatement coverage. Cobertura 1.8 was used to evaluatetest coverage, and generate reports in extensible markuplanguage (XML) format.

C. Self-Test Subsystem Implementation

As previously mentioned the self-test subsystem ofthe prototype, replicaTest in Figure 7, containsmanagers that coordinate testing activities. These testmanagers, as well as the autonomic managers forself-configuration and self-protection, are based ona reusable manager design depicted by the packageedu.fiu.strg.ACSTF.manager in Figure 8. Thispackage is the first component developed for an Auto-nomic Computing Self-Testing Framework (ACSTF). Thedesign uses synchronized threads, remote method calls,generics, and reflection in Java [23] to implement theMAPE functions of the manager and the knowledgecomponent.

The knowledge repository of the manager, representedby the class KnowledgeData in Figure 8, realizes theinterface KnowledgeInterface that is used by theMAPE functions to access shared information. Queuesto hold newly generated change requests and changeplans to be executed have also been incorporated into theknowledge component. When a manager is instantiated,the KnowledgeCoordinator class loads a policyfile containing the following information: (1) the fullyqualified class name of the touchpoint object to be usedfor sensing and effecting; (2) the method name of thesensor function to be polled by the monitor; (3) a setof symptoms defined by relational mappings betweenthe variables of the touchpoint and desired values; and(4) a set of executable change plans represented bysequences of effector methods to be executed and theiractual parameters.

All high-level policies for the prototype are storedin XML 1.0 format, using the ISO-8859-1 encodingstandard. Each policy contains name and version attributeswhich distinguish it from other policies and facilitateautomation with respect to upgrades. A snippet of the

TouchData SymptomListExecutionQueue

RequestQueue

edu.fiu.strg.autonomic.manager

edu.fiu.strg.ACSTF.manager

+updateSD()+run()

Monitor

TouchData

+loadPolicy()

KnowledgeCoordinator

‐sensedData : TouchData‐symptoms : SymptomList‐requests : RequestQueue‐exPlans : ExecutionQueue

KnowledgeData

TouchData

+updateCRs()+run()

Analyzer

TouchData

+updateCRs()+updateCPs()+run()

Planner

TouchData

+updateCPs()+run()

Executer

TouchData

«interface»KnowledgeInterface

in

updates

invokes

updates

invokes

updates

invokes invokes

updates

Figure 8. Generic design of autonomic and test managers.

<?xml version=“1.0” encoding=“ISO-8859-1”?>

<policy name=“OTM Self-Test” version=“1.0”>

<monitor touchpackage=“edu.fiu.strg.STAC”touchclass = “OTMTouchData”sensormethod=“getCoordState”/>

<analyzer><symptom sid=“OTM ST 001”>

<mapping var=“requiresSC”op =“==”val=“true” />

</symptom></analyzer>

<planner><plan pid=“OTM ST 001”>

<action effector=“dequeueReqSC”/><action effector=“copyChangeSC”/><action effector=“setupTouchTM”/>

</plan></planner>

</policy>

Figure 9. Snippet of validation policy for the Orchestrating TM.

validation policy used by the Orchestrating TM of theprototype is shown in Figure 9. Policies are dividedinto three major sections represented by <monitor>,<analyzer>, and <planner> tags. Each section con-tains the portion of knowledge that will be primarilyused by the component corresponding to its tag name.For example, combining the values of the attributestouchpackage, touchclass, and sensormethodof the <monitor> tag in Figure 9, produces the fullyqualified method name of the sensor function used by themonitor of the Orchestrating TM. This method collectsstate information on the two autonomic managers, auxil-iary test services, and Touchpoint TM to coordinate test-ing. Polling is achieved by first passing the fully qualifiedclass name edu.fiu.strg.STAC.OTMTouchDatato the monitor as the template parameter TouchData,shown in Figure 8. The monitor then uses reflec-tion to instantiate the class and continuously invokesgetCoordState.

The object returned from the invocation of the sen-sor method is compared with one or more predefinedsymptoms, which were loaded from the <analyzer>tag of the policy. Figure 9 shows a test symptom with its



OTMSelfTest[from replicaTest] TTMSelfTest

[from replicaTest]

OAMSelfProtect[from selfProtect]

OAMSelfConfig[from selfConfig]

Stack[from copyContainer]

KnowledgeCoordinator[from ACSTF.manager]

Main[from cobertura]

StackTest[from replicaTest]

TestCase[from junit]

StackCover[from auxTest]

ProjectData[from cobertura]

Figure 10. Class diagram of self-test and associated components.

symptom identifier (sid) attribute set to “OTM ST 001”.This symptom consists of a single mapping betweenthe touchpoint state variable “requiresSC” and thevalue “true”, using the relational comparison operator“==”. The requiresSC variable is defined in termsof the state of the request queue of the OrchestratingAM for self-configuration, and evaluates true whenevera change request for self-configuration is pending. There-fore, if a self-configuration change request is generated,and since this is the only relational condition requiredby the symptom, the manager will recognize that thesymptom has been satisfied and locate an appropri-ate test plan. Note that a similar technique is usedfor specifying self-management symptoms. For example,the self-configuration policy of the prototype’s Touch-point TM contains a symptom defined by the relation“usedSpacePercent >= 80”. Our prototype designis highly extensible as it allows multiple variable map-pings to be defined for a single symptom, and multiplesymptoms to be defined for any manager.

Test symptoms are associated with test plans whichprovide remedies as sequences of low-level testing tasks.Test plans consist of a plan identifier (pid) attributethat is logically linked to a test symptom identifier,along with one or more actions. Figure 9 shows atest plan with pid “OTM ST 001”, which correspondsto the previously mentioned test symptom for detectingrequests for self-configuration. The actions of the testplan consist of the effector methods dequeueReqSC,copyChangeSC, and setupTouchTM, which dequeuethe self-configuration change request; implement thechange on the stack copy; and set up the Touchpoint TMfor change validation. Actions may also contain one moreparameter tags (not shown in Figure 9) that provide theactual parameters to be used when executing the effectormethods. In a similar fashion to the monitor, the executorcomponent uses Java reflection to invoke the effectormethods with the specified parameters.

Figure 10 shows the minimal class diagramof the self-test subsystem, along with classesfrom other communicating subsystems. The fourmanager classes OTMSelfTest, TTMSelfTest,OAMSelfConfig, and OAMSelfProtect, all extendthe KnowledgeCoordinator class from the ACSTFsgeneric manager component. The OTMSelfTest class

implements MAPE functionality of the Orchestrating TMthat manages the Touchpoint TM TTMSelfTest, andmonitors the two Orchestrating AMs OAMSelfConfigand OAMSelfProtect for newly generated changerequests. Test cases for the stack were developed byextending the JUnit class TestCase and stored in theclass StackTest. The class StackCover providesfunctionality for dynamically generating and executinga batch file, which sets the Main class of Cobertura toinstrument the Stack class during testing.

D. Evaluation

The main objective of developing the prototype was tovalidate the architectural model of the integrated frame-work shown in Figure 3. Although the prototype onlyimplements a minimal autonomic system, it providesevidence to support that the replication with validationstrategy is feasible. A mutation testing technique wasused to evaluate the prototype with respect to its abilityto detect faulty change requests for managed resources.Mutation operators were applied to the stack class tocreate mutant stacks, which were used to assess theeffectiveness of the prototype’s self-testing framework.

Our evaluation involved analyzing correct and incorrect(mutant) change request scenarios for self-configurationand self-protection of the stack. The scenario for incorrectself-configuration was achieved by altering the methodthat resizes the stack to cause a decrease rather than anincrease in the stack capacity. Similarly, incorrect self-protection of the stack involved altering the method thatdisables a specific user account to enable rather thandisable the user. The self-testing framework was thenallowed to operate on the original version of the stack,and the two mutants with the faulty methods.

In the correct change request scenarios for self-configuration and self-protection, all of the tests passedand the following code coverage measurements wererecorded: self-configuration – 85% branch coverage,100% statement coverage; and self-protection – 88%branch coverage, 100% statement coverage. The scenariosfor the incorrect changes each produced two test casefailures, and hence the code coverage measurements havebeen omitted. Our mutation analysis produced favorableresults as validation passed for the correct change requestscenarios, but failed for the incorrect ones. Therefore, theself-testing framework would have prevented undesirableand potentially harmful changes to the managed resourceof the autonomic system.

Building the prototype provided us with insight on thescope of the self-test subsystem in terms of the operationsthat should be performed by the autonomic system andthe self-testing framework. However, the current versionof the prototype utilizes a static lookup of predefined testplans and therefore does not adequately address the needfor dynamic test planning in autonomic systems. Further-more, the prototype only implements the replication withvalidation strategy and is therefore limited with respect toadaptive systems.



VI. RELATED WORK

Self-testing as an implicit feature of autonomic com-puting systems has received little attention in the researchcommunity. The research focus of autonomic computinghas been on the major self-management properties ofself-configuring, self-healing, self-optimizing and self-protecting. The pioneers of autonomic computing systems[4], [6] clearly state that one of the major challenges isvalidating the correctness of such systems during devel-opment and after changes have been made at runtime.It is important to note however that aspects related tothe validation approaches presented in the paper havebeen studied in the literature. We now compare two suchaspects, self-testing software/components and dynamicadaptation, along with the authors’ previous work to thework presented in this paper.

Several researchers have investigated the notion of self-testing software [24]–[28]. Blum et al. [24] introduceda technique which uses self-testing/correcting pairs toverify a variety of numerical problems. The idea is that auser can take any program and its self-testing/correctingpair of programs and, once the program passes the self-test, on any input call the self-correcting program whichwill make the appropriate calls to the original programto compute the correct value. If this notion of self-testing/correcting program pairs worked for complex pro-grams then self-testing for autonomic computing systemswould be trivial. However, this technique only works forvery well defined functions.

Denaro et al. [29] present an approach that automati-cally synthesizes assertions from the observed behavior ofan application with the objective of adaptive applicationmonitoring. The proposed approach embeds assertionsinto the communication infrastructure of an applicationthat describes the legal interactions between the com-municating entities. These assertions are then checkedat runtime to reveal misbehaviors, incompatibilities, andunexpected interactions that may occur because of hiddenfaults. The focus of the work is to provide systemswith the ability to automatically synthesize assertionsthat evolve over time and adapt to context-dependentinteractions. The synthesis of assertions at runtime canbenefit the self-testing of adaptive systems by providinga way to generate additional test cases, which can be usedto test components after an autonomic change has beenimplemented.

The work by Le Traon et al. [27] is closely related toour work; it describes a pragmatic approach to developself-testable components that link design to the testingof classes. Components are self-testable by including testsequences and test oracles in their implementation. Forthis approach to be practical at the system level, struc-tural test dependencies between self-testable componentsmust be considered at the system architecture level. Theconcept of self-testable components strongly supports theidea of self-testing in autonomic computing systems. Inour approach we do not consider the notion of self-testable components. However, such components can be

easily incorporated into our strategy, thereby improvingthe overall approach.

Zhang et al. [8], [15] present an approach that formal-izes the behavior of adaptive programs using state ma-chines and Petri nets, respectively. The approach separatesthe adaptative behavior from the non-adaptive behavior,thereby making the models easier to specify and verifyusing automated techniques. The contributions of thework by Zhang and Chen [15] that can be applied toour work include: (1) specification of global invariantsof the properties of adaptive programs regardless of theadaptations and (2) creation of state-based models that aidin the generation of rapid prototypes. Using the specifi-cations for global invariants and state-based models, testcases can be dynamically generated to test an adaptedprogram at runtime.

The work presented in this paper is an extension of thework done by King et al. [7] and Stevens et al. [20]. Kinget al. proposed a framework that dynamically validateschange to autonomic systems. The framework includesself-testing as an implicit characteristic to support self-management in autonomic systems. Stevens et al. [20]introduced the notion of a self-testing autonomic containerthat uses the self-testing framework proposed by King etal. [7]. The autonomic container uses the replication withvalidation approach to testing the autonomic container.We extended the initial work in [7] by: (1) augmentingthe self-testing framework with an auxiliary test servicescomponent and test knowledge sources to assist high-leveltest coordination and facilitate manual test management;(2) explicitly illustrating of how the test managers interactwith these new components; and (3) stating researchdirections that address some of the key challenges facedby this work. The prototype presented in this paper ex-tended the autonomic container in [7], [20] to include bothdynamic self-configuration and self-protecting features.In addition, the container was implemented as a dis-tributed system using a client-server architecture. Finally,additional details of the implementation were presentedincluding the generic manager design, and the structureof the policies used by the managers.

VII. CONCLUSION

In this paper we presented an integrated self-testingframework for autonomic computing systems based ontwo strategies: safe adaptation with validation, and repli-cation with validation. Our framework extends the currentarchitecture of autonomic systems to include self-testingas an implicit characteristic. We developed a prototype ofan autonomic computing system which incorporates self-testing. Clearly many challenges still remain, however,this work aims to stimulate the convergence of a set oftesting and development methodologies that facilitate theconstruction of dependable autonomic systems.

ACKNOWLEDGMENT

This work was supported in part by the NationalScience Foundation under grants IIS-0552555 and HRD-0317692. The authors would like to thank Djuradj Babich,



Jonatan Alava, Ronald Stevens, and Mitul Patel for theircontributions to this work.

REFERENCES

[1] IBM Corporation., “IBM Research,” 2002,http://www.research.ibm.com/autonomic/ (August 2006).

[2] Enterprise Managment Associates, “Practical autonomiccomputing: Roadmap to self managing technology,” IBM,Boulder, CO, Tech. Rep., Jan. 2006.

[3] IBM Autonomic Computing Architecture Team, “An ar-chitectural blueprint for autonomic computing,” IBM,Hawthorne, NY, Tech. Rep., June 2006.

[4] J. Kephart and D. Chess, “The vision of autonomic com-puting,” Computer, vol. 36, no. 1, pp. 41–52, January 2003.

[5] J. O. Kephart, “Research challenges of autonomic comput-ing,” in ICSE ’05: Proceedings of the 27th internationalconference on Software engineering, 2005, pp. 15–22.

[6] H. A. Muller, L. O’Brien, M. Klein, and B. Wood,“Autonomic computing,” Carnegie Mellon Univeristy andSoftware Engineering Institute, Tech. Rep., April 2006.

[7] T. M. King, D. Babich, J. Alava, R. Stevens, and P. J.Clarke, “Towards self-testing in autonomic computing sys-tems,” in ISADS ’07: Proceedings of the Eighth Interna-tional Symposium on Autonomous Decentralized Systems.Washington, DC, USA: IEEE Computer Society, 2007, pp.51–58.

[8] J. Zhang, B. H. C. Cheng, Z. Yang, and P. K. McKinley,“Enabling safe dynamic component-based software adap-tation.” in WADS, 2004, pp. 194–211.

[9] B. Beizer, Software Testing Techniques, 2nd ed. NewYork: Van Nostrand Reinhold, 1990.

[10] H. Zhu, P. A. V. Hall, and J. H. R. May, “Software unittesting coverage and adequacy,” ACM Computing Surveys,vol. 29, no. 4, pp. 366–427, December 1997.

[11] D. Mosley and B. Posey, Just Enough Software TestAutomation. Upper Saddle River, NJ, USA: Prentice HallPTR, 2002.

[12] Mosaic, Inc., “Staffing your test automation team,” Mosaic,Inc., Chicago, IL, Tech. Rep., April 2002.

[13] I. Sommerville, Software Engineering: Seventh Edition.Essex, England: Addison-Wesley, 2004.

[14] T. L. Graves, M. J. Harrold, J.-M. Kim, A. Porter, andG. Rothermel, “An empirical study of regression testselection techniques,” ACM Trans. Softw. Eng. Methodol.,vol. 10, no. 2, pp. 184–208, 2001.

[15] J. Zhang and B. H. C. Cheng, “Model-based developmentof dynamically adaptive software,” in ICSE ’06: Pro-ceeding of the 28th international conference on Softwareengineering. New York, NY, USA: ACM Press, 2006,pp. 371–380.

[16] E. Bernard, B. Legeard, X. Luck, and F. Peureux, “Gen-eration of test sequences from formal specifications: Gsm11-11 standard case study,” Softw. Pract. Exper., vol. 34,no. 10, pp. 915–948, 2004.

[17] N. Dulac, T. Viguier, N. G. Leveson, and M.-A. D.Storey, “On the use of visualization in formal requirementsspecification,” in RE ’02: Proceedings of the 10th Anniver-sary IEEE Joint International Conference on RequirementsEngineering. Washington, DC, USA: IEEE ComputerSociety, 2002, pp. 71–80.

[18] K. Feeney, K. Quinn, D. Lewis, D. O’Sullivan, andV. Wade, “Relationship-driven policy engineering for au-tonomic organisations,” in POLICY ’05: Proceedings ofthe Sixth IEEE International Workshop on Policies forDistributed Systems and Networks (POLICY’05). Wash-ington, DC, USA: IEEE Computer Society, 2005, pp. 89–98.

[19] Y. Chen, R. L. Probert, and D. P. Sims, “Specification-based regression test selection with risk analysis,” inCASCON ’02: Proceedings of the 2002 conference of theCentre for Advanced Studies on Collaborative research.IBM Press, 2002, p. 1.

[20] R. Stevens, B. Parsons, and T. M. King, “A self-testingautonomic container,” in ACM-SE 45: Proceedings of the45th annual southeast regional conference. New York,NY, USA: ACM Press, 2007, pp. 1–6.

[21] E. Gamma and K. Beck, “JUnit 4.3.1 - TestingResources for Extreme Programming,” 2005,http://www.junit.org/index.htm (June 2007).

[22] M. Doliner, G. Lukasik, and J. Thomerson, “Cobertura,”2002, http://cobertura.sourceforge.net/ (June 2007).

[23] Sun Microsystems, Inc., “Core Java J2SE 5.0,” February2005, http://java.sun.com/j2se/1.5.0/index.jsp (Aug. 2006).

[24] M. Blum, M. Luby, and R. Rubinfeld, “Self-testing/correcting with applications to numericalproblems,” Journal of Computer and System Sciences,vol. 45, no. 6, p. 549595, 1993.

[25] G. Denaro, L. Mariani, and M. Pezze, “Self-test compo-nents for highly reconfigurable systems,” Electronic Notesin Theoretical Computer Science, vol. 82, no. 6, 2003.

[26] R. Kumar and D. Sivakumar, “On self-testing withoutthe generator bottleneck,” in Proceedings of the 15thConference on Foundations of Software Technology andTheoretical Computer Science. London, UK: Springer-Verlag, 1995, pp. 248–262.

[27] Y. L. Traon, D. Deveaux, and J.-M. Jezequel, “Self-testable components: From pragmatic tests to design-for-testability methodology,” in Technology of Object-OrientedLanguages and Systems (TOOLS 99). IEEE ComputerSociety, 1999, pp. 96–107.

[28] H. Wasserman and M. Blum, “Software reliability via run-time result-checking,” J. ACM, vol. 44, no. 6, pp. 826–849,1997.

[29] G. Denaro, L. Mariani, M. Pezze, and D. Tosi, “Adap-tive runtime verification for autonomic communicationinfrastructures,” in First International IEEE WoWMoMWorkshop on Autonomic Communications and Computing(ACC’05), 2005, pp. 553–557.

Tariq M. King is currently a Ph.D. candidate at FloridaInternational University (FIU). He received his MS and BSdegrees in computer science from FIU and the Florida Instituteof Technology in 2007 and 2003, respectively. His researchinterests include autonomic computing, model-based testing, andmodel checking. Tariq is a member of the ACM, IEEE ComputerSociety, and Software Testing Research Group (STRG) at FIU.

Alain E. Ramirez is an undergraduate student in ComputerScience at FIU who is currently participating in the Summer2007 Research Experience for Undergraduates (REU) program,which is sponsored by the National Science Foundation. He isa member of the ACM, IEEE Computer Society, and STRG.

Rodolfo Cruz is a graduate student at FIU currently pursuingan MS degree in Computer Science. He received his BS degreein computer science from FIU in 1997. Rodolfo has over 10years experience in the software development industry, and iscurrently a project manager for Oceanwide USA, Inc. He is amember of the ACM, IEEE Computer Society, and STRG.

Peter J. Clarke received his Ph.D. in Computer Sciencefrom Clemson University in 2003, and his M.S. degree fromBinghamton University in 1996. His research interests includesoftware testing, software metrics, software maintenance, andmodel-driven development. He is currently an Assistant Pro-fessor in the School of Computing and Information Sciences atFIU. He is a member of the ACM, and IEEE Computer Society.



An Integrated Self-Testing Framework for Autonomic Computing...

Documents

Transcript of An Integrated Self-Testing Framework for Autonomic Computing...