Relative debugging in an integrated development environment

SOFTWARE—PRACTICE AND EXPERIENCESoftw. Pract. Exper. 2009; 39:1157–1183Published online 7 July 2009 inWiley InterScience (www.interscience.wiley.com). DOI: 10.1002/spe.932

Relative debugging in anintegrated developmentenvironment

David Abramson1,∗,†, Clement Chu1, Donny Kurniawan1

and Aaron Searle2

1Faculty of Information Technology, Monash University, Clayton, Vic. 3800,Australia2Faculty of Information Technology, Queensland University of Technology,Gardens Point, Brisbane, Australia

SUMMARY

Relative Debugging allows a user to compare the internal state of two programs as they run, making itpossible to test whether two programs perform the same function given the same input. When implementedwith a command line user interface, a relative debugger looks like traditional debugging tools with theaddition of commands that describe which structures should be equivalent in the two programs. In thispaper, we discuss relative debugging within an integrated development environment, and show that thereare significant advantages over a command line form. We describe a pluggable, modular, architecturethat works with a variety of different products, including Microsoft’s Visual Studio, SUN’s NetBeans, andIBM’s Eclipse. Copyright © 2009 John Wiley & Sons, Ltd.

Received 16 June 2008; Revised 23 April 2009; Accepted 2 May 2009

KEY WORDS: debugging; integrated development environments; software tools

1. INTRODUCTION

It is not uncommon to have multiple versions of a program—one that ‘works’ and one that is‘broken’. A new version may not work correctly because the programmer changed some code.These errors can often be quite difficult to find, because a change in one place may have an effectsomewhere completely different. Alternatively, something may have changed in the environment.

∗Correspondence to: David Abramson, Faculty of Information Technology, Monash University, Clayton, Vic. 3800, Australia.†E-mail: [email protected]

Contract/grant sponsor: Australian GovernmentContract/grant sponsor: Object Technology InternationalContract/grant sponsor: MicrosoftContract/grant sponsor: IBMContract/grant sponsor: SUN

Copyright q 2009 John Wiley & Sons, Ltd.

1158 D. ABRAMSON ET AL.

For example, the operating system or compilers may have been upgraded, and exactly the samesource code may produce different results. Finally, the programmay work correctly on one platform,but fail on another. This could occur because of differences in the hardware, compilers, operatingsystems, or any number of differences between the two platforms. Finding such bugs can be tediousand error prone.Relative debugging is a technique with which it is possible to compare the internal state of two

executing programs [1]. It is an amalgam of test and debugging methodologies. It enables a user totestwhether two programs perform the same task by comparing the output of the two programs giventhe same input. However, it goes beyond traditional test tools by facilitating the comparison of arbi-trary data structures at arbitrary points in the execution of the two codes. Thus, it supports debuggingof a new version of the program by helping to identify the point at which the internal states of thetwo codes diverge. Relative debugging is a powerful technique for refining a region where a programfirst produces incorrect results. A programmer does this by following the uses of key variables fromtheir use points back to their definition points, in an attempt to isolate the region of code in error.Initially, we implemented a relative debugger, called Guard, using a traditional command line

user interface. To the user, Guard looks like standard debugging tools such as gdb, dbx, etc. Infact, many of the commands available in existing debuggers are also provided in Guard. On top ofthese, Guard adds a few key commands for performing relative debugging—typically commandsthat describe the structures that should be equivalent in the two programs. However, while powerful,we believe there are significant advantages in using powerful graphical user interfaces (GUIs) suchas those exposed in integrated development environments (IDEs). In this paper, we discuss relativedebugging in an IDE, and show that there are significant advantages to this. We describe a pluggable,modular, architecture that works with a variety of different products, including Microsoft’s VisualStudio, SUN’s NetBeans, and IBM’s Eclipse.The paper begins by introducing relative debugging, followed by a discussion of various commer-

cial IDEs, and highlights their functionality. The material provided in Section 2 is by way ofbackground, and is not intended to serve as a single source of information on relative debugging.Indeed, the major contribution of this paper is the design, implementation, and demonstration ofrelative debugging in an IDE. We then discuss some generic advantages that an IDE offers relativedebugging, followed by a discussion of the architecture that we developed. In particular, an IDEcan automate some of the more tedious and error prone aspects of the previous implementations.We then show how the various features of a relative debugger in an IDE are implemented in oneor more of the IDEs.

2. RELATIVE DEBUGGING

Programs evolve for a large number of reasons. In many situations, large portions of the modifiedprogram should produce the same results as the existing version. For instance, if a program is:

• enhanced with new functionality, the existing functionality should remain the same;• modified to remove an error, the existing unrelated functions should produce the same results.That is, the removal of an error should not introduce new errors;

• ported from one platform and/or language to another, the functionality in the ported programshould remain the same.

Copyright q 2009 John Wiley & Sons, Ltd. Softw. Pract. Exper. 2009; 39:1157–1183DOI: 10.1002/spe

RELATIVE DEBUGGING IN AN IDE 1159

In these cases, it is common that the unaffected functionality or program code in the developmentversion will contain data structures that should contain the same values as the original version whenthe two programs are run with similar inputs.Relative debugging exploits the information available in previous versions to allow the contents

of key data structures to be compared. This allows the user to locate errors by identifying datastructures that contain different values when the two programs are executed.The concept of relative debugging is both language and machine independent. This allows a user

to concentrate on the cause of the error rather than the underlying implementation details. Theresults of several case studies have been reported and show that relative debugging is an extremelyeffective technique that allows users to locate errors in programs that have been modified or portedto another platform and/or language. In one case study we compared the execution of two largeclimate models, and were able to isolate errors that were introduced during the parallelization ofthe algorithms [2]. In this example, subtle semantic errors occurred when the code was modifiedto allow it to run in parallel, and while it was originally believed that these would not affect theoutput, our experiments not only illustrated that they did cause different results to be computed, butalso helped us locate the source. In another study, we isolated errors in a complex photochemicalpollution program as it was ported from one platform to another [1]. In this example, relativedebugging allowed us to isolate significant differences in output that were caused by very minornumerical differences in some external library functions (transcendental), and actually highlighteda much more significant error in the overall numerical stability of the code. The technique has alsobeen used to find errors in a nuclear transport kernel as it was translated from one language (C) toan experimental parallel language (ZPL) [3]. This example was interesting because it isolated fourerrors in the codes—three of which were syntactic errors in the new program, and one of which wasincorrect mathematics in the original C program. The latter error had subsequently been correctedin the new ZPL program. The case study also demonstrated that relative debugging is effectiveeven when the two programs are written in different languages and with different programmingparadigms. Many of these examples concerned quite large, complex, scientific codes. For example,the MM5 weather model used in [2] consists of around 300 000–400 000 lines of Fortran code. Inaddition to various research projects, relative debugging has been deployed commercially [4], andwas recently adopted by a major supercomputer vendor to aid productivity in the development ofsupercomputing applications [5].Figure 1 illustrates the relative debugging paradigm using a very simple example. While this

program is trivial, it allows us to illustrate the way the technique works. The program on theleft depicts the original program, written in Visual Basic, containing a function that computes thefactorial for a given number. The program on the right represents the later development versionthat has been rewritten in C++ and uses a different algorithm to compute the factorial. Despitethese differences, the two programs should compute the same factorial value for the given number.Relative debugging may be used to compare the factorial value computed by each program. If thecomparison fails, the error can be attributed to the modified factorial function in the developmentversion. In this case, the user may use traditional debugging techniques or continue to apply relativedebugging to localize the source of the error even further.Relative debugging requires the user to identify the corresponding data structures and program

points at which comparison should be made within the two programs when they are concurrentlyexecuted. The choice of data structures and program points must be determined by the user, basedon some knowledge about the two programs. It is not necessary to test all data structures, but only



Figure 1. The relative debugging paradigm.

those that help uncover the source of the error. The user provides the comparison details by definingone of the following two types of assertions: imperative assertions and declarative assertions.

2.1. Imperative assertions

An imperative assertion allows the user to manually invoke the comparison of key data structureswhen the two programs under consideration are suspended. Typically, the user creates a numberof breakpoints, at key locations, in the two programs before they are simultaneously executed.When the breakpoints are reached and the programs are subsequently suspended, the user can issuea compare command to compare the contents of key data structures. For example, the followingcompare command compares the variable factVal in the program orig, with the variable fact in theprogram dev.

compare $orig :: factVal=$dev :: fact

The user may perform a number of comparisons and gain insight into the error by identifyingvariables that contain different values across the two programs. To identify where the two programsfirst begin to differ and isolate the source of the error, the user can define additional breakpoints,resume or restart the two programs, and continue to compare the contents of key data structures.This process can become tedious and error prone if the user needs to iterate through this process

a number of times. The process also becomes laborious if the user wishes to compare the results ofa computation that occur within a loop. In this case, the user must manually invoke the comparisoneach time an iteration is performed.



2.2. Declarative assertions

A more flexible and less tedious method for defining the comparison details of corresponding datastructures is specified by a user-supplied declarative assertion. The declarative approach allows theuser to assert that two data structures should contain the same values at particular locations in thetwo programs.Declarative assertions are defined before the programs are executed and state that a data structure

at a certain line in the original program should contain the same value as the corresponding datastructure, at a specific line, in the development program.Declarative assertions are defined by the assert command. The following assertion statement, for

example, states that the variable factVal, in line 114, in the original program should be comparedwith the variable fact, in line 121, in the development program.

assert $orig :: factVal@114=$dev :: fact@121

The additional line information provided with declarative assertions allows the creation of break-points and the comparison of variables to be automated by the relative debugger. In particular,the relative debugger creates a breakpoint in both the original and development versions, for eachassertion, at the specified lines. When these breakpoints are reached during execution of the twoprograms, the relative debugger automatically compares the variables specified by the assertion. Ifthe comparison does not detect an error, the programs are automatically resumed without any userinteraction.Declarative assertions provide a convenient mechanism for stating a set of conditions that must

be satisfied if the development version is to be deemed correct. Hence, not only are they usefulfor isolating the source of an error, but they can also be used to automatically test a new programversion against a previous version.Declarative assertions are also effective if the user wishes to compare the results of a computation

that occur within a loop. Unlike the imperative approach, the comparison is automatically performedeach time the loop is iterated. As the user is not involved until an error is detected, little userinteraction is required to actually detect the presence of an error.

2.3. A systematic approach to relative debugging

Once the user has defined the assertions, the programs are executed and the values of correspondingdata structures are compared at the specified locations. If they are run on different platforms theycan be executed concurrently, thereby overlapping the two computations and saving time. If acomparison fails, execution is halted and the difference is reported to the user. In the case of an error,the user normally repeats the process of program comparison, with additional or refined assertions,in order to identify the point where values first begin to differ.To identify the point where values first begin to differ, the user typically formulates additional

assertions based on the variables used in the computation of the erroneous value. The methodologyfor deciding where to place assertions is based on following the data and control flow of the twoprograms. For example, if the output of a computation is reported to be incorrect, the user woulddefine additional assertions to follow the inputs to the faulty computation—typically by findingthe definition points of the variables used in the computation. This process usually continues by a



process of refinement until the faulty section of code is localized after which traditional debuggingtechniques can be used to correct the error.This systematic approach allows users to locate an error by iteratively defining assertions until the

erroneous region is small enough to inspect manually. The technique is extremely efficient becausethe amount of code that needs to be considered is dramatically reduced after each refinement step.Hence, even in large programs, the user can localize the error to a small region of code with alimited number of refinement iterations.

2.4. Approaches to automatic debugging

There have been attempts over the years to automate the debugging process fully. In 1991, Agrawalstated that ‘automatic debugging has been investigated but never really achieved’ [6]. A review in1988 had already noted that ‘most automatic debugging tools were currently only prototypes anddid not scale easily to large systems’ [7], mainly due to the difficulty involved with capturing theprogrammer’s intuition.Since the early eighties, a number of research projects and experiments that adopt novel

approaches for automatic debugging have been conducted and have provided promising results.These approaches rely on the correctness of the earlier versions of a program to assist theprogrammer’s understanding of the development system and to help with the isolation of faults inthe development versions. The approaches attempt either to localize faults or to reduce the searcharea to a minimal one that allows the errors to be reproduced.Shapiro proposed diagnostic algorithms [8] to identify procedures or functions that behave incor-

rectly. He proposed an approach known as algorithmic debugging, which is a semi-automatictechnique where the user is required to answer questions concerning the intermediate results ofprocedures within the program.He explains that the ‘approach is geared toward languages in which the basic computation

mechanism is a procedure (or function) call, but is insensitive to the inner workings of procedures.The diagnosis algorithms abstract away all the details of the computation, except the procedurecalls performed, their inputs, and their outputs.’ [8].In 1994 Whalley presented a tool, vpoiso [9], that automatically localizes errors in the vpo

compiler system. The tool isolates optimization errors by determining the first transformation thatcauses the compiled program to produce incorrect results. Further, vpoiso determines a minimalinput that causes the compiled program to produce the incorrect result. Although the tool cannotisolate non-optimizing errors, the minimal input narrows the scope of the problem thereby assistingthe developer to localize the error.DynaDiff [10] is a testing and debugging tool that uses path profiling to isolate potential faults.

The tool runs a program twice, with different inputs, and compares the executed control path. Theuser is informed that an error may exist if an alternate control path is taken or if a control path isexecuted at different number of times.Delta Debugging [11] attempts to isolate the cause of an error without user involvement altogether.

The technique can be used on any artifact or circumstance that influences the execution of a program.Example circumstances are program input, user interaction, and the actual program code.The Delta Debugging algorithm uses systematic testing to automatically isolate failure-inducing

circumstances. A failure-inducing circumstance is a subset of the original circumstance that containsthe minimal number of elements that cause the original error to re-occur. Each element in the



failure—inducing circumstance must be present in order for the error to occur—removing a singleelement eliminates the occurrence of the failure. Removing the elements that do not influence orimpact the failure allows the user to concentrate on the cause of the error.To identify the failure-inducing circumstances, Delta Debugging systematically removes elements

from the original circumstance and re-executes the program to determine if the error remains. Thisprocess continues in an iterative fashion until the original circumstance is decomposed to the subsetof minimal circumstances that re-produce the original error.A similar approach to Delta Debugging has been adopted by Renieris and Reiss [12], but they

attempt to address the assumption that similar inputs result in similar runs. To overcome thislimitation they select a successful run, from a repository of program runs, that mostly resemblesthe faulty run. The selected successful run is then executed and compared with the execution ofthe faulty run. The difference of these runs is then reported to the user to assist with locating thesource of the error.This difference also eliminates the assumption imposed by Delta Debugging that similar inputs

result in similar runs. WHITHER attempts to address this assumption by selecting a successfulrun, from a repository of program runs, that mostly resembles the faulty run. The comparisonapproach using potential invariants, adopted byWHITHER, is similar to relative debugging becauseit compares a variable from the faulty run with the expected value in the successful run. However, theapproach compares arbitrary variables based on the successful runs and assumes that the variablesin the successful run should be the same for all runs in the faulty run. Furthermore, the comparedvariables may not actually influence the faulty program run. Inversely, the approach does notconsider situations where the compared variables should actually differ between the two programruns.In [13], Zeller states ‘All in all, relative debugging exploits the existence of a reference run in

a classical interactive debugging session. The more of the state and the run is covered by relativeassertions, the easier it will be to catch infections early; best results are achieved when porting anotherwise identical program from one environment to another’.In comparison with the techniques discussed here, relative debugging is not fully automatic, and

still involves the programmer in deciding where to place assertions and in using their intuitionin finding errors. The goal of this paper is to illustrate that IDEs play a role in exposing thesemechanisms to the user.

3. INTEGRATED DEVELOPMENT ENVIRONMENTS

3.1. Functionality

Developing an application and supporting the full software development life cycle often requirea variety of programming tools from different tool vendors. A closer integration of these toolscould expedite and streamline the development process [14]. This observation leads to the creationof an IDE. The first IDE was arguably built for the BASIC programming language [15], andwas a text-based system that seamlessly integrated code editing, compilation, and programexecution [16].An IDE combines various programming tools such as a source code editor, a compiler, and

a debugger into one cohesive environment. The integration allows the tools to work together



seamlessly; for example, the debugger can highlight source code errors in the editor and the compilercan be invoked automatically when the code changes. An IDE is a type of computer-aided soft-ware engineering (CASE) tool [17], and encompasses a wide range of tools and methods that areused to support software engineering activities such as requirement analysis, system modeling, andtesting [18].IDEs often have a steep learning curve and can be difficult to master [19,20]. However,

several studies have shown that once a particular development environment has been mastered,it can significantly increase programmers’ productivity [21–25]. IDEs are effective becausethey improve productivity in three areas, namely efficiency [21], management control, andquality [25]:

• Efficiency: IDEs provide various automation functions, such as refactoring and code-generationsupport, that can substantially reduce the amount of time required to develop an application.IDEs also promote efficiency by providing development tools that have consistent and cohesiveuser interfaces.

• Management control: IDEs equip programmers with better control over the developmentprocess through features such as automatic recompilation and source code tracking in thedebugging phase. In addition, tools such as class browsers and method flow tracers simplifyprogrammers’ task in comprehending the source code.

• Quality: IDEs improve code quality by providing tools such as an automatic syntax checker,an integrated help system, and a source code documentation generator. IDE utilities such asunit testers, profilers, and program analyzers assist programmers in detecting code defects atthe early stage of development.

3.2. Popular IDEs

3.2.1. Visual Studio

Visual Studio is a suite of application development tools from Microsoft for the Windows platformson servers, workstations, personal digital assistants (PDAs), and smartphones [26]. Visual Studio canalso be used to create applications that target Microsoft Office, the .NET framework, and ASP.NETweb framework. It supports several programming languages such as C, C++, C#, J#, ASP.NET,and Visual Basic.NET. Additional third party extensions are available to support other languages.Visual Studio includes various Windows software development kits (SDKs) and it utilizes Microsoftcompilers and debuggers.Microsoft also provides Visual Studio SDK for Visual Studio Industry Partner (VSIP) members

that can be used to extend Visual Studio with additional functions [27]. The core foundation of theSDK is the Visual Studio Shell that provides a base IDE that can host custom tools and programminglanguages. The Shell comes in two modes of operation:

• Integrated mode: Applications built on the integrated Shell will be hosted in the Visual StudioIDE. The mode is used if a vendor wants to provide additional functions in the IDE, forexample, support for new programming languages.

• Isolated mode: Applications built with the isolated Shell will be unique stand-alone programs.The mode is used if a vendor wants to develop a specialized tool that does not interact withthe Visual Studio IDE, but still leverages the Shell’s features.



3.2.2. NetBeans

NetBeans from Sun Microsystems refers to two distinct products: a platform and an IDE. TheNetBeans Platform is a software framework for building cross-platform Java desktop applications[28]. It facilitates the development of rich client applications by providing a number of advancedfeatures such as

• User interface management: The Platform provides high-level presentation components, forexample windows, menus, toolbars, editors, palettes, and wizards. It results in standard andconsistent user interfaces for rich client applications.

• Data and presentation management: The NetBeans Platform contains a rich toolset for datamanipulation and presentation.

• Setting management: A simple and transparent mechanism for saving and restoring usersettings and preferences is provided by the Platform.

• Storage management: The Platform offers an abstraction of file-based data access. It providesa common set of APIs to access files regardless of where they are stored. A file object in theNetBeans paradigm may be physically stored on an FTP server, a local disk, or in a database.

• Additional components: The NetBeans Platform has been designed in a modular way. Thus, itis possible to extend the Platform with additional functions required by the rich client appli-cations. Available extensions include specialized editors, remote data access, and automaticnetwork update support.

The NetBeans IDE is an IDE written in Java and built on top of the NetBeans Platform [29].It can be used to create desktop, web, mobile, and NetBeans-based rich client applications. It hasa module-based architecture and the IDE can be extended with various packs. A NetBeans IDEpack is a collection of related modules to support a new feature. Some examples of the packs are:C/C++ Pack for developing C/C++ applications, Enterprise Pack for writing service-orientedarchitecture (SOA) applications, and Ruby Pack for programming in the Ruby language. Othernotable IDE features are:

• A NetBeans Swing GUI builder for developing Java applications with the Swing toolkit [30].• Project and build systems based on Apache Ant [31].• An integrated version control system that supports CVS and Subversion [32].• An extensive support for developing rich client applications based on the NetBeans Platform.

3.2.3. Eclipse

Eclipse refers to both a software platform and an IDE built on top of the Eclipse Platform [33].Eclipse is written in Java and it runs on multiple platforms where the Java Virtual Machine isavailable.The Eclipse Platform defines a set of frameworks and common services that collectively facilitate

the creation and integration of IDE-based products and rich client applications [34]. The frameworksand services provide facilities that are required by most tool builders such as [35]:

• A platform runtime system for loading and managing plug-ins.• A generic windowing system with portable native widget toolkits.



• Incremental project builders and compilers support.• A language-independent debug infrastructure.• Tools and APIs for managing workspace resources such as files, folders, and projects.• An integrated help system.

The Eclipse Platform is modular and all platform functions are implemented as plug-ins. An Eclipse plug-in is the smallest unit of the platform that implements a well-definedfeature.The Eclipse IDE is a generic IDE built on top of the Eclipse Platform. It has the standard

IDE user-interface with multi-window editors, refactoring, syntax highlighting, and other featurescommonly found in advanced IDEs. It can be used to develop Eclipse-based rich client applicationsand software written in various programming languages such as C/C++, PHP, Fortran, Python, andPerl. The IDE can be augmented with new features due to its extensible plug-in system. There aremore than 1000 Eclipse plug-ins providing rich and diverse features for software development [19].A number of notable Eclipse plug-ins provide support for parallel programming, data modeling,embedded devices, database development, web services, and so forth.Eclipse provides a complete suite of frameworks, tools, and environment for application devel-

opment. Similar to NetBeans, it allows developers to create a specialized tool to suit a particularprogramming task.

4. INTEGRATED RELATIVE DEBUGGING

As discussed, IDEs offer significant benefits that aid software development. Importantly, manynew tools are available in commercial IDEs, and these plug-and-play together creating a verypowerful environment. Our experience in integrating Guard into a number of IDEs has highlightedspecific advantages to a relative debugger, namely an improved user interaction, interactive assertionmanagement, integrated data visualization, and language neutrality. Here we explore each of thesein more detail.

4.1. Improved user interaction

In the command line version of Guard, a user creates assertions by typing information aboutthe structures, files, and line numbers. Thus, in order to create an assertion, a user must haveaccess to the two source programs in order to capture the variable names and line numbers.This is potentially error prone and cumbersome, especially when a large number of assertion areinvolved.One of the more significant issues is that assertions are tied to particular line numbers; if the

line numbers change, then the assertions are no longer valid. Further, while it is possible to saveassertions to a file for later restoration, it becomes unlikely that they will still be correct once thefiles have been edited and changed.An IDE offers advantages for both the creation and management of assertions. Since the entire

experience is interactive, it is possible to create a point-and-click technique for making assertions inthe first place. Most IDEs support multiple concurrent open files that can be viewed and manipulatedby language specific source code editors. Many IDEs also allow two totally separate programs to



be open at the same time, making it very easy to build an assertion from two different programs.For example, as will be demonstrated later in the paper, a natural interactive interface involvesthe user pointing to variables in both source windows for both programs, and then requesting theenvironment extract the variable names and line numbers for the assertion.It is also much easier to keep assertion line numbers in step with the source code in an IDE.

When a source file is edited, the IDE can generate an exception that causes the assertions to beupdated. Thus, providing the files are not edited outside the IDE, the assertions can be adjustedautomatically. Further, it becomes fairly easy and natural to allow assertions to be saved during aninteractive session and restored at a later time. This means that assertions can be used for regressiontesting as an application evolves.Integration of the assertion mechanism in the IDE facilitates interaction with other tools’ plug-

ins. One of the more interesting combinations involves the use of source code control systems,such as Microsoft’s SourceSafe or open source projects like CVS and SVN. Source code controlsystems allow a user to maintain multiple versions of a program, and make it easy to movebetween versions. When integrated into the IDE, a user can check out a particular version andbuild it into an executable with ease. It is possible to store partial assertion information, inparticular the variable name and line number, as a named pragma in a source file. Then whentwo different versions of a source file are checked into the IDE, these partial assertions canbe matched and merged into a set of complete assertions. This technique makes it possible todecide points of interest in a program as it is being written. These test points can then be usedlater when a version of a program misbehaves, and it can be regression tested to an earlierversion.

4.2. Building assertions interactively

The relative debugging methodology, presented in Section 2, defines the steps that a user shouldfollow in order to localize faults. In general, if the output of a computation is reported to be incorrect,the user should define additional assertions to follow the inputs of the faulty computation. To dothis, the user will typically locate the definition points of the variables used in the computation.This process usually continues, in an iterative fashion, until the faulty section of code is locatedand traditional debugging techniques can be used to correct the error.To locate variable definitions and construct useful assertions require the user to have a detailed

knowledge of the two programs under consideration. Without this, the task of locating defi-nitions becomes difficult and time consuming. This is especially true in complex multi-fileprograms where there may be multiple definitions that are sparsely scattered throughout theprogram.By applying the relative debugging methodology, it can be seen that the data flow properties of

a program are needed to identify useful data structures and comparison points. Using the data flowproperties of programs, it is possible to build a special data flow browser, which locates and displaysthe definitions that may assign a value to a selected variable in a particular computation. Such atool provides invaluable support for relative debugging by reducing the complexity of identifyingvariable definitions. Furthermore, it decreases the need for users to have a detailed knowledge aboutthe program under consideration. Importantly, such a browser can be added as a plug-in to an IDEand can be integrated with the assertion generator. Thus, as a user navigates the Define-Use chainsthey can build assertions relatively easily.



4.3. Integrated data visualization

In the command line version of Guard, the results of comparisons are reported to standard out. Inan IDE, at the very least, these differences can be reported in a special output window. However,it is possible to improve error reporting by integrating more completely in the environment. Forexample, many IDEs provide sophisticated tree viewers for examining the state of structured data.Record or struct viewers allow a user to fold and unfold sub-fields. Multi-dimensional arrays canbe folded and only those elements of interest can be displayed.Further, it is possible to interact more closely with third party visualization packages. While

this function was supported in the command line version of Guard, it usually involved writingfiles from the debugger, which could then be displayed in a visualization package like IBM’s DX.This technique is extremely powerful and has allowed us to locate very complex errors in the past.However, because the tools are separate, the process tends to be done sequentially—first run thecomputations, then visualize the results. However, by using a visualization package that is integratedinto the IDE, it is possible to use the graphics and displays when refining the assertions.

4.4. Language neutrality

Many IDEs support more than one language system. Specifically, they often provide syntax-directededitors for specific languages, and the variable displays are performed using the same notation asthe base language. For example, multi-dimensional arrays, which often have different syntaxes indifferent languages, adopt the syntax of their host language syntax when manipulated in the envi-ronment.Some IDEs, such as Visual Studio go as far as to define a language neutral virtual machine.

In this case, the environment is able to provide access to the internal state of any programminglanguage supported by the base runtime environment. Visual Studio, through the .NET runtime,currently supports an enormous range of different languages, but the interfaces that are exposed toplug-ins are the same. This makes it very easy to build a plug-in that is independent of the hostlanguage.

5. THE GUARD-IDE

5.1. Architecture

The plug-in architecture and platforms for IDEs vary significantly, so we have designed the Guard-IDE to be as generic as possible. Figure 2 shows the architecture and identifies two main parts:

• an IDE package, which resides within the target IDE. This provides an implementation of theGUI for managing assertions, and exposes important debugging interfaces in the IDE that arerequired by the Guard Interpreter. Importantly, while the interface to this package is the sameacross different IDEs, each implementation requires different code because of the significantdifferences in platform;

• the Guard package, which is external to the IDE. This provides the core relative debuggingfunctionality (including a sophisticated data-driven execution engine), as well as necessary



Figure 2. The Guard-IDE architecture.

infrastructure for communication between the IDE and Guard. This component is genericacross all implementations.

The Guard package is actually built from original ClassicGuard components. It encapsulatesGuard’s core relative debugging functionality within a module called the Guard Interpreter. InClassicGuard, debug functions (such as managing breakpoints, evaluating expressions, etc.) areprovided by a separate client–server-based debugger, built on top of GDB. However, in Guard-IDEthese debug interfaces are provided instead by the target IDE itself. This leverages the functionsthat are already available in the IDE, and allows us to support multiple languages in a way that isconsistent with the basic IDE operation.Both the Guard package and the specific IDE packages utilize different communication mecha-

nisms depending on the available infrastructure. The Java-based IDE, such as Eclipse, WebSphere,and Netbeans all use Java Native Interface (JNI) as the communication infrastructure; however,VS.Net uses COM under Windows.The Guard package is almost identical in each of the different Guard implementations, except

for the VS.NET version that uses COM instead of Java. Its interface is shown in Figure 3, and isdivided into four main categories. The interface provides functions that:

• start, stop, and control the core debugging engine (Core). A detailed description of the engine,which is based on a dataflow graph interpreter, is beyond the scope of this paper, but isdescribed in more detail in [36];

• build and maintain assertions (Assertion management);• attach to and manage individual processes (Processor Control); and• receive notification of breakpoints (Breakpoint).

The first three interface classes are invoked by the IDE in response to user interactions, and theBreakpoint interface is invoked when the IDE is notified of a breakpoint event.The IDE packages, on the other hand, differ from each other because there is no common stan-

dard structure for IDE plug-ins; each system has different techniques and underlying assumptions.



Core: StartGuard(), StopGuard(), CreateGraph(), StartGraph(), StopGraph()Assertion management: AddMap(), AddAssignment(), AddAssertion()Process Contol: AttachProc(), InvokeProc(), CreateProcess(), DestroyProcess()Breakpoint: BreakpointHit()

Figure 3. Guard package interface.

Process control: AutoSetLineBreak(), AutoDeleteBreak(), AutoGo(), AutoStop(), AutoBreak()Data: AutoEvalExpr()Display: DisplayData(), DebugMessage(), ErrorMessage()

Figure 4. IDE package interface.

However, they required roughly the same amount of code to expose the core Guard functionsthrough the user interface. The interface is divided into three classifications, shown in Figure 4,and these provide functions that:

• allow the core debug engine to set andmanage breakpoints (Process control). These are requiredin order to build assertions, and are typically supported by the debugger interface used by theIDE;

• evaluate expressions (Data). This is required to execute assertions when they fire, and again,is supported by the native IDE debugger; and

• display information such as general messages in the IDE (Display).

5.2. Eclipse implementation

While each of the IDEs discussed in Section 3 offer similar user-level functionality, their imple-mentation details differ significantly. As a representative example, we highlight some details of theEclipse plug-in called EclipseGuard (Figure 5).EclipseGuard makes significant use of Eclipse’s plug-in and extensibility concept. At its core,

Eclipse has a modular Java runtime called Equinox. Equinox is an implementation of the OSGiR4 core framework specification, which provides a set of bundles and services required to supportrunning OSGi-based systems. The platform is the middle tier of the architecture, and consistsof a set of components that provide core services and frameworks to higher tiers. The top tierof the architecture incorporates plug-in features that provide the functionality most visible to theusers.Importantly, Eclipse plug-ins allow other plug-ins to extend or customize portions of their func-

tionality by declaring extension points and extensions. For example, many programming languages’development tools are already available in Eclipse, such as JDT for Java development tools and CDTfor C/C++ development tools, and these provide language specific editors, views, and debugging



Figure 5. EclipseGuard architecture.

tools, to name a few. Importantly, each of the language plug-ins can be extended by Guard, whichallows us to perform relative debugging across multiple languages and development environments.Figure 6 shows the twomain packages of the architecture of EclipseGuard as well as the individual

components that define the relationship between Eclipse and Guard.

5.2.1. Guard package

As discussed, the Guard package is external to Eclipse and provides relative debugging functionalityby reusing the original components of ClassicGuard. Since the Guard Interpreter is written in C,we implemented an interface layer as a wrapper on top of the Guard Interpreter to interact with theJava-based IDE package. The JNI is used to facilitate calls out of the Guard Interpreter to the IDEpackage, such as those required to place breakpoints, perform expression evaluation, etc.

5.2.2. IDE package

The Eclipse IDE package is responsible for implementing Guard’s GUI and provides functions suchas debugger control, setting breakpoints, receiving breakpoint events, and expression evaluation.



Figure 6. EclipseGuard packages.

This package uses a model-view-controller (MVC) design pattern and provides a generic interfacethat handles controlling and monitoring of relative debugging:

• A generic high-level relative debug model that represents the assertions and comparison results.• Views that give the user graphical visualization of the model’s status and provide a userinterface to control relative debugging by transferring commands.

• A controller that keeps the model updated by generating events based on the messages fromthe external Guard package.

6. IMPLEMENTATION

In this section we illustrate the ideas discussed in Section 4 through a set of small applications. Wedeliberately chose small examples to highlight the features of the user interface rather than to showthat relative debugging scales to quite large applications (as discussed in Section 2). Importantly,not all of the ideas from Section 4 have been implemented in all IDEs. The table below summarizeswhich features have been implemented in the various IDEs.



Feature Visual Studio Eclipse Netbeans

Improved User Interaction√ √ √

Building assertions interactively√

Integrated Data Visualization√

Language Neutrality√ √ √

6.1. Improved user interaction

A screenshot of the EclipseGuard plug-in running under the Eclipse IDE is shown in Figure 7. Thisscreen clearly shows the two source editors, one for Java and the other for C, supported by JDT andCDT plug-ins, respectively. In order to retrieve variable information from different language editors,small third party EclipseGuard language plug-ins are implemented to extend those developmenttools’ plug-ins. The screen also shows information about an assertion in the pane labeled with A,the comparison output for that assertion in pane labeled B, the debug messages in the pane labeledC, and a button to add an assertion (D).

Figure 7. Main screen of EclipseGuard plug-in.



Figure 8. Assertion dialog. Creating an assertion interactively.

Figure 9. The output of assertion results.

An assertion is created in a few simple steps by selecting a statement in one program and (right)clicking on the variable in a source editor. The variable information such as line number, variablename, source file, and project name are extracted automatically and added to an assertion dialogas shown in Figure 8. The information for the other side of the assertion is added in the sameway, except using the alternate source window. The user is also able to specify properties aboutthe assertion such as what comparison method should be displayed, when the debugger should bestopped, and whether the debugger should be stopped on a failed assertion. A completed assertionand its details will be displayed on the Assertion view while the view is able to contain a numberof assertions by repeating this process. Thus, it is very easy to create a number of assertions.Importantly, because the system is integrated into the IDE, when the source code is edited and theline numbers change, the assertions are adjusted automatically.



Once both halves of an assertion have been encountered, the comparison result of the assertionis displayed. After Guard performs the comparison, the result is sent to the IDE package via JNI.The comparison result view shows the assertion result by applying the MVC design pattern. Thecontroller updates the assertion result model and generates an event to notify the viewer to displaythis model. Figure 9 shows the output of the result of an assertion, in which two scalars havedifferent values.

6.2. Building assertions interactively

To support the ideas discussed in Section 4.2, we have developed a relative debugging data flowbrowser, named DUCT [37,38], which finds and highlights the definitions of a selected variable ata specified program location. DUCT generates use–define (UD) chains for programs that executewithin the Microsoft .NET framework and displays the results within the Microsoft Visual Studioenvironment. A .NET program is contained within a portable executable (PE) file that containscompiler-generated Microsoft Intermediate Language (MSIL) [39] and metadata [40] that conformto the common type specification (CTS) [41]. MSIL is a CPU-independent instruction set that canbe efficiently transformed into native code (by a just-in-time (JIT) compiler) at runtime. Metadataallows a .NET program to be self-describing by detailing the types, signatures, and other data thatthe .NET framework uses.In practice, this allows DUCT to be used with a larger number of development languages

(i.e. any language that targets the Microsoft .NET framework). For research purposes, the frame-work provides a common, well-defined platform in which the tool and exercised techniques can beevaluated. To date, DUCT has been successfully tested on .NET programs built by the GPCP [42]compiler as well as the Microsoft compilers for VB, C#, C++, J#.DUCT has been integrated into Microsoft Visual Studio .NET using the Visual Studio Integration

Program (VSIP). VSIP is a component framework that allows Visual Studio .NET to be extended.The framework allows DUCT to be hosted by Microsoft Visual Studio .NET and utilize the IDEand services offered by the environment.VSIP allows Microsoft Visual Studio .NET to be extended by providing custom functionality

in VSPackages. VSPackages are components that implement interfaces defined by VSIP, allowingMicrosoft Visual Studio .NET and VSPackages to interact. VSPackages can proffer services thatother VSPackages consume. Microsoft Visual Studio .NET also exposes a number of VSPack-ages that proffer a broad range of services and provide access to the environment’s windowingfunctionality.To locate the definition(s) reaching a variable, the user must select the variable in the source

program and inform DUCT to locate the definitions. This is achieved by:

• Placing the cursor over the variable of interest.• Selecting ‘Construct UD Chain’ on the popup context menu, which is activated by clickingthe right mouse button.

DUCT currently recognizes scalar variables, objects, object member fields, and arrays. Thelocated definition(s) are clearly highlighted in the source code window of the IDE and listed in aseparate window, allowing the user to easily navigate through the definitions.For example, Figure 10 illustrates the UD chain constructed by DUCT for the variable x in line

22 in a simple C# program. Each definition that reaches the use of x in line 22 is highlighted in



Figure 10. Example DUCT screenshot.

the source code window and detailed in the output window at the bottom of the IDE. The usermay navigate to a particular definition by clicking on the definition in the output window. This isparticularly useful when a definition is not displayed on the screen or is located in another file.In our experience, the use of a tool like DUCT improves the user productivity significantly

because the navigation of UD chains becomes much simpler—especially when the code is spreadover many screens and even source files.

6.3. Integrated data visualization

As discussed, it is possible to leverage the existing display techniques when a relative debuggeris integrated into a particular IDE. Here we show that EclipseGuard is able to reuse the standarddisplay methods to not only show state values, but also differences in values.



Figure 11. EclipseGuard comparison results.

Figure 11 shows differences reported between two simple scalars being evaluated within a loop.Consequently there are multiple values reported for a single variable. Here it is only necessary toreport the values of the two variables at each iteration, the difference (in this case 0) and a graphicalicon that shows whether they are the same or not (in this case, ticks).Figure 12, on the other hand, shows the display of differences in dynamic structured data types.

Here, EclipseGuard has traversed a linked list, displayed each of these, and has also displayed thedifference between the structures as a list. Again, it uses a simple graphical icon to indicate thatthe structures are actually the same.Figure 13 shows how a tree viewer can help display much more complex structures. In this case,

the display shows the values from either structure and clearly highlights (using a shaded line) wherethe structures differ—in this case, the data was not only different but the type definitions wereincompatible.Finally, Figure 14 shows the difference between two arrays, with the locations of the differ-

ences highlighted. This particular visualization can display multi-dimensional arrays by takingtwo-dimensional slices.An important aspect of all of these data visualization techniques is that they leverage interfaces

that are already available in Eclipse or the various plug-ins. For example, the Array Viewer shown



Figure 12. Display of structured data.

Figure 13. Leveraging native structure viewers.

in Figure 12 was originally developed as part of the Parallel Tools Platform (PTP) project [43], butis equally accessible in Eclipse Guard.On the other hand, it is also possible to invoke powerful external packages to augment the internal

mechanisms. For example, Figure 15 shows the types of images that are possible using external



Figure 14. Array viewer.

visualization packages. In this example, a scientific code simulates the effect of dropping a solidobject into a tank of liquid. The system is modeled as a collection of discrete, but interacting,particles, governed by Newton’s laws of motion. In Figure 15 we show the contents of the keydata structure for both the reference (a) and debugged code (b) where each particle is mappedto its coordinates in two dimensions, as produced by IBM Data Explorer. While each of thesevisualizations is useful in its own right, it is very difficult to see any difference between them whenshown as separate images. However, when Guard subtracts the two images, the differences becomevisible. By augmenting each particle to indicate the magnitude of the difference in position (sizeof each particle) and speed (grey scale density or colour in online version), it is clear that mostparticles have different positions, and some have dramatically different speeds.

6.4. Language neutrality

All three IDEs exhibit some degree of language neutrality and allow users to manage source projectsin different languages. However, Visual Studio not only maintains a common environment acrossa huge range of languages, but also leverages the .NET machinery to provide a language neutralexecution (and thus debugging) platform. VSGuard makes significant use of this functionality andallows cross language relative debugging.In one case study, we used VSGuard to find errors in a program that was converted from Visual

Basic to C# [38]. The program used in this case study accepts a number expressed in base 2, 8,10, or 16, via a graphical interface, and converts it to the equivalent number in base 2, 8, 10, or 16as requested by the user. The first problem we encountered was that the ported program failed to



Figure 15. Scientific visualization of differences: (a) and (b) are original and (c) is difference.

display a result when converting a number from base 16 to base 10. To determine the source of theproblem, we located the functions, in the original and ported program, that perform the conversion.Figure 16 shows the original VB .NET code and the ported C# code in the left and right source codepanes, respectively. Using VSGuard, it was possible to build assertions across the two programs,execute them, and trace the error. Because the runtime system and Visual Studio present uniforminterfaces for both languages, this debugging session proceeded without regard to the language.



Figure 16. Multiple languages in VSGuard.

7. FUTURE WORK AND CONCLUSIONS

In this paper we have shown how our prior work in relative debugging can be extended to operate inIDEs, and importantly, we have shown that there are significant advantages in doing this. Moreover,we were able to build additional tools, like the DUCT dataflow browser, and use this to supportthe creation of assertions—one of the more difficult activities in relative debugging. The richenvironment of an IDE also brings powerful data visualization tools, and these can be adaptedto work with a relative debugger, further enhancing the system capabilities. Finally, the focus ona single environment across different languages means that we can debug two programs that aresupposed to perform the same function, even when they are written in different languages.Some of the capabilities not discussed in this paper concern debugging parallel programs. We

are currently involved in two projects that should deliver significant results in this area. First, theEclipse PTP [43] concerns the development of an open and extensible parallel debugger in anIDE. We have contributed a significant number of components to PTP to date. Second, we arecurrently converting EclipseGuard so that it leverages the PTP runtime, making it possible to userelative debugging on parallel applications. Even though much of this capability was previouslydemonstrated in ClassicGuard [3], we have not been able to show this functionality in an IDE.We are currently investigating extensions to EclipseGuard to allow it to better accommodate

complex project structures. For example, at present, we assume that the two programs beingcompared are actually contained in two separate projects. However, in real world applicationsof Eclipse, users often build programs across multiple projects, which means we need a morecomplex project management approach in EclipseGuard. We are also exploring more powerful data



visualization techniques and integration with source code control systems as discussed inSection 3.2.2. We also have an experimental version of an automatic debugging tool based onrelative debugging, called AutoGuard. AutoGuard builds on DUCT, and is able to automati-cally generate assertions between programs under some circumstances. This has the potential todramatically improve user efficiency and interaction [38].

ACKNOWLEDGEMENTS

This work has been supported by the Australian Government through the ARC Discovery grant scheme. It hasalso been supported by Object Technology International, Microsoft, IBM, and Sun. We wish to thank variouscolleagues who have contributed over the years, including Wojtek Goscinski, Tim Ho, Le Phu, and Greg Watson.

REFERENCES

1. Abramson D, Foster I, Michalakes J, Sosic R. Relative debugging: A new paradigm for debugging scientific applications.The Communications of the Association for Computing Machinery (CACM) 1996; 39(11):67–77.

2. Abramson D, Foster I, Michalakes J, Sosic R. Relative debugging and its application to the development of largenumerical models. Proceedings of IEEE Supercomputing 1995, San Diego, CA, U.S.A., December 1995.

3. Watson G, Abramson D. Relative debugging for data parallel programs: A ZPL case study. IEEE Concurrency 2000;8(4):42–52.

4. Available at: http://www.guardsoft.com [16 June 2009].5. Available at: http://www.guardsoft.com/news.html [16 June 2009].6. Agrawal H. Towards automatic debugging of computer programs. PhD Thesis, Software Engineering Research Centre

(SERC), Purdue University, 1991; 194.7. Ducasse M, Emde A. A review of automated debugging systems: Knowledge, strategies, and techniques. Tenth

International Conference on Software Engineering (ICSE 10), Singapore, 1988; 162–171.8. Shapiro E. Algorithmic Program Debugging. MIT: Cambridge, U.K., 1982.9. Whalley DB. Automatic isolation of compiler errors. ACM Transactions on Programming Languages and Systems

(TOPLAS) 1994; 16(5):1648–1659.10. Reps T, Ball T, Das M, Larus J. The use of program profiling for software maintenance with applications to the year

2000 problem. Sixth European Software Engineering Conference (ESEC 97) held jointly with the Fifth ACM SIGSOFTInternational Symposium on Foundations of Software Engineering (FSE 97), Zurich, Switzerland, 1997; 432–449.

11. Zeller A, Hildebrandt R. Simplifying and isolating failure-inducing input. IEEE Transactions on Software Engineering2002; 28(2):183–200.

12. Renieris M, Reiss S. Fault localization with nearest neighbor queries. Eighteenth IEEE International Conference onAutomated Software Engineering, Montreal, Que., Canada, 2003; 30–39.

13. Zeller A. Why Programs Fail: A Guide to Systematic Debugging. Morgan Kaufmann: Los Altos, CA, October 2005.ISBN: 1558608664.

14. Brown A, McDermid J. Learning from IPSE’s mistakes. IEEE Software 1992; 9(2):23–28.15. Kurtz T. BASIC. History of Programming Languages, Wexelblat R (ed.). Academic Press: New York, 1987; 515–537.16. Boekhoudt C. The big bang theory of IDEs. ACM Queue 2003; 1(7):74–82.17. Fuggetta A. A classification of CASE technology. IEEE Computer 1993; 26(12):25–38.18. Sommerville I. Software Engineering (8th edn). Addison-Wesley: Reading, MA, 2006.19. Kline R, Seffah A. Evaluation of integrated software development environments: Challenges and results from three

empirical studies. International Journal of Human–Computer Studies 2005; 63(6):607–627.20. Reis C, Cartwright R. Taming a professional IDE for the classroom. SIGCSE 2004: Proceedings of the 35th SIGCSE

Technical Symposium on Computer Science Education, Norfolk, VA, U.S.A., 3–7 March 2004; 156–160.21. Case A. Computer-aided software engineering (CASE): Technology for improving software development productivity.

ACM SIGMIS Database 1985; 17(1):35–43.22. Glass R. The realities of software technology payoffs. Communications of the ACM 1999; 42(2):74–79.23. Norman R, Nunamaker J. CASE productivity: Perceptions of software engineering professionals. Communications of the

ACM 1989; 32(9):1102–1108.24. Tsuda M, Morioka Y, Takadachi M, Takahashi M. Productivity analysis of software development with an integrated

CASE tool. ICSE 1992: Proceedings of the 14th International Conference on Software Engineering, Melbourne, Australia,11–15 May 1992; 49–58.



25. Zeller A. The future of programming environments: Integration, synergy, and assistance. FOSE 2007: Future of SoftwareEngineering 2007. IEEE Computer Society: Washington, DC, U.S.A., May 2007; 316–325.

26. Available at: http://msdn.microsoft.com/vstudio/ [22 January 2008].27. Available at: http://msdn.microsoft.com/vsip/ [22 January 2008].28. http://www.netbeans.org/products/platform/ [27 August 2007].29. http://www.netbeans.org/products/ide/ [27 August 2007].30. http://java.sun.com/products/jfc/tsc/articles/architecture/ [28 August 2007].31. http://ant.apache.org/ [28 August 2007].32. http://subversion.tigris.org/ [28 August 2007].33. http://www.eclipse.org/ [29 August 2007].34. des Rivieres J, Beaton W. Eclipse platform technical overview. Available at: http://www.eclipse.org/articles/Whitepaper-

Platform-3.1/eclipse-platform-whitepaper.html [29 August 2007].35. Yang Z, Jiang M. Using eclipse as a tool-integration platform for software development. IEEE Software 2007; 24(2):

55–67.36. Watson G. The design and implementation of a parallel relative debugger. PhD Thesis, School of Computer Science and

Software Engineering, Monash University, Clayton, Australia, 2001.37. Searle A, Gough JK, Abramson DA. DUCT: An interactive define-use chain navigation tool for relative debugging.

AADebug’03, Ghent, Belgium, September 2003.38. Searle A. Automatic relative debugging. PhD Thesis, Queensland University of Technology, 2006.39. Microsoft. Common language infrastructure (CLI): Partition III: CIL instruction set. ECMA TC39/TG3, 2001.40. Microsoft. Common language infrastructure (CLI): Partition II: Metadata definition and semantics. ECMA TC39/TG3,

2001.41. Microsoft. Common language infrastructure (CLI): Partition I: Concepts and architecture. ECMA TC39/TG3, 2001.42. Available at: http://www.plas.fit.qut.edu.au/gpcp/ [16 June 2009].43. Available at: http://www.eclipse.org/ptp/ [16 June 2009].


Relative debugging in an integrated development environment

Documents

Transcript of Relative debugging in an integrated development environment