Post on 01-Jul-2015
description
PROJECT FOXA TOOL THAT OFFERS AUTOMATED TESTING USING A FORMAL
APPROACH
Ivo Neskovic
CITY College Thessaloniki, an International Faculty of the University of Sheffield
462
2
AGENDA
> SOFTWARE ENGINEERING: WHAT COULD GO WRONG?
> FORMAL METHODS
> PROJECT FOX
> CASE STUDY: THE BUFFER SYSTEM
> CONCLUSIONS AND FUTURE WORK
> BIBLIOGRAPHY
3
THE PROBLEM OF SOFTWARE ENGINEERING
> Faulty systems are a common notion nowadays.
> SE is an engineering discipline, yet lacking the engineering formality.
> Subjective and informal testing.
> Impossible to prove that the system:
– Does what it is supposed to do.
– Does not do what it is not supposed to do.
> Needs structured and precise system designs.
4
FORMAL METHODS
> The applied mathematics of computer systems engineering, used to specify and model the behaviour of a system and mathematically verify that the system design and implementation satisfy functional and safety properties.
> Specification Languages:
– Abstract State Machines
– Generalized State Machines
– Communicating Sequential Processes
– Specification and Description Language
– Petri Nets
– Temporal Logic of Actions
– B and event-B method
– Z
5
FORMAL METHODS AT A TRIAL
> Benefits:
– Specification may be used as a basis for proving the presence or lack of certain properties in the design and by inference in the developed system.
– Mathematical proof of correctness (Theorem proving).
– Model checking (Proving desired properties in a design).
– Formal Testing.
> Used mainly for safety critical systems such as aerospace engineering.
> Criticism:
– Expensive and time consuming approach (though questionable).
– Lack of tooling support.
6
INCORPORATING FORMAL METHODS IN THE DEVELOPMENT CYCLE
7
PROJECT FOX
> Produce the complete set of test cases from a formal specification.
> Execute the tests on the systems implementation.
> Locate errors and non-equivalences and report them to the user.
> Developed in Java for Java.
> Compatible with Java Standard Edition, Enterprise Edition, Mobile Edition.
> Can be extend to work in conjunction with popular Java frameworks.
> Operates on compiled bytecode with the addition of a few specific annotations.
> Utilizes the test drivers of JUnit.
> FoX provides a bridge between regular Java developers and the benefits of complete positive and negative testing, proven to find all faults.
8
USING PROJECT FOX
> Two artefacts necessary:
– Formal specification of the system.
– The system's implementation.
9
BUFFER CASE STUDY – DESCRIPTION
> Simple buffer in a factory.
> Accepts parts, any parts.
> Parts have a name and an ID.
> The buffer has a capacity of 2.
> The buffer can be empty, partially full or completely full.
> Supports adding and removing items.
> If the capacity is reached, no additional items can be placed in the buffer unless an item is removed first.
10
BUFFER CASE STUDY – FORMAL SPECIFICATION> Modeled as a Generalized State
Machine (stream X-Machine).
> A theoretical model of computing, pioneered by Samuel Eilenberg in 1974 (X-Machine).
> Separates flow control from processing.
> Flow control is abstracted to a level suitable for representation as a finite state machine.
> Complex data structures are modeled as an infinite memory.
> Able to model both static (data) and dynamic (control) parts of a system.
11
BUFFER CASE STUDY – FORMAL SPECIFICATION (cont.)> Simple buffer in a factory.
< xMachine name = " Buffer " >
> The buffer can be empty, partially full or completely full.
< states >
< state initialState = " true " > empty </ state >
< state > non_empty </ state >
< state > full </ state >
</ states >
12
BUFFER CASE STUDY – FORMAL SPECIFICATION (cont.)> Accepts parts, any parts.
< input name = " part " ref = " BufferObject " / >
> The buffer has a capacity of 2.
< types >
< builtInType name = " capacity " type = " integer " / >
< builtInType name = " buffer " type = " set: BufferObject " / >
</ types >
< memory >
< memoryBlock ref = " buffer " initialValue = " null " / >
< memoryBlock ref = " capacity " initialValue = " 2 " / >
</ memory >
13
BUFFER CASE STUDY – FORMAL SPECIFICATION (cont.)> Parts have a name and an ID.
< types >
< complexType name = " ItemType " >
< attributes >
< builtInType name = " type " type = " string " / >
</ attributes >
</ complexType >
< complexType name = " BufferObject " >
< attributes >
< complexType name = " type " ref = " ItemType " / >
< builtInType name = " itemId " type = " integer " / >
</ attributes >
</ complexType >
< /type >
14
BUFFER CASE STUDY – FORMAL SPECIFICATION (cont.)> Supports adding and removing items.
< functions >
< function name = " add_part " >
< guard >
!buffer. contains ( part ) && buffer . size () + 1 < capacity . value ()
</ guard >
< body > buffer . add ( part ) ; </ body >
< output > Part Added </ output >
</ function >
...
</ functions >
< transitions >
< transition >
< startingState > empty </ startingState >
< appliedFunction > add_part </ appliedFunction >
< endingState > non_empty </ endingState >
</ transition >
...
</ transitions>
15
BUFFER CASE STUDY – IMPLEMENTATIONpublic class ItemType {
private String type;
public ItemType(String type) {
this.type = type;
}
}
public class BufferObject {
private int itemId;
private ItemType type;
public BufferObject(int itemId, ItemType type) {
this.itemId = itemId;
this.type = type;
}
}
16
BUFFER CASE STUDY – IMPLEMENTATION
> @Xmachine annotating the class representing the system modeled with the specification.
> XMachineModel – a class representing the model, containing a number of useful helper methods.
@XMachine(inputType = "BufferObject",
sampleInputs = {
"integer: 10, ItemType: (string:Box)",
"integer: 17, ItemType: (string:HeavyBox)",
"integer: 25, ItemType: (string:ReallyHeavyBox)",
"integer: 20, ItemType: (string:Dragon)",
"integer: 17, ItemType: (string:Planeswalker)",
"integer: 187, ItemType: (string:Nekrataal)",
"integer: 23, ItemType: (string:Michael Jordan)"
})
public class Buffer extends XMachineModel {
17
BUFFER CASE STUDY – IMPLEMENTATION
> @XMMemoryBlock – a field level annotation, associating Java data structures with their specification equivalents
@XMMemoryBlock(name = "buffer")
private List<BufferObject> buffer;
@XMMemoryBlock(name = "capacity")
private int capacity;
public Buffer() {
super("Buffer");
buffer = new LinkedList<BufferObject>();
capacity = 2;
}
18
BUFFER CASE STUDY – IMPLEMENTATION
> @XMFunction – a method level annotation, referencing the modeled functions implementations.
> reportOutcome( outcome: String) – one of the many helper methods of the XmachineModel class.
@XMFunction(name = "add_part")
public void addPart(BufferObject part) {
if (!buffer.contains(part) && buffer.size() + 1 <
capacity) {
buffer.add(part);
reportOutcome("Part Added");
}
}
19
BUFFER CASE STUDY – EXECUTING FOX
20
BUFFER CASE STUDY – EXECUTING FOX (implanted error)if (!buffer.contains(part) && buffer.size() + 1 <
capacity) {
buffer.add(part);
capacity++;
reportOutcome("Part Added");
}
21
BUFFER CASE STUDY – GENERATED TEST CASES> Tests report the sequence of inputs used for the specific scenario, the sequence
of expected outputs and the actual output.
> Outcome is reported to the user via the usual JUnit red / green notifications.
<tests> <test testID=”1”> <input>[ itemId: 187 type: Nekrataal ]</input> <expectedOutput>[ Part Added ]</expectedOutput> <output>[ Part Added ]</output> </test> <test testID=”2”> <input>[ itemId: 17 type: Planeswalker, itemId: 20 type: Dragon]</input> <expectedOutput> [ Part Added, Part Added – Become Full ] </expectedOutput> <output>[ Part Added, Part Added – Become Full ]</output> </test></tests>
22
CONCLUSIONS AND FUTURE WORK
> FoX enables developers to leverage the already proven theories for formal testing.
> Provides a fully automated testing process, ranging from complete test set generation (satisfying some design for test conditions), to test preparation and execution.
> Operates on any Java based software system, being transparent to it's underlining technologies.
> Provides complete positive and complete negative testing.
> Nest steps:
– Thorough evaluation.
– An additional tool to make the specification step easier and closer to the developer, aiming to “hide” the formality as much as possible.
– NetBeans and Eclipse integration.
– A standalone X-Machine IDE providing additional related functionalities.
– Branch out to other languages and frameworks (eg. C# and .NET).
23
BIBLIOGRAPHY
> S. Eilenberg, Automate, Languages and Machines, Vol. A. Academic Press, London, 1974.
> M. Holcombe, “X-Machines as a basis for dynamic system specification,” Software Engineering Journal, vol. 3(2), pp. 69-76, 1988.
> F. Ipate and M. Holcombe, “Specification and Testing using Generalized Machines: a Presentation and a Case Study,” Softw. Test. Verif. Reliab, vol. 8, pp. 61-81, 1998.
> M. Holcombe and F. Ipate, Correct Systems: Building a Business Process Solution. Springer, Applied Computing Series, November 1998.
> G. Eleftherakis and A. Cowling, “An Agile Formal Development Methodology,” in 1st South Eastern European workshop on Formal Methods (SEEFM 03), (Thessaloniki), pp. 36-47, Nov. 2002. Agile Formal Methods: Practical, Rigorous Methods for a changing world.
> P. Kefalas, G. Eleftherakis, and E. Kehris, “Communicating X-Machines: a practical approach for formal and modular specification of large systems,” Information and Software Technology, vol. 45, pp. 269-280, Apr. 2003.
Ivo Neskovic http://twitter.com/trumpets
CITY College ivo.neskovic@gmail.com