SAM 2014 » Formal Technical Process Specification and Verification for Automated Production Systems
Project FoX: A Tool That Offers Automated Testing Using a Formal Approach
-
Upload
trumpets -
Category
Technology
-
view
1.571 -
download
0
description
Transcript of Project FoX: A Tool That Offers Automated Testing Using a Formal Approach
![Page 1: Project FoX: A Tool That Offers Automated Testing Using a Formal Approach](https://reader034.fdocuments.us/reader034/viewer/2022042816/559345461a28abbb728b4710/html5/thumbnails/1.jpg)
PROJECT FOXA TOOL THAT OFFERS AUTOMATED TESTING USING A FORMAL
APPROACH
Ivo Neskovic
CITY College Thessaloniki, an International Faculty of the University of Sheffield
462
![Page 2: Project FoX: A Tool That Offers Automated Testing Using a Formal Approach](https://reader034.fdocuments.us/reader034/viewer/2022042816/559345461a28abbb728b4710/html5/thumbnails/2.jpg)
2
AGENDA
> SOFTWARE ENGINEERING: WHAT COULD GO WRONG?
> FORMAL METHODS
> PROJECT FOX
> CASE STUDY: THE BUFFER SYSTEM
> CONCLUSIONS AND FUTURE WORK
> BIBLIOGRAPHY
![Page 3: Project FoX: A Tool That Offers Automated Testing Using a Formal Approach](https://reader034.fdocuments.us/reader034/viewer/2022042816/559345461a28abbb728b4710/html5/thumbnails/3.jpg)
3
THE PROBLEM OF SOFTWARE ENGINEERING
> Faulty systems are a common notion nowadays.
> SE is an engineering discipline, yet lacking the engineering formality.
> Subjective and informal testing.
> Impossible to prove that the system:
– Does what it is supposed to do.
– Does not do what it is not supposed to do.
> Needs structured and precise system designs.
![Page 4: Project FoX: A Tool That Offers Automated Testing Using a Formal Approach](https://reader034.fdocuments.us/reader034/viewer/2022042816/559345461a28abbb728b4710/html5/thumbnails/4.jpg)
4
FORMAL METHODS
> The applied mathematics of computer systems engineering, used to specify and model the behaviour of a system and mathematically verify that the system design and implementation satisfy functional and safety properties.
> Specification Languages:
– Abstract State Machines
– Generalized State Machines
– Communicating Sequential Processes
– Specification and Description Language
– Petri Nets
– Temporal Logic of Actions
– B and event-B method
– Z
![Page 5: Project FoX: A Tool That Offers Automated Testing Using a Formal Approach](https://reader034.fdocuments.us/reader034/viewer/2022042816/559345461a28abbb728b4710/html5/thumbnails/5.jpg)
5
FORMAL METHODS AT A TRIAL
> Benefits:
– Specification may be used as a basis for proving the presence or lack of certain properties in the design and by inference in the developed system.
– Mathematical proof of correctness (Theorem proving).
– Model checking (Proving desired properties in a design).
– Formal Testing.
> Used mainly for safety critical systems such as aerospace engineering.
> Criticism:
– Expensive and time consuming approach (though questionable).
– Lack of tooling support.
![Page 6: Project FoX: A Tool That Offers Automated Testing Using a Formal Approach](https://reader034.fdocuments.us/reader034/viewer/2022042816/559345461a28abbb728b4710/html5/thumbnails/6.jpg)
6
INCORPORATING FORMAL METHODS IN THE DEVELOPMENT CYCLE
![Page 7: Project FoX: A Tool That Offers Automated Testing Using a Formal Approach](https://reader034.fdocuments.us/reader034/viewer/2022042816/559345461a28abbb728b4710/html5/thumbnails/7.jpg)
7
PROJECT FOX
> Produce the complete set of test cases from a formal specification.
> Execute the tests on the systems implementation.
> Locate errors and non-equivalences and report them to the user.
> Developed in Java for Java.
> Compatible with Java Standard Edition, Enterprise Edition, Mobile Edition.
> Can be extend to work in conjunction with popular Java frameworks.
> Operates on compiled bytecode with the addition of a few specific annotations.
> Utilizes the test drivers of JUnit.
> FoX provides a bridge between regular Java developers and the benefits of complete positive and negative testing, proven to find all faults.
![Page 8: Project FoX: A Tool That Offers Automated Testing Using a Formal Approach](https://reader034.fdocuments.us/reader034/viewer/2022042816/559345461a28abbb728b4710/html5/thumbnails/8.jpg)
8
USING PROJECT FOX
> Two artefacts necessary:
– Formal specification of the system.
– The system's implementation.
![Page 9: Project FoX: A Tool That Offers Automated Testing Using a Formal Approach](https://reader034.fdocuments.us/reader034/viewer/2022042816/559345461a28abbb728b4710/html5/thumbnails/9.jpg)
9
BUFFER CASE STUDY – DESCRIPTION
> Simple buffer in a factory.
> Accepts parts, any parts.
> Parts have a name and an ID.
> The buffer has a capacity of 2.
> The buffer can be empty, partially full or completely full.
> Supports adding and removing items.
> If the capacity is reached, no additional items can be placed in the buffer unless an item is removed first.
![Page 10: Project FoX: A Tool That Offers Automated Testing Using a Formal Approach](https://reader034.fdocuments.us/reader034/viewer/2022042816/559345461a28abbb728b4710/html5/thumbnails/10.jpg)
10
BUFFER CASE STUDY – FORMAL SPECIFICATION> Modeled as a Generalized State
Machine (stream X-Machine).
> A theoretical model of computing, pioneered by Samuel Eilenberg in 1974 (X-Machine).
> Separates flow control from processing.
> Flow control is abstracted to a level suitable for representation as a finite state machine.
> Complex data structures are modeled as an infinite memory.
> Able to model both static (data) and dynamic (control) parts of a system.
![Page 11: Project FoX: A Tool That Offers Automated Testing Using a Formal Approach](https://reader034.fdocuments.us/reader034/viewer/2022042816/559345461a28abbb728b4710/html5/thumbnails/11.jpg)
11
BUFFER CASE STUDY – FORMAL SPECIFICATION (cont.)> Simple buffer in a factory.
< xMachine name = " Buffer " >
> The buffer can be empty, partially full or completely full.
< states >
< state initialState = " true " > empty </ state >
< state > non_empty </ state >
< state > full </ state >
</ states >
![Page 12: Project FoX: A Tool That Offers Automated Testing Using a Formal Approach](https://reader034.fdocuments.us/reader034/viewer/2022042816/559345461a28abbb728b4710/html5/thumbnails/12.jpg)
12
BUFFER CASE STUDY – FORMAL SPECIFICATION (cont.)> Accepts parts, any parts.
< input name = " part " ref = " BufferObject " / >
> The buffer has a capacity of 2.
< types >
< builtInType name = " capacity " type = " integer " / >
< builtInType name = " buffer " type = " set: BufferObject " / >
</ types >
< memory >
< memoryBlock ref = " buffer " initialValue = " null " / >
< memoryBlock ref = " capacity " initialValue = " 2 " / >
</ memory >
![Page 13: Project FoX: A Tool That Offers Automated Testing Using a Formal Approach](https://reader034.fdocuments.us/reader034/viewer/2022042816/559345461a28abbb728b4710/html5/thumbnails/13.jpg)
13
BUFFER CASE STUDY – FORMAL SPECIFICATION (cont.)> Parts have a name and an ID.
< types >
< complexType name = " ItemType " >
< attributes >
< builtInType name = " type " type = " string " / >
</ attributes >
</ complexType >
< complexType name = " BufferObject " >
< attributes >
< complexType name = " type " ref = " ItemType " / >
< builtInType name = " itemId " type = " integer " / >
</ attributes >
</ complexType >
< /type >
![Page 14: Project FoX: A Tool That Offers Automated Testing Using a Formal Approach](https://reader034.fdocuments.us/reader034/viewer/2022042816/559345461a28abbb728b4710/html5/thumbnails/14.jpg)
14
BUFFER CASE STUDY – FORMAL SPECIFICATION (cont.)> Supports adding and removing items.
< functions >
< function name = " add_part " >
< guard >
!buffer. contains ( part ) && buffer . size () + 1 < capacity . value ()
</ guard >
< body > buffer . add ( part ) ; </ body >
< output > Part Added </ output >
</ function >
...
</ functions >
< transitions >
< transition >
< startingState > empty </ startingState >
< appliedFunction > add_part </ appliedFunction >
< endingState > non_empty </ endingState >
</ transition >
...
</ transitions>
![Page 15: Project FoX: A Tool That Offers Automated Testing Using a Formal Approach](https://reader034.fdocuments.us/reader034/viewer/2022042816/559345461a28abbb728b4710/html5/thumbnails/15.jpg)
15
BUFFER CASE STUDY – IMPLEMENTATIONpublic class ItemType {
private String type;
public ItemType(String type) {
this.type = type;
}
}
public class BufferObject {
private int itemId;
private ItemType type;
public BufferObject(int itemId, ItemType type) {
this.itemId = itemId;
this.type = type;
}
}
![Page 16: Project FoX: A Tool That Offers Automated Testing Using a Formal Approach](https://reader034.fdocuments.us/reader034/viewer/2022042816/559345461a28abbb728b4710/html5/thumbnails/16.jpg)
16
BUFFER CASE STUDY – IMPLEMENTATION
> @Xmachine annotating the class representing the system modeled with the specification.
> XMachineModel – a class representing the model, containing a number of useful helper methods.
@XMachine(inputType = "BufferObject",
sampleInputs = {
"integer: 10, ItemType: (string:Box)",
"integer: 17, ItemType: (string:HeavyBox)",
"integer: 25, ItemType: (string:ReallyHeavyBox)",
"integer: 20, ItemType: (string:Dragon)",
"integer: 17, ItemType: (string:Planeswalker)",
"integer: 187, ItemType: (string:Nekrataal)",
"integer: 23, ItemType: (string:Michael Jordan)"
})
public class Buffer extends XMachineModel {
![Page 17: Project FoX: A Tool That Offers Automated Testing Using a Formal Approach](https://reader034.fdocuments.us/reader034/viewer/2022042816/559345461a28abbb728b4710/html5/thumbnails/17.jpg)
17
BUFFER CASE STUDY – IMPLEMENTATION
> @XMMemoryBlock – a field level annotation, associating Java data structures with their specification equivalents
@XMMemoryBlock(name = "buffer")
private List<BufferObject> buffer;
@XMMemoryBlock(name = "capacity")
private int capacity;
public Buffer() {
super("Buffer");
buffer = new LinkedList<BufferObject>();
capacity = 2;
}
![Page 18: Project FoX: A Tool That Offers Automated Testing Using a Formal Approach](https://reader034.fdocuments.us/reader034/viewer/2022042816/559345461a28abbb728b4710/html5/thumbnails/18.jpg)
18
BUFFER CASE STUDY – IMPLEMENTATION
> @XMFunction – a method level annotation, referencing the modeled functions implementations.
> reportOutcome( outcome: String) – one of the many helper methods of the XmachineModel class.
@XMFunction(name = "add_part")
public void addPart(BufferObject part) {
if (!buffer.contains(part) && buffer.size() + 1 <
capacity) {
buffer.add(part);
reportOutcome("Part Added");
}
}
![Page 19: Project FoX: A Tool That Offers Automated Testing Using a Formal Approach](https://reader034.fdocuments.us/reader034/viewer/2022042816/559345461a28abbb728b4710/html5/thumbnails/19.jpg)
19
BUFFER CASE STUDY – EXECUTING FOX
![Page 20: Project FoX: A Tool That Offers Automated Testing Using a Formal Approach](https://reader034.fdocuments.us/reader034/viewer/2022042816/559345461a28abbb728b4710/html5/thumbnails/20.jpg)
20
BUFFER CASE STUDY – EXECUTING FOX (implanted error)if (!buffer.contains(part) && buffer.size() + 1 <
capacity) {
buffer.add(part);
capacity++;
reportOutcome("Part Added");
}
![Page 21: Project FoX: A Tool That Offers Automated Testing Using a Formal Approach](https://reader034.fdocuments.us/reader034/viewer/2022042816/559345461a28abbb728b4710/html5/thumbnails/21.jpg)
21
BUFFER CASE STUDY – GENERATED TEST CASES> Tests report the sequence of inputs used for the specific scenario, the sequence
of expected outputs and the actual output.
> Outcome is reported to the user via the usual JUnit red / green notifications.
<tests> <test testID=”1”> <input>[ itemId: 187 type: Nekrataal ]</input> <expectedOutput>[ Part Added ]</expectedOutput> <output>[ Part Added ]</output> </test> <test testID=”2”> <input>[ itemId: 17 type: Planeswalker, itemId: 20 type: Dragon]</input> <expectedOutput> [ Part Added, Part Added – Become Full ] </expectedOutput> <output>[ Part Added, Part Added – Become Full ]</output> </test></tests>
![Page 22: Project FoX: A Tool That Offers Automated Testing Using a Formal Approach](https://reader034.fdocuments.us/reader034/viewer/2022042816/559345461a28abbb728b4710/html5/thumbnails/22.jpg)
22
CONCLUSIONS AND FUTURE WORK
> FoX enables developers to leverage the already proven theories for formal testing.
> Provides a fully automated testing process, ranging from complete test set generation (satisfying some design for test conditions), to test preparation and execution.
> Operates on any Java based software system, being transparent to it's underlining technologies.
> Provides complete positive and complete negative testing.
> Nest steps:
– Thorough evaluation.
– An additional tool to make the specification step easier and closer to the developer, aiming to “hide” the formality as much as possible.
– NetBeans and Eclipse integration.
– A standalone X-Machine IDE providing additional related functionalities.
– Branch out to other languages and frameworks (eg. C# and .NET).
![Page 23: Project FoX: A Tool That Offers Automated Testing Using a Formal Approach](https://reader034.fdocuments.us/reader034/viewer/2022042816/559345461a28abbb728b4710/html5/thumbnails/23.jpg)
23
BIBLIOGRAPHY
> S. Eilenberg, Automate, Languages and Machines, Vol. A. Academic Press, London, 1974.
> M. Holcombe, “X-Machines as a basis for dynamic system specification,” Software Engineering Journal, vol. 3(2), pp. 69-76, 1988.
> F. Ipate and M. Holcombe, “Specification and Testing using Generalized Machines: a Presentation and a Case Study,” Softw. Test. Verif. Reliab, vol. 8, pp. 61-81, 1998.
> M. Holcombe and F. Ipate, Correct Systems: Building a Business Process Solution. Springer, Applied Computing Series, November 1998.
> G. Eleftherakis and A. Cowling, “An Agile Formal Development Methodology,” in 1st South Eastern European workshop on Formal Methods (SEEFM 03), (Thessaloniki), pp. 36-47, Nov. 2002. Agile Formal Methods: Practical, Rigorous Methods for a changing world.
> P. Kefalas, G. Eleftherakis, and E. Kehris, “Communicating X-Machines: a practical approach for formal and modular specification of large systems,” Information and Software Technology, vol. 45, pp. 269-280, Apr. 2003.