Automated Testing · Automated Testing - EPICS Option 1.Integration Test. • Create a complete...
Transcript of Automated Testing · Automated Testing - EPICS Option 1.Integration Test. • Create a complete...
![Page 1: Automated Testing · Automated Testing - EPICS Option 1.Integration Test. • Create a complete test system that includes hardware. – The expense precludes this for all software.](https://reader035.fdocuments.us/reader035/viewer/2022081407/604b4cccc2c4a90cf3516009/html5/thumbnails/1.jpg)
Automated Testing
Presentation to EPICS Collaboration Meeting 2007 @ DESY
Paul Gibbons PhDDiamond Light Source Ltd.
![Page 2: Automated Testing · Automated Testing - EPICS Option 1.Integration Test. • Create a complete test system that includes hardware. – The expense precludes this for all software.](https://reader035.fdocuments.us/reader035/viewer/2022081407/604b4cccc2c4a90cf3516009/html5/thumbnails/2.jpg)
Automated Testing – Why?
• Code Guru– Edit– Compile– Link– Publish
• No testing needed.(Also turn off all compiler warnings as I never write incorrect
code.)
![Page 3: Automated Testing · Automated Testing - EPICS Option 1.Integration Test. • Create a complete test system that includes hardware. – The expense precludes this for all software.](https://reader035.fdocuments.us/reader035/viewer/2022081407/604b4cccc2c4a90cf3516009/html5/thumbnails/3.jpg)
Automated Testing – Why?
• We live in the real world
![Page 4: Automated Testing · Automated Testing - EPICS Option 1.Integration Test. • Create a complete test system that includes hardware. – The expense precludes this for all software.](https://reader035.fdocuments.us/reader035/viewer/2022081407/604b4cccc2c4a90cf3516009/html5/thumbnails/4.jpg)
Automated Testing – Why?
• Others will need to change it.
• Can you trust them not to mess up?
![Page 5: Automated Testing · Automated Testing - EPICS Option 1.Integration Test. • Create a complete test system that includes hardware. – The expense precludes this for all software.](https://reader035.fdocuments.us/reader035/viewer/2022081407/604b4cccc2c4a90cf3516009/html5/thumbnails/5.jpg)
Automated Testing – Why?InitialRequirements
Develop
Build 0.1 Test 0.1Build 0.2Time
Test 0.2Release 0.2
DevelopNew Requirement Build 0.3
Test 0.3 Release 0.3
![Page 6: Automated Testing · Automated Testing - EPICS Option 1.Integration Test. • Create a complete test system that includes hardware. – The expense precludes this for all software.](https://reader035.fdocuments.us/reader035/viewer/2022081407/604b4cccc2c4a90cf3516009/html5/thumbnails/6.jpg)
Automated Testing – Why?
• Manual testing is:– Expensive– Time consuming– Error prone.
![Page 7: Automated Testing · Automated Testing - EPICS Option 1.Integration Test. • Create a complete test system that includes hardware. – The expense precludes this for all software.](https://reader035.fdocuments.us/reader035/viewer/2022081407/604b4cccc2c4a90cf3516009/html5/thumbnails/7.jpg)
Automated Testing – Why?
• Automated Testing:– Repeatable. – Running the test requires little human intervention.– Can encapsulate the requirements better than a
requirements document – is always up to date.– Tests not only the code you know you have changed
but also the code you think you have not.– Expensive up front investment.
![Page 8: Automated Testing · Automated Testing - EPICS Option 1.Integration Test. • Create a complete test system that includes hardware. – The expense precludes this for all software.](https://reader035.fdocuments.us/reader035/viewer/2022081407/604b4cccc2c4a90cf3516009/html5/thumbnails/8.jpg)
Automated Testing – Why?
Late night call out to modify Fred’s module.• Make modification.• Automated tests show no problems.• Manually test new feature.• Go home confident ☺.
Next day review change with Fred and add test to automated tests.
![Page 9: Automated Testing · Automated Testing - EPICS Option 1.Integration Test. • Create a complete test system that includes hardware. – The expense precludes this for all software.](https://reader035.fdocuments.us/reader035/viewer/2022081407/604b4cccc2c4a90cf3516009/html5/thumbnails/9.jpg)
Automated Testing – Why?
• New requirement calls for re-factoring– Before adding any new feature you gradually
refactor the code. – At each point run automated test.– At end of refactoring you have confidence that
the software still behaves correctly and you are ready to add new features.
![Page 10: Automated Testing · Automated Testing - EPICS Option 1.Integration Test. • Create a complete test system that includes hardware. – The expense precludes this for all software.](https://reader035.fdocuments.us/reader035/viewer/2022081407/604b4cccc2c4a90cf3516009/html5/thumbnails/10.jpg)
Automated Testing – Why?
• Testing on its own is not a panacea.– Different defect-detection techniques
• Reviews, formal/informal design/code inspections, personal desk checking, prototyping, unit test, integration test, system test, beta tests
– Defect Detection Rates:• None above 75%• Testing gives 30-40%
– A combination is required to get > 95%.
![Page 11: Automated Testing · Automated Testing - EPICS Option 1.Integration Test. • Create a complete test system that includes hardware. – The expense precludes this for all software.](https://reader035.fdocuments.us/reader035/viewer/2022081407/604b4cccc2c4a90cf3516009/html5/thumbnails/11.jpg)
Automated Testing – Why?
• Many other advantages of thinking about testing up front.
• See books on Test Driven Development.
![Page 12: Automated Testing · Automated Testing - EPICS Option 1.Integration Test. • Create a complete test system that includes hardware. – The expense precludes this for all software.](https://reader035.fdocuments.us/reader035/viewer/2022081407/604b4cccc2c4a90cf3516009/html5/thumbnails/12.jpg)
Automated Testing – How
Module to Be tested
Test Script
Interface being tested.
Report. OK or Failure?
Environment in which moduleExecutes.
![Page 13: Automated Testing · Automated Testing - EPICS Option 1.Integration Test. • Create a complete test system that includes hardware. – The expense precludes this for all software.](https://reader035.fdocuments.us/reader035/viewer/2022081407/604b4cccc2c4a90cf3516009/html5/thumbnails/13.jpg)
Automated Testing – EPICSTypes of IOC software at Diamond.
VME Register Interface Serial Device Interface
Device Independent Records – e.g. Motor
Frameworks: asyn, streamGensub records with logic built into C function Groups of ‘standard’ records that
together provide functionality – aperture slits, diffractometer table.
Sequences
![Page 14: Automated Testing · Automated Testing - EPICS Option 1.Integration Test. • Create a complete test system that includes hardware. – The expense precludes this for all software.](https://reader035.fdocuments.us/reader035/viewer/2022081407/604b4cccc2c4a90cf3516009/html5/thumbnails/14.jpg)
Automated Testing – EPICS
Current testing is 99% manual.
Normally requires a complete system with hardware.
![Page 15: Automated Testing · Automated Testing - EPICS Option 1.Integration Test. • Create a complete test system that includes hardware. – The expense precludes this for all software.](https://reader035.fdocuments.us/reader035/viewer/2022081407/604b4cccc2c4a90cf3516009/html5/thumbnails/15.jpg)
Automated Testing - EPICS
Option 1.Integration Test. • Create a complete test system that
includes hardware.– The expense precludes this for all software.– Testing is only possible once hardware is
availableOption 2. Isolation
– Requires simulation of environment.
![Page 16: Automated Testing · Automated Testing - EPICS Option 1.Integration Test. • Create a complete test system that includes hardware. – The expense precludes this for all software.](https://reader035.fdocuments.us/reader035/viewer/2022081407/604b4cccc2c4a90cf3516009/html5/thumbnails/16.jpg)
Automated Testing - EPICSYou want to test the logic of code.
Use the language of the interface.
A test environment should :– not impose any constraints on the interface– be simple and quick to construct.
![Page 17: Automated Testing · Automated Testing - EPICS Option 1.Integration Test. • Create a complete test system that includes hardware. – The expense precludes this for all software.](https://reader035.fdocuments.us/reader035/viewer/2022081407/604b4cccc2c4a90cf3516009/html5/thumbnails/17.jpg)
Automated Testing – IOC
Logical Units that interface via records:
•Accompanying Test IOC•Channel access script to perform the tests and generate a report.
What about the environment it requires?Simulation mode to be used in Test IOCs of other modules.
![Page 18: Automated Testing · Automated Testing - EPICS Option 1.Integration Test. • Create a complete test system that includes hardware. – The expense precludes this for all software.](https://reader035.fdocuments.us/reader035/viewer/2022081407/604b4cccc2c4a90cf3516009/html5/thumbnails/18.jpg)
Automated Testing – IOCC functions that can be built for Linux e.g. genSub
systems:Use CPPUnit
![Page 19: Automated Testing · Automated Testing - EPICS Option 1.Integration Test. • Create a complete test system that includes hardware. – The expense precludes this for all software.](https://reader035.fdocuments.us/reader035/viewer/2022081407/604b4cccc2c4a90cf3516009/html5/thumbnails/19.jpg)
Automated Testing – Device Simulation
Module to Be tested
Device Simulation
Test Script
Interface being tested.
Back doorReport. OK or Failure?
![Page 20: Automated Testing · Automated Testing - EPICS Option 1.Integration Test. • Create a complete test system that includes hardware. – The expense precludes this for all software.](https://reader035.fdocuments.us/reader035/viewer/2022081407/604b4cccc2c4a90cf3516009/html5/thumbnails/20.jpg)
Automated Testing – Device Simulation
Serial( RS232/IP) -• Hardware loopback• asyn IP client• Java – COSYLAB• asynPort that runs
Python
Record Simulation Mode
VME Register – simulation device drivers.
![Page 21: Automated Testing · Automated Testing - EPICS Option 1.Integration Test. • Create a complete test system that includes hardware. – The expense precludes this for all software.](https://reader035.fdocuments.us/reader035/viewer/2022081407/604b4cccc2c4a90cf3516009/html5/thumbnails/21.jpg)
Automated Testing – Plan• Many options. Need to agreed on standards for:
– Test language – Should match interface being tested. For Channel Access use Python or Jython.
– How to build the test.– How to run the test.– Reporting success and failure. xUnit XML format
Provide parsers to convert from other formats
Current work:PyUnit + serial device simulation + use of
record simulation mode
![Page 22: Automated Testing · Automated Testing - EPICS Option 1.Integration Test. • Create a complete test system that includes hardware. – The expense precludes this for all software.](https://reader035.fdocuments.us/reader035/viewer/2022081407/604b4cccc2c4a90cf3516009/html5/thumbnails/22.jpg)
Automated Testing - References
• Code Complete, Second Edition by Steve McConnell. ISBN-10: 0-7356-1967-0
• Various work by Kent Beck a founder of the Agile Manifesto.