Test Protocol Minesweeper - isy.liu.se · E-mail: [email protected] Group members Name...

Test ProtocolMinesweeper

Version 1.0

Editor: Mattias AndreassonDate: December 13, 2017

Status

Reviewed Elin Näsholm 2017-12-01Approved Martin Lindfors 2017-12-01

Course name: Automatic Control - Project Course Course code: TSRT10Project group: Smaugs E-mail: [email protected]: Minesweeper Document name: Test plan

Minesweeper II

Project Identity

Linköping University, Department of Electrical Engineering (ISY)Autumn 2017

Client: Martin Lindfors, Linköping UniversityPhone: +46 13 28 13 65, E-mail: [email protected]

Customer: Torbjörn Crona, Saab DynamicsE-mail: [email protected]

Course Examiner: Daniel Axehill, Linköping UniversityPhone: +46 13 28 40 42, E-mail: [email protected]

Project Manager: Hampus AnderssonAdvisors: Per Boström-Rost, Linköping University

E-mail: [email protected] Ekelund, Saab DynamicsE-mail: [email protected] Reizenstein, Saab DynamicsE-mail: [email protected]

Group members

Name LiU-id Phone number ResponsibilityAndreas Hägglund andha796 072-713 38 70 Head of HardwareAndreas Lundgren andlu901 070-022 56 44 Head of DesignElin Näsholm elina044 073-094 94 03 Head of DocumentationFredrik Gustafsson fregu856 070 578 63 48 Head of Slam ImplementationFredrik Tormod freto995 073-775 05 36 Head of Control and Route PlanningHampus Andersson haman657 073-675 93 56 Project ManagerJonathan Jerner jonje173 070 296 61 50 Head of SoftwareMattias Andreasson matan461 070-822 53 67 Head of Testing

Group E-mail: [email protected]: TBA

Mail to each individual can be sent to "LiU-id"@student.liu.se.


Minesweeper III

Contents

1 Introduction 1

2 Definitions 2

3 Guidelines 3

3.1 Test Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

3.2 Test Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

4 Test sessions 4

4.1 Balrog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

4.1.1 SLAM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

4.1.2 Navigation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

4.1.3 Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

4.1.4 GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

4.2 Sauron . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

4.2.1 Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

4.2.2 Tracking & Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

4.2.3 GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24


Minesweeper IV

Document History

Version Date Changes made Sign Reviewer0.1 6/11-17 First draft. - FG0.2 28/11-17 Second draft - EN0.3 30/11-17 Third revision - EN0.4 1/12-17 Forth revision - HA


Minesweeper 1

1 Introduction

The test protocol is a document that verifies that all tests are performed and that theresults are documented. The presented tests are for the functional requirements, whichare listed in the Test Plan [1].


Minesweeper 2

2 Definitions

AprilTags A matrix of bar codes used to identify predefined objects.Balrog The ground vehicle aimed for minesweeping used in the

project.EKF Extended Kalman Filter.GUI Graphical user interface.HAL Hardware abstraction layer.MAV-link Micro Air Vehicle Link.Mine An object or image on the ground acting as a mine.Object Any physical item in the testing area such as a mine, an

obstacle or a wall.Obstacle An object in the test area that acts as an obstacle for the

Balrog.ROS nodes Units of code that can be run independently of each other

and communicate via ROS topics.ROS topics Named data channels that ROS nodes can send messages

to by publishing on the topic, and read messages from bysubscribing on the topic.

ROS Robot Operating System. A set of open source softwarelibraries and tools enabling communication between differ-ent modules in robotic systems.

Route A set of way-points created by the user or the platformitself.

rqt Qt-based framework for GUI development in ROS.RViz Visualization tool for ROS.Sauron The drone used in the project for collaborating with the

ground vehicle.Search area The area that has not yet been searched for mines by Bal-

rog.Searched area The area that has been visited by Balrog.SLAM Simultaneous localization and mapping.TCP Transmission Control Protocol.Test area An indoor area created in Visionen with fixed position on

the boundaries and obstacles.Test locations Locations where Balrog is stopped in the test route and

the functional requirements for positioning and mappingare evaluated.

Test map A fully known map for which the project group is providedwith an "optimal" exploration route.

Test point In a test location, this is a point on an obstacle to evaluatethe functional requirements on mapping accuracy.

Test route A route in the test area for manual drive with the handcontroller to evaluate the functional requirements for theSLAM algorithm.

The project group The group of students involved in the project.UAV Unmanned Aerial Vehicle.UDP User Datagram Protocol.User The human interacting with the system in any way.


Minesweeper 3

3 Guidelines

3.1 Test Protocol

A test protocol shall be written by the person responsible for the test. The test protocolis an extension of the description in the Test Plan [1].

3.2 Test Environment

As described in the Definitions, Section 2, a test area will be created for testing Balrog.The test area will be created as described in the Requirement description [2]:

Req. no. 3 Original The test area shall have a size of 5 by 5 meters. 1Req. no. 4 Original The test area shall have at least 5 obstacles. 1Req. no. 5 Original The test route shall include at least 4 turns. 1Req. no. 6 Original The test route shall be at least 20 meters long. 1Req. no. 7 Original There shall be at least 3 test locations in the test route. 1Req. no. 8 Original Each test location shall have at least two test points on

obstacles.1

The test area used is seen in Figure 1, where post-it notes on the floor mark test locationsin the test route, and post-it notes on obstacle walls mark test points. A visualization ofthe complete test route is seen in Figure 2.

Figure 1: The test area.


Minesweeper 4

Figure 2: The test route displayed in RViz.

4 Test sessions

The tests will be described as seen below.

Test no. xDescription: Brief description of what the test does

Requirement no. tested: Number in the requirement specification [2]Requirement description: The description copied from the requirement specification

Test description: A detailed description on how the test will be performed and howevaluate the requirement to be tested.

Responsible for test: A person responsible for the test protocolOther participants: Other group members that will be present during the testWeek of test: Week in which the test was performed.

Test protocolTest result (pass/fail): Result if the test passed or failed.Comment on test result: Description of how the test went.Test approved by: Initials of the group member which has written the test protocol.

A shorter version to the name of the members in the project group is introduced.

• Andreas Hägglund - AH• Andreas Lundgren - AL• Elin Näsholm - EN


Minesweeper 5

• Fredrik Gustafsson - FH

• Fredrik Tormod - FT

• Hampus Andersson - HA

• Jonathan Jerner - JJ

• Mattias Andreasson - MA

4.1 Balrog

4.1.1 SLAM

Test no. 1Description: SLAM position accuracy, manual drive

Requirement tested: 18a-cRequirement description: When driving the test route, the estimated position shall have an

accuracy of x m

Test description: Drive the test route and stop at the test locations. The estimatedcoordinates in the SLAM algorithm is noted and compared to itsactual position. The accuracy shall be below the required accuracyin all test locations to fulfill that requirement and is calculated as:

Accuracy > ||EstimatedPos−ActualPos||

Responsible for test: MAOther participants: FGWeek of test: Week 44

Test protocolTest result (pass/fail): PassComment on test result: Balrog was driven the test route and the distance to the start-

ing point was measured in 2 locations along the route and whenreaching close to the starting point again. A measurement tapewas used to measure the distance and compared to the coordi-nate in the topic estimated_pose, which gives the estimated posefrom the SLAM algorithm. The formula above was evaluated inall three points and gave the accuracies: 0.0583, 0.0424 and 0.051meters, which fulfill requirement 18 a and b. The level 3 require-ment on 0.05 m accuracy was fulfilled in only 1 test location. Thelevel 2 requirement was however fulfilled in all three test locations.

Test approved by: MA


Minesweeper 6

Test no. 2Description: SLAM angular accuracy, manual drive

Requirement tested: 19a-cRequirement description: When driving the test route, the estimated angle shall have an

accuracy of x degrees.

Test description: Drive the test route and stop at the test locations. The estimatedangle from the SLAM algorithm is noted and compared to itsactual angle. The accuracy shall be below the required accuracyin all test locations to fulfill that requirement and is calculated as:

Accuracy > |EstimatedAngle−ActualAngle|

Responsible for test: MAOther participants: FG and ENWeek of test: Week 44

Test protocolTest result (pass/fail): PassComment on test result: The angle of the robot was slightly difficult to measure, so a simple

solution using cardboard pieces cut into specific angles was usedto compare the angle of the robot to the walls of the test area, seeFigure 3. The estimated angle errors in the three test locationswere: 1.1, 2.3 and 1.0 degrees. All levels of requirements arefulfilled, where the level 3 one was 2.5 degrees.


Figure 3: Visualization of the method used to measure the robot angle.


Minesweeper 7

Test no. 3Description: SLAM mapping, manual drive

Requirement tested: 20a-cRequirement description: The system shall detect and map fixed objects with an accuracy

of x % relative to the distance to the object, when driving the testroute.

Test description: Drive the test route and stop at the test locations. Measure thedistance to the test point in the map given by SLAM as well as thereal distance. The accuracy shall be below the required accuracyfor all test points to fulfill that requirement and is calculated as:

Accuracy > |(SLAMDistance− RealDistance)/RealDistance|

Responsible for test: MAOther participants: FG and ENWeek of test: Week 44

Test protocolTest result (pass/fail): PassComment on test result: The accuracy was evaluated in two test points for all three test lo-

cations. This gave six accuracies: 0.72, 2.86, 0, 1.02, 0.37 and 2.34percent. The most strict level 3 requirement is 5 percent, whichwas fulfilled for all points and thus requirement 20 a-c fulfilled.



Minesweeper 8

4.1.2 Navigation

Test no. 4Description: Balrog avoid getting stuck

Requirement tested: 27Requirement description: Balrog shall avoid getting stuck by re-optimizing the route.

Test description: A large object should be placed in front of the Balrog while run-ning so it stops. It should then notice that the goal point hasn’tbeen reached and turn into another direction.

Responsible for test: FTOther participants: EN, FGWeek of test: Week 43

Test protocolTest result (pass/fail): PassComment on test result: Figure 4, 5 and 6 shows the process of rerouting when an obstacle

appears in a simulated environment. Figure 4 shows the intendedgoal position for the current pathway. Figure 5 shows the navi-gation reacting when obstacles appear in the current pathway. Awarning message is generated: "current path is NOT ok!". Atlast, Figure 6 shows how the navigation reroutes and generates anew pathway. This pathway is illustrated with a green line. Thesame thing happens when driving in a real environment. (SeeFigure 7 and 8.)

Test approved by: FG

Figure 4: In the simulator, Balrog is trying to reach the area circled in red.


Minesweeper 9

Figure 5: During the current pathway an obstacle appears. Making the current path notvalid. This is seen in the terminal window in the picture.

Figure 6: New goal node is set and a new path is generated, as shown by the green line.


Minesweeper 10

Figure 7: An obstacle placed in the path of Balrog.

Figure 8: The GUI when the obstacle is detected returning ”current path is NOT ok!”.


Minesweeper 11

Test no. 5Description: Covering tiles, entire area

Requirement tested: 28Requirement description: When navigating an area free from obstacles, Balrog shall cover

at least 95 % of the area.

Test description: Run Balrog in an empty test area. The platform should visit 95%of the grid cells on the GUI map. This does not include area whichlies within a safety distance from the outer edges.

0.95 <NumberOfGridCellsVisited

TotalArea− SafetyAreaFromEdges

Responsible for test: FTOther participants: FG, ENWeek of test: Week 43

Test protocolTest result (pass/fail): PassComment on test result: When driving in an area of 2x10 m, Balrog is covering the whole

area (100%) except for the safety margin of 0.5 m to the edges.The test environment is seen in Figure 9 and the resulting map inFigure 10.

Test approved by: EN

Figure 9: The test environment for test no. 5.


Minesweeper 12

Figure 10: The area visited shown in yellow.

Test no. 6Description: Exploring away from obstacles

Requirement tested: 29a-cRequirement description: When navigating the test area, Balrog shall explore at least 95 %

of the map that is more than x meters from any obstacle.

Test description: Run Balrog in a test area with 5 obstacles placed randomly. Theplatform should visit 95% of the grid cells on the GUI map xmeters from any obstacle. This does not include area which lieswithin a safety distance from the outer edges.

0.95 <NumberOfGridCellsVisited

TotalArea− SafetyArea−AreaForObstacles

Responsible for test: FTOther participants: EN, FGWeek of test: Week 44

Test protocolTest result (pass/fail): PassComment on test result: When driving in the test area, Balrog is covering the whole area

(100%), taking into consideration the safety margin of 0.5 m toobstacles and edges. The test environment is seen in Figure 11and the resulting map in Figure 12.

Test approved by: EN


Minesweeper 13

Figure 11: The test environment in test no. 6.

Figure 12: The map when trying to cover an area with obstacles.


Minesweeper 14

Test no. 7Description: Extra distance acceptable exploring

Requirement tested: 30a-cRequirement description: When mapping and covering the test map in simulation, Balrog

shall cover at least 95 % of the area that is no more than 0.5meters from any obstacle, by following a route that is no morethan x % longer than the shortest known route.

Test description: Run Balrog in mapping and covering mode, in an area with threeobstacles. Measure how long the actual route is becoming andcompare with a approximation of the optimal route for the samemap.

Ratio =LengthTransferred− LengthOptimalRoute

LengthOptimalRoute

Responsible for test: FTOther participants: ENWeek of test: Week 44

Test protocolTest result (pass/fail): PassComment on test result: The considered optimal path is measured to 62.2 m, compared to

80.4 m when driving in autonomous mode. This gives a path 29% longer. See Figure 13, 14 and 15

Test approved by: EN, FG

Figure 13: The approximated optimal route for the map where red lines are followed onceand blue twice.


Minesweeper 15

Figure 14: Figure of the map when measuring the route given above.

Figure 15: The path when driving autonomously in the same map.


Minesweeper 16

4.1.3 Control

Test no. 8Description: Deviation from intended route and goal node accuracy.

Requirement tested: 34 and 35Requirement description: The Balrog shall not deviate more than 0.1 m from the intended

route provided by the navigation module and the Balrog shallfinish within 0.2 m of the intended goal node.

0.1 > ||CurrentPosition−DesiredPosition||

0.2 > ||GoalnodeCoord−ActualCoord||

Test description: Balrog is given a route consisting of eight points. At each pointthe system will be paused and the current estimated position,outputted by SLAM, will be read. The difference from the desiredposition should not be more than 0.1 m. The last point in thisroute is viewed as the goal node.

Responsible for test: FTOther participants: ENWeek of test: Week 45

Test protocolTest result (pass/fail): PassComment on test result: The test tested both deviation from intended route and distance

to goal node. Both these conditions were met with a large margin,as seen in Table 1.

Test approved by: FT

Desired x-pos. Actual x-pos Desired y-pos Actual y-pos Euclidean distance deviation [m]0 0.00743 1 0.988 0.01414 3.985 1 0.991 0.01754 3.995 0 0.0293 0.02978 7.977 0 -0.00868 0.02468 8.009 1 0.970 0.03134 4.041 1 1.0244 0.04774 4.011 0 -0.0186 0.02160 0.0341 0 0.00995 0.0355

Table 1: Observed data in test no. 8.


Minesweeper 17

4.1.4 GUI

Test no. 9Description: GUI functionality, binary

Requirement tested: 40Requirement description: The GUI shall function on the Base station.

Test description: Open the GUI and check the following:

• GUI displays sensor data

• GUI displays SLAM data and map

• GUI displays visited/non-visited nodes

Responsible for test: MAOther participants: ENWeek of test: Week 46

Test protocolTest result (pass/fail): PassComment on test result: All requirements fulfilled, see Figure 16 with sensor data to the

right and the map including visited nodes to the left.Test approved by: EN

Figure 16: Image of the GUI.


Minesweeper 18

Test no. 10Description: GUI data handling, binary

Requirement tested: 41Requirement description: The data presented in the GUI shall be displayed in less than 1

second from real time.

Test description: Open the GUI and check the following:

• GUI updates sensor data within 1 second

• GUI updates SLAM data within 1 second

Responsible for test: FGOther participants: MAWeek of test: Week 46

Test protocolTest result (pass/fail): PassComment on test result: When Balrog is turned the GUI data updates within 1 s.Test approved by: FG

Test no. 11Description: GUI send commands, binary

Requirement tested: 42Requirement description: From the GUI the user shall be able to send commands to control

the platform.

Test description: Open the GUI and check that the platform responds to any inputdata given.

Responsible for test: ENOther participants:Week of test: Week 46

Test protocolTest result (pass/fail): PassComment on test result: The GUI needs to send a start command for the platform to begin

mapping.Test approved by: EN


Minesweeper 19

4.2 Sauron

4.2.1 Detection

Test no. 12Description: Video stream from UAV

Requirement tested: 45Requirement description: The system shall stream the video feed to the base station in real

time.

Test description: A live camera feed is transmitted from Sauron to the basestation.Sauron shall be able to be at a distance of at least 15 m from thebasestation and the quality of the feed should be at least 410x308and 10 fps.

Responsible for test: AHOther participants: HA, JJ or ALWeek of test: Week 41

Test protocolTest result (pass/fail): PassComment on test result: Since Sauron is equipped with a camera there are several ways

to get a camera feed. In Figure 17 a screenshot of a camera feedwith Apriltag detection is displayed. A resolution of 600x600 isused with a framerate of 15 fps. The delay of the video feed isestimated to be less than 0.3 seconds in the worst case.

Test approved by: JJ

Figure 17: Screenshot of camera feed with Apriltag detection


Minesweeper 20

Test no. 13Description: Detection of object

Requirement tested: 46a-cRequirement description: Sauron shall when operating at a constant height of 3 m, estimate

its relative position in 2D to a known object with an accuracy ofx m.

Test description: When manually operating at a constant height of 3m the positionestimate of Sauron relative an AprilTag is obtained from the de-tection module. The actual position of Sauron relative the April-Tag is measured by hand and the following requirement is to befulfilled:

Accuracy > ||EstimatedPosition−ActualPosition||

This test is then repeated for different positions, always with aconstant height of 3m, for Sauron relative the Apriltag.

Responsible for test: AHOther participants: HA or ALWeek of test: Week 44 and 45

Test protocolTest result (pass/fail): PassComment on test result: Sauron was held at a 3 m (measured by hand with measuring tape)

distance from a surface with an Apriltag placed on the surface. Bylooking at the camera feed from Sauron a square of roughly 3x3m was marked on the surface. Placing the Apriltag within thissquare area the worst result from the detection module was barelydeviating noticeable from the measured distance in X, Y and Zdirections. Therefore, the detection module is far more accuratethan 0.1 m.

Test approved by: JJ


Minesweeper 21

4.2.2 Tracking & Control

Test no. 14Description: Localizing Balrog

Requirement tested: 52a-bRequirement description: The system shall be able to locate the stationary Balrog with an

accuracy of x m.

Test description: An AprilTag is attached on top of the Balrog. Sauron is thenmanually positioned at different 3D positions relative Balrog andthe localization module is applied. The actual position of Sauronrelative Balrog is measured by hand and the following requirementis to be fulfilled:


Responsible for test: AHOther participants: HA or ALWeek of test: Week 45

Test protocolTest result (pass/fail): PassComment on test result: Sauron was manually placed at different 3D locations relative to

the stationary Apriltag placed on the ground. The output dis-tances from the detection module is within centimeters from mea-sured values (measured by hand with measuring tape). Therefore,the requirement of 0.1 m accuracy is passed with distinction.

Test approved by: HA


Minesweeper 22

Test no. 15Description: Hold position in air

Requirement tested: 53Requirement description: Sauron should be able to move to and maintain a position hovering

at constant height. The position should not deviate more than 1m in any direction.

Test description: Sauron is set to hover at a specific position. An AprilTag is placeddirectly below Sauron. The position estimate of Sauron relativethe Apriltag is obtained from the detection module and the actualposition is measured by hand. The following requirement is to befulfilled:

1 > ||EstimatedPosition−ActualPosition||


Test protocolTest result (pass/fail): PassComment on test result: The measurements of Sauron relative the AprilTag are measured

using the AprilTag positioning data and the known error of thisdata. During 30 seconds Sauron was set to hover above the April-Tag and did not deviate more than 1 m.

Test approved by: AH


Minesweeper 23

Test no. 16Description: Object tracking

Requirement tested: 54a-bRequirement description: The UAV should be able to track a known object in a 2D space

while the height of the UAV is kept constant. The accuracy of thetracking shall be within x m.

Test description: Sauron is set to hover at a constant height and an AprilTag isplaced on the ground. Sauron is then set to hover straight abovethe AprilTag as the AprilTag is moved to different locations andtracking is achieved. The detection module will return a positionof Sauron relative the AprilTag. A ruler will move along withthe AprilTag and a person will observe the system sideways andestimate the length of which Sauron lags behind the AprilTag.The following requirement is to be fulfilled during the tracking:



Test protocolTest result (pass/fail): FailComment on test result: Sauron does deviate more than 0.5 m when trying to hover above

the AprilTag as the AprilTag is being moved around (in the testit was moved by Balrog). To measure the deviation the AprilTagdetection algorithm is used, taking in consideration its error inposition estimates which has been established in previous tests.Saurons position relative the AprilTag is as high as 1.5 m some-times and the test is therefore failed.

Test approved by: AH


Minesweeper 24

Figure 18: Data from camera feed, example image

4.2.3 GUI

Test no. 17Description: GUI with live video

Requirement tested: 69Requirement description: The GUI shall include a live stream of the video from Sauron.

Test description: Video with the same resolution and framerate captured by thecamera onboard Sauron will be transmitted to the GUI on theBaseStation.


Test protocolTest result (pass/fail): PassComment on test result: The Apriltag detection module opens a window displaying video

feed with detected Apriltags highlighted. The video stream is ofthe same resolution and framrate as captured by the Raspberrycamera. See Figure 18.

Test approved by: HA


Minesweeper 25

Test no. 18Description: GUI position highlight

Requirement tested: 70Requirement description: The GUI shall highlight the position of Balrog (if applicable) in

the video.

Test description: Video with the same resolution and framerate as captured by thecamera onboard Sauron will be shown in the GUI on the BaseS-tation. AprilTags shall be highlighted in the feed.


Test protocolTest result (pass/fail): PassComment on test result: The Apriltag detection module starts a window with the camera

feed. The detected Apriltags are highlighted, see Figure 18.Test approved by: HA


Minesweeper 26

References[1] E. Näsholm F. Gustafsson F. Tormod H. Andersson J. Jerner M. Andreasson A. Häg-

glund, A. Lundgren. Test plan minesweeper. Unpublished technical document, 2017.

[2] E. Näsholm F. Gustafsson F. Tormod H. Andersson J. Jerner M. Andreasson A. Häg-glund, A. Lundgren. Requirement specification minesweeper. Unpublished technicaldocument, 2017.


Test Protocol Minesweeper - isy.liu.se · E-mail: [email protected] Group members Name...

Documents

Transcript of Test Protocol Minesweeper - isy.liu.se · E-mail: [email protected] Group members Name...