- 1 -

A Motivational Exercise for Software Testing Courses

The 5th Workshop on the Teaching of Software Testing (WTST 5), February 3 - 5, 2006, Florida Institute of Technology, Melbourne, Florida

Chang Liu

School of Electrical Engineering and Computer Science

Russ College of Engineering and Technology Ohio University

Athens, Ohio 45701, U.S.A.

http://www.ent.ohiou.edu/~liuc liuc@ohio.edu

Abstract Before taking a software testing course or a software engineering course with a software testing component, most computer science students have a rough idea of what software testing is, although many of them are not aware of the sophistication of software testing theories and techniques and have misconceptions about software testing. Many students do not appreciate the value of good software testing practices and the career outlook of good testers. To help motivate students, we designed a web application testing exercise to demonstrate the challenge of software testing, illustrate various aspects of software testing, and to leave a lasting impression in students’ minds on the value of software testing. In addition, various elements of this exercise can be used as starting points in later lectures that cover software testing concepts such as white-box testing, black-box testing, and usability testing.

1. Motivation Software testing is not always a popular subject among Computer Science students. Before taking a software testing course or a software engineering course with a software testing component, most computer science students have a rough idea of what software testing is. However, many of them are not aware of the sophistication of software testing theories and techniques, and may have misconceptions about software testing. For example, some of them think that software testing is a less challenging task than programming and therefore software testers are paid less than programmers, which, to an extent, reflects the reality in the industry. The software industry, plagued by numerous quality and security issues, has in the past paid less attention to product qualities than other mature industries. What these students fail to see is that effective and efficient software testing is just as challenging and valuable as programming.

- 2 -

Such shallow understanding and misconceptions of software testing tend to lead many students to under-appreciate the value of good software testing practices and the career outlook of good testers. To help motivate students, we designed a software testing exercise to demonstrate the challenge of software testing, illustrate various aspects of software testing, and to leave a lasting impression in students’ minds on the value of software testing. To catch students by surprise and to increase the impact of this exercise, we chose an ordinary website that students were familiar with as the subject-under-test. 2. The Exercise The motivational exercise consists of two parts. The first part is a testing assignment of the website at http://www.ent.ohiou.edu/~liuc/, as shown in Figure 1. Students are not allowed to view the source code of this website in this part of the exercise. The second part is a testing assignment of the same website. This time, students are asked to improve their testing with what they can learn from the source code of the website.

Figure 1. The instructor’s website at http://www.ent.ohiou.edu/~liuc.

This exercise was designed for two courses. One is Ohio University CS456/556 “Software Design and Development,” a dual-listed software engineering course that has a software testing component. The other is “Software Testing and Analysis,” a graduate-only course that focuses on software testing. Since the website-under-test in this exercise

- 3 -

is the official website of the instructor of these two courses, it is likely that students have visited this site before the exercise. It is also likely that students looking for course information view this as a simple webpage and notice nothing out of ordinary. Part One In part one of the exercise, students are asked to “test the website by all means with the only restriction that the source code is off-limits.” The use of the term “website” is intentional to avoid giving hints to students by using the term “webpage” or “web application.” After testing, students are required to compile a list of defects and respond to a survey with the following questions: • On a scale of 1 to 10, 10 being the highest, how would you rate your knowledge and

skills of software testing? • How much time did you spend on this task? • Do you have prior knowledge of this website? If yes, describe what you knew prior to

this exercise. • What did you do during this testing exercise? • List the bugs you found on this website. Be sure to number the bugs and give detailed

descriptions. • On a scale of 1 to 10, how thorough do you think are your test results? (Ten being the

most thorough.) Why? • When you decided you were done with testing, what was your reason? • Do you have any questions regarding this task? If yes, what are they? Part Two In part two of the exercise, students are asked to read the source code of the website and perform further testing. They are again required to compile a list of defects and respond to a survey afterwards. In addition, they are asked to reflect on the difference between their testing in part one and part two. The questions in the survey are listed below. • How much time did you spend on this task? • Describe your familiarity with JavaScript, HTML, and Ajax. • What did you do during this part of the testing exercise? • List new bugs you found on this website. Be sure to number the bugs and give

detailed descriptions. • On a scale of 1 to 10, how thorough do you think are your test results? (Ten being the

most thorough.) Why? • When you decided you were done with testing, what was your reason? • Do you have any questions regarding this task? If yes, what are they? Defects In the website-under-test, there were a number of known defects, some of which are listed in Table 1. There were at least a dozen more defects that are not listed in this table.

- 4 -

Some listed defects may be easily observed but may not be recognized by students, such as defects 5 or 6. Some may be hidden and may be missed by many students, such as defects 3 and 4. Some defects were highly technical, such as defect 9; others could be understood easily by anyone, such as defects 2 and 10. We expected all students to find at least a few of these listed defects. We also expected many of them to find new defects. We did not expect any of them to find all potential defects. Even if some do exhaust our list of known bugs, which is highly unlikely, an instructor can always come up with new bugs and surprise the students. An instructor can do so because no requirement specification is provided in this exercise, which we did not expect students to question. Table 1. A partial list of potential defects in the website-under-test and how an instructor can relate them to more serious real-world cases.

Defects Description of potential defects Potential lessons for the students 1 Many actions do not work in IE. Browser compatibility testing is important. 2 The mcc.com link does not work. Permanently unavailable links are different from

temporarily unavailable links. 3 The hidden link to the “office hours” story

is accessible. Something does not show up on a web page does not mean it is not accessible. Real-world example: • Hidden words in Microsoft Word documents

and PDF documents.1 4 There is an inappropriate hidden title in

“null.html.” Developers sometimes leave inappropriate hidden materials in products. Real-world example: • Yahoo! Map prank on the Google address2. • “Hot coffee” scene in game “Grand Theft

Auto: San Andreas.”3 5 The background color changes when the

pointer moves to the “four seasons” section. Without a requirement specification, how does one know if this is a feature?

6 The background color does not change when the pointer moves to other sections.

Without a requirement specification, how does one know if this is not a bug?

7 The publication list is not up-to-date; the person in the photo on the upper left corner is much younger then the actual Chang Liu.

Depending on the purpose of this website, this could be a serious defect.

8 Repeatedly press the mouse button down on the “four season” section, release it outside. The orange tip will display redundantly, as shown in Figure 2.

Some subtle programming bugs are difficult to find without access to source code.

9 Press the mouse button down elsewhere, release it on “Last updated,” a “function not defined” error will be reported on the JavaScript Console.

New programming paradigms, in this case Ajax [1,2], may bring in new types of defects.

10 The preview areas are too wide for smaller screens.

Usability testing is important.

1 http://www.newscientist.com/article.ns?id=dn4057 2 http://www.siliconvalleysleuth.com/2005/11/yahoo_maps_pran.html 3 http://www.mtv.com/news/articles/1516675/20051130/index.jhtml

- 5 -

Figure 2. The orange tip could show up redundantly, which is a software defect.

Lasting Impression The Web is something that today’s students are familiar and comfortable with. Software testing is something that many of them think they know, to various degrees. By asking them to test a simple, innocent looking website that they most likely have visited before the exercise, we give students confidence in doing a good job. Thus, we set them up for big surprises later on. We expect such surprises to leave a lasting impression in students’ minds on the challenge, difficulty, and importance of software testing, and to motivate them to learn software-testing topics better. 3. Preliminary Results A small number of students4 have taken part one of this exercise, including three graduate students and one undergraduate student. Among them one graduate student and the undergraduate student also completed part two and responded to a final questionnaire. All four of them had taken CS456/556 “Software Design and Development,” and had a basic knowledge of black-box and white-box testing. All four of them were among top 25% of the class. As expected, students failed to discover most of the defects listed in Table 1 as well as most of other defects that we knew ahead of the exercise. Several students caught additional defects that were not planted and that we were not aware of. In addition, students actually observed defects 5, 6, 7, and 10, but did not consider them as defects. 4 The number of students in this study was smaller than expected because the graduate course “Software Testing and Analysis,” one of the courses for which this exercise was designed for, was scheduled for Winter Quarter 2006 but was cancelled less than two weeks before the quarter started for unexpected reasons.

- 6 -

This showed that software testing in their mind had a narrow definition and included only things that were broken or would crash. Unexpectedly, neither student who reached part two of the exercise caught the two logical defects 8 and 9. The final survey showed that both of them considered these behaviors as software errors, but neither of them observed these behaviors even though the source code was accessible to them. Most importantly, no students asked the question “What is this website for?” They all approached this testing problem with an assumed but not acknowledged requirement. After the exercise, students took a final questionnaire, in which behaviors of known defects and known testing methods were listed. Students were asked whether they observed the described behaviors, whether they considered them as defects, and whether they adopted the listed testing methods. At the end of this survey, we asked students to again provide a self-rating of their testing efforts. Among the two students who took the final survey, one gave a lower self-rating as compared to the self-rating provided in part two of the exercise; the other gave the same self-rating. While the sample size was too small to prove anything, at least the trend was consistent with our expectation that the listed defects and testing methods in the final questionnaire would show them what they had missed in the testing of such a simple and familiar subject. We hope that when this exercise is used in the beginning of a testing course, it can therefore help motivate students. 4. Further Use of the Exercise in Lectures Later on in a testing course, an instructor can refer back to this testing exercise when discussing topics such as black-box testing, white-box testing, usability testing, compatibility testing, security testing, and load testing. In the third column of Table 1, a number of examples are provided that can be used to relate elements of this exercise to real-world cases. In addition, by asking questions such as those listed in Table 2, the instructor can motivate the discussion of specific topics such as testing tools (question 2) and security testing (question 7). Table 2. Questions that an instructor may ask after the exercise.

Question 1 Which browsers/version of browsers/operating system did you use in testing? 2 Did you open the JavaScript Console during testing? 3 Did you print the page out on a printer? 4 Did you load the page on a PDA or a cell phone? 5 Have you considered load testing? 6 Did you measure the average latency of the web server? 7 Did you attempt to find out the configuration of the web server hosting this page and research its

vulnerability?

- 7 -

5. Summary In summary, we have designed a seemly simple website testing exercise. Our preliminary results indicate that as expected, students missed most of the defects and did not use many well-known testing strategies in this exercise. We think that when students find out what they missed on such a simple and familiar task, they will be motivated to learn more about the challenges and the techniques of software testing and will better appreciate the importance of software testing. References [1] Greg Murray, “Asynchronous JavaScript Technology and XML (AJAX) with Java 2

Platform, Enterprise Edition,” Sun Developer Network, June 9, 2005. http://java.sun.com/developer/technicalArticles/J2EE/AJAX/index.html?cid=59754

[2] Brett McLaughlin, “Understanding Ajax, a productive approach to building Web sites, and how it works,” IBM DeveloperWorks, December 6, 2005. http://www-128.ibm.com/developerworks/web/library/wa-ajaxintro1.html