Testing User Manuals

8/2/2019 Testing User Manuals

http://slidepdf.com/reader/full/testing-user-manuals 1/12

1 | P a g e

TESTING USER MANUALS

INTRODUCTION

Testing is the process of testing software for defects. It requires a tester to play the role of an end user,

and use most of all features of the application to ensure correct behavior. To ensure completeness of

testing, the tester often follows a written test plan that leads them through a set of important test cases.

OVERVIEW

A key step in the process of software engineering is testing the software for correct behavior prior to

release to end users.

For small scale engineering efforts (including prototypes), exploratory testing may be sufficient. With thisinformal approach, the tester does not follow any rigorous testing procedure, but rather explores the user

interface of the application using as many of its features as possible, using information gained in prior

tests to intuitively derive additional tests. The success of exploratory manual testing relies heavily on the

domain expertise of the tester, because a lack of knowledge will lead to incompleteness in testing. One of

the key advantages of an informal approach is to gain an intuitive insight to how it feels to use the

application.

Large scale engineering projects that rely on manual software testing follow a more rigorous methodology

in order to maximize the number of defects that can be found. A systematic approach focuses on

predetermined test cases and generally involves the following steps

1. Choose a high level test plan where a general methodology is chosen, and resources such as

people, computers, and software licenses are identified and acquired.

2. Write detailed test cases, identifying clear and concise steps to be taken by the tester, with

expected outcomes.

3. Assign the test cases to testers, who manually follow the steps and record the results.

4. Author a test report, detailing the findings of the testers. The report is used by managers to

determine whether the software can be released, and if not, it is used by engineers to identify and

correct the problems.

A rigorous test case based approach is often traditional for large software engineering projects that follow

a Waterfall model. However, at least one recent study did not show a dramatic difference in defectdetection efficiency between exploratory testing and test case based testing.



2 | P a g e

STAGES

There are several stages. They are:

Unit Testing

This initial stage in testing normally carried out by the developer who wrote the code and

sometimes by a peer using the white box testing technique.

Integration Testing

This stage is carried out in two modes, as a complete package or as a increment to the earlier

package. Most of the time black box testing technique is used. However, sometimes a

combination of Black and White box testing is also used in this stage.

System Testing

In this stage the software is tested from all possible dimensions for all intended purposes and

platforms. In this stage Black box testing technique is normally used.

User Acceptance Testing

This testing stage carried out in order to get customer sign-off of finished product. A 'pass' in this

stage also ensures that the customer has accepted the software and is ready for their use.

DOCUMENTATION EFFECTIVENESS

Documentation is fully effective with following benefits:

Enhances salability: Documentation quality is often a selling feature. It helps retail salespeople

explain and demonstrate the product. It also plays a major role in many software reviews.

Improves installability: Customers have to install software products on their computers. They

may have to copy the software onto their systems, then customize it or load a special database

that reflects their needs and equipment. Installation software is written last. Developers take it

less seriously than other parts of the product because customers will install the product only once

or twice. These routines get the least amount of testing and development support. However, your

customer's first experience with the product is installation. Difficulties will lead to refunds and

expensive technical support demands. Clear, correct installation instructions are among the most

important parts of the documentation. Installation of some types of products (such as telephone

systems) is so complex that customers hire an installer. As a rule, installers work with many

different products. Don't expect them to be expert users or installers of your product. Expect themto have to look things up in the manual. The harder it is for them to find information, the more

they have to charge the customer. Some vendors refuse to carry products that cost too much to

install. Installation software also needs re-installation instructions. The manual must explain how

to change options and how to upgrade from a previous version.

Reduces liability: Your Company makes a false claim about the program's capabilities when the

manual says that the program does something that it doesn't. Your company misleads the reader



3 | P a g e

when the manual says to do something in a way that doesn't work. Incorrect instructions cost the

reader unnecessary time, effort, and mental anguish.

Improves usability: A well-documented product is easier to use. Schneider man (1987) reviews

evidence that better documentation leads to faster learning, fewer errors, and better throughput

(i.e., people get more work done).

Lowers customer support costs: When one of your customers can't figure out how to use the

product, she'll call for help (or for a refund). Telephone-based technical support is an expensive

service. A good manual prevents many unnecessary support calls.

Improves reliability: Unclear and inaccurate documentation makes the product less reliable

because the people who use it make errors. Excellent documentation helps reduce the number of

user errors, even (or especially) when the program's design is awkward.

Increases maintainability: Much time and money is spent tracking down problems that turn out

to be user errors. Many product changes merely repackage the same capabilities. They're made in

the hope that people will finally understand how to use these capabilities and quit complainingthat they aren't there or don't work. Better manuals reduce these problems substantially; poor

documentation contributes to them

OBJECTIVES OF THE DOCUMENTATION TESTER

Reviewers of documentation are concerned with improving its accuracy, completeness, clarity, ease of

use, and the degree to which it captures the spirit of the design. The documentation will have problems in

all of these areas.Plan to test the manual, online help, and any other documentation many times.

As a tester working with the manual (or help), you are responsible for checking the technical accuracy of

every word. There is no substitute for a thorough, thoughtful comparison between the claims and

implications in the manual and the actual behavior of the running program. This is what you do. (As a

side benefit, you find lots of bugs this way too). Be on the lookout for confusions in the writing.

Many of them stem from confusions and complexities inherent in the product's design. The writers must

describe the product as it is. You can help them substantially by pressing for changes that make the

program more documentable (and more usable). Look out, too, for missing features. Writers work from

specifications, notes, and rumors. Developers try to keep writers up to date, but sometimes they forget to

mention recently added features. You often discover changes much sooner than the writer — make sure

these get into the manual. And don't assume that they got into the help text just because they got into themanual.

These are probably being written by different people; updating information can easily get lost.

Realize that you are a reviewer, not the writer. Most testers who think they know more about writing than

the writer are wrong. You have no more right to demand changes in the manual than in the program. It is

your responsibility to find and identify problems, but after giving them a fair hearing, writer and

programmer alike can choose not to follow your suggestions.



4 | P a g e

TESTING DOCUMENTATION CONTRIBUTES TO SOFTWARE

RELIABILITY

You 'II find more bugs than you expect Surprisingly many bugs show up when a competent tester

thoroughly checks the manual (or help) against the program. The writer looks at the program from a

different angle than the programmer and the tester, so the manual will reveal different problems than the

ones programmers and testers look for. We've seen this happen on so many projects that we now take it

for granted that tests of the manual will reveal many serious errors that other testing has not yet turned up.

Documentation testing doesn't always reveal significant new problems. Testers who don't do a thorough

job don't find many problems. A full test of the manual takes about an hour per three to five pages.

Testers who speed through the material more quickly find much less in the book and in the program. We

make a point of monitoring for this problem and retraining or reassigning staff to deal with it.

It's an important source of real world combination test cases. You can't hope to test all the combinations

of features or other options in the product; there are just too many. But you can test every combinationthat the manual describes as interesting or useful. Any time the manual even hints that two aspects of the

program work well together, test them together.

Bug reports arising out of documentation testing are particularly credible. The manual is your company's

instruction to the customer on how to use the product. It's hard to dismiss a bug as "esoteric" when you

report that the program failed while you were following the manual's instructions or suggestions, or were

checking one of its statements of fact. These are mainstream tests. These are things many people will do.

These are errors that magazine reviewers can publish without fear of correction. These bugs are hard to

defer — either the manual will change or the program will change.

We've often seen previously deferred bugs reconsidered and fixed when they showed up again during

testing of the manual.

In your main test of the manual, you should sit with it at the computer and:

Use the program exactly as the manual says. Enter every keystroke in every example.

Customers make mistakes when they try to follow instructions. Feel free to make mistakes too. How does

the computer respond? Bad error handling in the program will look worse when you show that it happens

in response to an obvious, common mistake that several people will make when they try to follow the

manual's instructions.

Try every suggestion, even if the suggestions aren't fully spelled out, step by step. Do what a reasonable

customer would do who was trying to follow the suggestion. Check every statement of fact and every

obvious inference from the stated facts, instructions, and suggestions. The manual is the product's final

specification, and the customer's first place to check whether the program is working correctly.



5 | P a g e

It also pays to retest the documentation when you add a tester to the project. This keeps the manual

current while the software is changing, and it educates new testers about the program. Consider assigning

every new tester to work through the most recent draft of the manual as his first task.

BECOME THE TECHNICAL EDITOR

If possible, the Testing Group should assign one person to the manual as technical editor. He

might play other roles too, but in this role he is the primary technical reviewer, even if many

other people also review the manual.

It is very common when two or more people check a product for none of them to take ownership

of the task. Rather than improving the thoroughness of the review by adding another tester,

thoroughness declines because everyone expects the other person to do the job. The technical

editor should be encouraged to feel ownership of the technical accuracy of the book.

WORKING MANUALLY THROUGH ITS DEVELOPMENT STAGES

The manual is developed in stages. The four major stages are:

Conceptualization and initial design: The writer makes decisions about the scope, target

audience, degree of coverage, and general organization of the manual.

Preparation: The manual is written, reviewed, rewritten, etc. The manual is in the preparation

stage until its content is in final form.

Production: The manual is laid out for publication. This might involve typeset ting or making

revisions to make the manual look as good as possible when printed by a laser printer or a daisy

wheel. Typefaces are chosen, page style (margins, etc.) is designed, final artwork is prepared, and

so forth.

Publication: The manual is printed or copied and bound, ready for circulation.

Testing is concentrated in the preparation stage, with some spillover into production. You review the

manual's content, not the layout, unless you have expertise in layout. You probably won't be involved in

the initial design of the manual. You will rarely be involved in publication: the writer checks that the

printed manual contains all the pages, none upside down, and so forth.

Some comments are more welcome at some times than others. Sometimes the writer wants to work on

accuracy, sometimes on style, sometimes on organization. The following sections look at the value and

appropriateness of different types of comments across various preparation and production stages. These

notes describe the needs of the "typical" writer; there are many individual differences. Talk with the

person you work with about her needs and schedule.



7 | P a g e

Look for areas that need discussion. Some features have not yet been described in the manual.

The writer may not know that a new feature has finally made it into the program and is ready to

be examined, and described, carefully.

Look for violations of the spirit of the design. The writer might miss a simple conceptual

relationship between features, and describe each independently. Carefully devised large-scale

consistencies are lost. The writer might imply disapproval of a program's restrictions. While somerestrictions are arbitrary, often to simplify the user interface. The writer might approve if she

understood the reasoning. The manual might suggest inefficient strategies for some tasks -they

work, but a user who understood the program well would do things differently. In cases like

these, the writer has missed something fundamental. Once she gains understanding, she may

make significant changes, redoing not only this material but also discussions of related topics.

These might he big revisions. The writer must understand the need for them as soon as possible.

Look for things that mislead. Some examples and feature descriptions aren't incorrect, but a

reasonable reader might generalize incorrectly from them. She might expect the program to be

more capable than it is. She might believe she can do something under more generalcircumstances than she can. Or she might imagine restrictions that don't exist on the use of a

feature. It is particularly important to flag misleading material early because the writer might

believe those incorrect generalizations. She might make significant changes if she understood the

product differently.

Check the error message: The writer will probably include a list of error messages in an appendix,

with notes on how the reader probably got to this message and what to do about it. If you've been

keeping a list of every way you've found to get each message, this is invaluable for the writer.

The writer will base her explanations on your list and on information from the project manager

and the technical support staff (who rely on these explanations to help keep customers from

flooding the company with phone calls). It pays to test every message as it's explained in the

book — you'll find more bugs. After you've tested a message, give the writer your additional notes

about other message meanings, ways to get the message, or things the customer must do or avoid

doing as soon as he gets the message. • Look for confusions that reflect on the program. If the

writer can't describe some aspect of the program in a clear and consistent way, evaluate the

program before condemning the manual. If the program has many inconsistent options, it's a mess

and so will be the documentation. Suggest improvements to the writer (if you can). Don't spend

hours rewriting sections of the manual, but if you can do it quickly, do provide a description that

you think is clear and acceptable. Also, write Problem Reports if you think confusion in the

program is the cause of confusion in the manual. It's easy and common to condemn writers fordocumentation that accurately describes an incompetent design. We find it more profitable to start

with the assumption that the writer is competent and that bad text is telling us something about

the program.



8 | P a g e

THE BETA TEST DRAFT

This draft or a revision addressing comments to this draft, will be the last one you'll see before

production. (In companies that don't rely heavily on beta tests, the final circulating draft is the user

interface freeze draft, circulated after the software's design has frozen.)

Beta testers don't work for your company. They use the product in the same ways they would have had

they bought it in finished form. They report their difficulties with the product, their suggestions for

improvement, and any bugs they find. You should review their reports about the software and about the

documentation.

Up to this point the marketers, programmers, writers, and you have been making assumptions about how

people will react to the product and about what they'll understand. Some of those assumptions are wrong.

Some seemingly obvious aspects of the program may be incomprehensible to beta testers.

Many changes to the manual come from user testing. Users often complain if the documentation is not

task-oriented. A task-oriented manual anticipates what users want to do with the product and explainshow to do each task. It describes features in the context of using them to get specific tasks done. In

contrast, a feature-oriented manual describes features individually, maybe in alphabetical order. Each

section includes everything you ever wanted to know about one feature. Brockmann (1990) notes that

task-oriented manuals are much longer, but review some further evidence that they are more effective.

If the product is so widely useful that people could do thousands of different types of tasks with it, the

writer could never finish a task-oriented manual. As a compromise, writers often write a task-oriented

tutorial that covers the most popular tasks. Beta test comments may convince the writer to improve the

task orientation with more examples, more illustrations, a different index, or a different organization.

Customers will raise many other issues about the documentation. As always, ask yourself whether their

confusion is really due to poor documentation. We repeat this point because it's so often missed.

The manual is often blamed for the faults of a poorly designed program, but no accurate description of a

fundamentally confused program can be clear. Complaints about documentation should often lead you to

file Problem Reports about the program's user interface.



9 | P a g e

PRODUCTION

Your main concern during production is that the document stays accurate.

Someone in the documentation group, the writer or an editor or editorial assistant, will do the main

proofreading of the laid out or typeset manuscript. You too can note spelling mistakes, misaligned

headers, etc., and these notes will be welcome, but if that's all you can provide at this point, you're

wasting your time.

If the company wants to release the product as soon as the software is

complete and tested, documentation production must start 8 to 14 weeks

before the program is finished

The program will change over those many weeks. Some parts of the manual will no longer be correct.

Further, some bugs that everyone expected to be fixed will not be. Sections of the manual that assumed

that a given bug would be fixed, in good faith and on good authority, have to be revised.Not all desirable changes can be made during production. The writer will (should) change as little as she

can get away with. You can get her to make more changes, and help keep the cost of the changes down,

by designing the changes to match production constraints.

As soon as a manual enters production, it stops being an organized collection of words. It is now a bunch

of pages. There happen to be words and pictures on the pages, but each page is separate from all others.

Each was carefully laid out; each will be photographed on its own.

The writer will not make changes that affect more than one page unless they are essential.

At the other extreme, it is easy to make a change that affects only one line, without moving words down

to the next line or needing words from the line above. If you can keep a change within a line, a paragraph,

or a page, you have some hope of convincing the writer to make it. It is your responsibility to convince

her that the change will stay within those limits. Be prepared to provide a suggested wording, and to show

how the words fit on the lines. The wording must be stylistically acceptable to the writer.

This degree of editing is beyond the formal scope of your job. You can be asked to stop doing it. You

don't have to do it and you shouldn't try unless you can do it well without taking too long. If you don't do

it, you should send the writer a memo describing your problems with the manual. She will save it and

incorporate your comments in revisions made for the next printing of the manual, or in the manual for the

next version of the product. Also, if she thinks that one of the problems you raise is critical, she will work

on the wording and get it td fit.

Another area to test during production is the index. The earlier you can get your hands on a draft index

the better. You can improve the index's completeness by working with the draft manual and constantly

trying to look things up in the index as you use the manual. Many words you expect in the index won't bethere. Report them to the writer. You (or an editorial staff member) must also check the index when

everything is finished, just before the book goes to the printer. The "final" index may miss entries from

one chapter, or it may still be based on a previous version of one chapter. At a minimum, check at least

two index entries in each five pages of the book.



10 | P a g e

POST-PRODUCTION

Some companies don't print the manual until after the software is finished. In these cases, there

are no post-production tasks. (The writer still has some tasks, such as checking bluelines, a first

print run from the printer, but you probably won't have to check bluelines.)

If your company does send the manual to print before the software goes to the duplicator, the

writers probably have to write two further documents. One is a printed supplement that includes

corrections, troubleshooting notes, and discussion of additional features. The typical supplement

goes to print a few days before the disks go to the duplicator. Later-breaking information must

go into a README file on the disk.

Apart from checking the accuracy of material in the supplement and README, your most

valuable contribution during this period is identifying troubleshooting tips and explaining them

to the writer. Every deferred bug is a potential idea for a troubleshooting tip. If you (or the

writer) can describe the bug in a positive tone, and tell the customer something usefiil, it's a good

candidate for the troubleshooting section.

ONLINE HELP

Most of what we've said about the manual is equally true for help. Here are a few additional notes.

Accuracy: You must check the accuracy of help at least as closely as the manual. Help

text is generally not well done, not well tested, and not well respected by customers. A

customer will probably quit using the help immediately if she finds factual errors.

Good reading: The best book we've read (or seen) on online help is Horton (1990).

Help is a combination of writing and programming: You have to check the accuracy of

the text and the reliability of the code. Tf the programmers implement help using special

system-provided tools (common in GUI environments), it will pay to read the system's

instructions to the writer and to the programmer. You will find bugs that stem from the

writer and programmer not understanding each other's job well enough to cooperate

perfectly.

Test hypertext links: If the program includes hypertext links (cross-references that will

take you directly to other topics), you have to check each link. Suppose the writer cross-

references to "Keyboard layout" in two different places. In most systems, she could havethe program jump to one help message if you select the first "Keyboard layout" and to a

different message if you select the second "Keyboard layout." What you see doesn't

necessary identify correctly where you will go. You have to check it in each case.

Test the index: If the program includes an index or a list of topics, and lets you jump

from the index to the topics, you must check each one.



11 | P a g e

More on the index: If the program includes an index, or hypertext links, you should note

whether the index entries or the list of linked topics (per topic) are sensible. Some help

topics never appear in the index or appear only under odd names. Customers will back

away from the system if they can't quickly find information they need.

Watch the style: Few customers take a leisurely read through help. They come to_help

with a specific question, or specific task they have to do, or error state they have to attend

to. Help readers are often nervous or distracted, and they are often impatient. Expect the

help text to be much more concise than the manual. Its style should also be much simpler

(grade 5 reading level is sometimes recommended). Good help text is also very task- or

action-oriented. It must say something useful, which the customer can do right away. If

you find anything confusing or drawn out in help, report it as a problem.



12 | P a g e

REFERENCE

I hereby declare that all the contents of this report is taken from the book “Testing computer software-

2nd

edition” written by “ Jack L. Falk , Hung Quoc Nguyen”.

And the link “en.wikipedia.org/wiki/” & “http://naveengoogle.blogspot.in/ ”.

http://naveengoogle.blogspot.in/

http://naveengoogle.blogspot.in/

Testing User Manuals

Documents

Transcript of Testing User Manuals