Adam Wilson AP Final Project REVISED FINAL DRAFT

A STRUCTURED APPROACH TO SYSTEMS PROJECTS USING DESIGN CHECKLISTS 1

A Structured Approach to Systems Projects

Using Design Checklists

By Adam Wilson

DePaul University

Author Note

This paper was prepared for the Advanced Project course taught by Ed Paulson, Winter Quarter 2015.


Introduction

This paper addresses the problem of success in designing information systems. The best system

design occurs when a designer is familiar with good design approaches and can apply them appropriately

given the particular situation. Outcomes are less successful when designers do not choose intentionally

or skillfully but instead use approaches that are familiar or comfortable for them (Saffer, 2010). The

solution to this problem I will explore in this paper is the use of design checklists. The checklists will

consist of critical and easy to miss tasks taken from the Systems Development Lifecycle methodology as

well as other scholarly sources and my own personal experience. I hope to show in this paper that

checklists can help professionals in real situations to “make sense of… complex issues… in decision making

…” (McKeown, 2013). I will also address a number of obstacles to the successful use of checklists.

Obstacles include production pressures, personal inclinations, overloaded or unclear checklist design and

imposition of overly restrictive requirements (Katherine Radeka, 2015). There are three outcomes I hope

to achieve through my efforts. I would like my checklists to get used in actual design work, I would like to

achieve more consistent success in creating information systems and I would like to use the checklists in

project evaluations to better understand the influence of particular design tasks on system outcomes.

Advanced Project Competency Statements

F-11: Can design and produce a significant product that gives evidence of advanced competence.

F-12: Can design practical, holistic design checklists that allow flexibility while controlling for coverage of

primary value adding, critical and easy to miss project tasks.

1. Can describe primary value adding, critical and easy to miss project tasks in computer systems design.

2. Can define strategic pause points for checklist review amidst production pressures.

3. Can describe how checklists help individuals or teams take complex knowledge and consistently, correctly

and safely apply it under pressures and constraints.

The Problem

In today’s knowledge workforce end-users “design and develop an increasing number of their

own applications” (Turban, 2009). The discipline of design is therefore of increasingly broad relevance.

The best design occurs when the designer can apply the best approaches for the particular situation

however many designers include only those with which they are familiar or most comfortable (Saffer,

2010). The many phases and tasks of design, often with sequential dependencies present a problem of

complexity. These challenges can result in delays, abandoned projects, lost resources, communication

failures and tensions between stakeholders, poor usability, unmet system requirements, reduced internal

and external efficiency and effectiveness, poor usage rates and lost organizational knowledge (Plessis,


2007) and the dissolution of relationships. Even when we have years of experience and training,

“avoidable failures continue to plague us… the volume and complexity of knowledge… [Can] exceed our

ability as individuals to properly deliver it…” (Gawande, 2009). I’ve had varied success with system

projects, facing many of the challenges described above not always understanding the design tasks that

might help or how to consistently integrate them under constraints of time, budget, expertise, etc..

My Background

From 1999 to 2009 I worked as an office manager in a small private school. I oversaw customer

service and developed workflows for staff-client interactions. In 2009 I left this position and returned to

school at DePaul to complete my BA in Computer Information Systems. I hoped to bring my interest in

administrative process to the digital space as administrative work was becoming increasingly digital. The

Information Systems Focus Area courses have covered technical aspects of computing such as

programming and administering systems as well as people and process oriented aspects like systems

analysis and design, product and project management and organizational dynamics. I was in school for 2

years before I took another full time job as office manager at a church I admired. The church hoped that I

could bring some of their office processes “into the digital age”. Over that time I partnered with various

staff and others to design, built and support new systems for a number of core processes. These projects

included a public website redesign, implementation of Google docs and sites for real time document co-

authoring, an online room reservation system that has handled thousands of transactions and an online

staff resource portal among others.

Solution Overview: The Design Checklist

My solution to the problem of achieving consistent system design success is a design checklist.

Research reveals that checklists are powerful but they need to be created and implemented well

(Katherine Radeka, 2015). Checklists are everywhere in various forms from aircraft flight checklists to

construction site work schedules to rubrics used by teachers to grade assignments. Checklists are used to

support work by individuals and teams in complex, fast paced environments full of production pressures,

cultural factors, personal inclinations and regulatory issues (Asaf Degani, 1991). They can help

professionals “make sense of… complex issues… in decision making situations…” (McKeown, 2013).

Important checklist best practices I will incorporate include customization to individual needs, inclusion of

only critical or easy to miss items so the lists are non-cumbersome, a series of short lists instead of one

long list and usage guidelines for when and by whom each list should be reviewed (Katherine Radeka,

2015). The lists will be populated by design tasks taken from the Software Development Lifecycle (SDLC)

(Turban, 2009) and other methodologies including Interaction Design and User-Centered Design (UCD)

among others.


Intended Outcomes: Success, Consistency, Understanding

My intended outcome from the design checklists is consistent system design success. I would

also like to better understand the linkages between design tasks and outcomes and be more intentional

in choosing how to spend my time and energy. The next section on evaluation will discuss ways of

measuring success. I would like to replicate the outcomes I’ve achieved using checklists for my tasks as an

office manager. The partial image of my office management weekly checklist below demonstrates that

my prior work in administration has been highly checklist driven. Although I had not yet studied checklist

best practices the lists I created for various aspects of my job follow some of them. They include easy to

miss and critical items, fit on one page, are quickly reviewable and ordered sequentially and/or logically,

were used collaboratively by administrative staff to track and communicate among team members and

were online to facilitate sharing and frequent updates (Katherine Radeka, 2015). Comprehensive

instructions were hyperlinked to items to facilitate access without overloading the checklist itself. The

lists were built in Google Sheets and the Design Checklists at the end of this paper may eventually be

migrated there as well. My office management checklists helped make me a very reliable and consistent

administrator and I would like to expand that kind of competency into the slightly less routinized and

more creative space of design.

Office management checklist used by the author. Checklists helped make me a very reliable and consistent administrator and I’d like to generalize this to systems design.

Criteria and Methods of Checklist Evaluation

In the next few pages I’ll describe some criteria that makeup successful system outcomes as well

as how to evaluate them and at what stages of design projects. The activities will become part of my

design checklists. Joel Palmius states in the article Criteria for measuring and comparing information

systems (Palmius, 2007) that “With operational criteria… a foundation can be laid for improving the

system… A designer needs to know what the goal is, and a buyer… needs to know whether the goals have

been fulfilled. With criteria, there could be a checklist before, during and after a design phase… (Palmius,

2007).” Evaluations are conducted by designers looking independently at analytics, transactions and


other data but the sections below focus mostly on usability factors and subjective stakeholder feedback

via questionnaires and observation.

Evaluating the Existing System (Pre-Design)

The first half of the items on a Checklist for User-Centered Design (Oregon State Computer

Engineering Dept., 2000) address “Initial motivation” and “pre-Design Activities”. These activities involve

user’s right from the start with evaluation of the current system and other existing competitive systems.

This is crucial for establishing baseline measures against which to compare any new systems designed. It

is also crucial to put users at the center of the design process from the outset. User concerns involve

work goals, tasks, mental models, needs and desires. This pre-design evaluation also investigates

whether the system is actually the main problem as opposed to factors like user awareness, training or

trust in the current system.

Multiple Evaluations during the Development Lifecycle

Evaluation should occur at various points during design. Methods that involve multiple iterations

of a solution in the design, prototype and eventually the final system need to evaluate repeatedly.

Various methodologies all count on feedback at points (Turban, 2009). Examples include pre-design,

formal design proposal (design brief/system requirements document), low-fidelity prototypes like

wireframes, initial system rollouts, and ongoing during the operation of the system.

Project Management and Collaborative Criteria for Success

Communication is a prerequisite for design success. In the building construction industry experts

have said that communication and tracking advances are the biggest improvements to the trade in recent

decades (Gawande, 2009). Consultant Scott Vaughn (Vaughan) goes so far as to recommend signed

agreements between parties in settings that are usually more informal such as between colleagues. In

today’s workplace where “rapid expert design” is going on all the time (Saffer, 2010) the “designer” is

often a member of the team. The table below lists some metrics and ways to control for them.

Project management success Criteria Project Management Tasks (Coleman, 2015)

Project completion rates

Project delays

Failures of communication

Stakeholder relationships and

morale

upfront discussion of budget, timeline and goals

client questionnaire

defined stages and deliverables

designer proposal including features, costs, stages

and deliverables

contract to sign with paid deposit

Use, Validation and Refinement of the Checklists


We cannot evaluate the impact of the checklist on design projects unless we are using the

checklist. Project conditions present challenges for the successful implementation of checklists (Asaf

Degani, 1991). Checklists deal with complex situations as we’ll explore in the upcoming literature later in

this paper. Nobody gets them right on the first try and so they must be constantly reviewed, validated

and updated.

Designer Bias and Evaluation: Who Should Conduct Evaluations

Neutrality and freedom from bias is important in testing. The designer should not conduct or

collect evaluations if possible. If the designer must conduct evaluations they may choose not to identify

themselves as the designer to mitigate censorship (Saffer, 2010) and politeness by overtly inviting critique

and improvement, making clear that the design is a hypothesis. Usability testing specialists are often

involved in larger projects to navigate these issues and bring findings back to the design team.

Recruiting Subjects

The same techniques used for recruiting subjects for design input can be used for design

evaluation. In a website overhaul I performed I recruited ten users for design input. It’s a good practice

to thank people for input and to update them on what happened with the project (Saffer, 2010). I did

thank them but did not recruit them to evaluate the final product. I could have done so. New subjects can

also be recruited.

Observation and Interviews

Observation and interviews are two primary forms of evaluation that can work together. Design

rules apply when using observation: Go to the users (or stakeholders), use their computer, sit at their

desks, talk to them, take notes (Saffer, 2010). Bucket testing can be done where users are shown two

different design options and feedback is used to rank them (Saffer, 2010). A Test Plan should be devised

containing neutral questions as well as test routes “through the product that test functionality” (Saffer,

2010). Cognitive and behavioral measures like “time taken to complete a task or the number of errors or

deviations from a critical path” can be recorded (Jefsioutine, 2006). Dan Saffer adds that stakeholder or

user interviews are done best individually and in person to reveal thinking, emotion, as an opportunity to

challenge and should be conducted with stakeholders across all levels on the organizational chart especially

including those with power and/or direct client contact (Saffer, 2010). All observations and patterns should be

recorded and used to produce a summative “Opportunity Report” showing trouble spots and suggestions

for improvement (Saffer, 2010). Saffer suggests specific questions for Stakeholder interviews which

appear in the appendix of this paper.

Quick and Dirty Usability Evaluation using Heuristics

Prominent Designer Dan Saffer says that “if you don’t have the resources to do testing with users,

the least you can do is perform a heuristic evaluation” (Saffer, 2010). Heuristic evaluation is based on the


ten Usability Heuristics developed by Jacob Nielsen in 1990 and refined based on factor analysis of 249

usability problems (Nielsen, 1995). These heuristics were the basis for the rapid system usability

improvements discussed in the source elsewhere cited in this project at the University of Bari (Nicola

Convertini, 2004) and they are listed in an appendix to this document. They are broad rules of thumb for

what constitutes a useable system and include items such as “visibility of system status”, “consistency

and standards” etc. (Nielsen, 1995).

Quick and Dirty Evaluation using The System Usability Scale (SUS) Questionnaire

The SUS has been widely used and “provides a “quick and dirty”, reliable tool for measuring

usability (usability.gov, 2015). It consists of a 10 item questionnaire with five response options for

respondents: from strongly agree to strongly disagree.” It has become an industry standard with

references in over 1300 articles and publications. Notable benefits of the SUS include easy administration,

usability “on small sample sizes with reliable results” and proven validity for differentiating usable and

unusable systems. It does not diagnose particular issues but is useful for classifying overall usability

(usability.gov, 2015). SUS original creator John Brooke defends the concise nature of the questionnaire

saying that it is often “desirable to have measures which do not require vast effort and expense to collect

and analyze”, claiming that questionnaires over 25 questions are often not completed (Brooke, 1986).

Brooke goes on to state that systems are like apples and oranges and that the context of system use

varies widely ranging from word processing to chemical manufacturing. He argues therefore that a

subjective assessment measure like the SUS is the kind of evaluation tool which remains appropriate

across different system. The SUS is ideally administered just after the tester has used system and before

further discussion about the experience (Brooke, 1986). The SUS appears in the Appendix to this paper.

Questionnaire Items Supplementary to the SUS

The following questions are not directly addressed by the SUS. They could be part of a separate

evaluation questionnaire with follow-up interactions. User and stakeholder input is so important to good

design that I’ve taken the time to gather questions from additional survey sources and a few from my

own experiences that are not addressed directly by the SUS. I have limited them to 25 per the

recommendations by SUS article author John Brooke (Brooke, 1986). Like with the SUS the questions are

strong statements designed to minimize answers in the middle range of a 1-5 strongly disagree to

strongly agree Likert scale. The questionnaire is included in the Appendix of the paper.

Literature Review: Checklist Design and Human Factors

Atul Gawande writes in The Checklist Manifesto (Gawande, 2009) that “the volume and

complexity of knowledge today has exceeded our ability as individuals to properly deliver it to people –

consistently, correctly, safely. We can do better, using the simplest of methods: the checklist.” Research


reveals that checklists are indeed, very powerful but they need to be created and implemented well. A

checklist is not so simple after all. Checklists are everywhere in various forms from aircraft flight

checklists to construction site work schedules and rubrics used by teachers to grade assignments.

Checklists are used to support work by individual and teams or even teams of teams in complex, fast

paced environments full of production pressures, cultural factors, personal inclinations and regulatory

issues (Asaf Degani, 1991). These conditions and present challenges for the design and successful

implementation of checklists.

Buy-In: Making Checklist Usage a Norm

Buy-In is a critical factor in creating and implementing checklists. There are various strategies for

making checklist usage a norm including co-creation among professionals or in client-professional

relationships (McKeown, 2013), requiring users to make their own checklists and share them as part of

work interactions (Katherine Radeka, Whittier Consulting Group, Inc., 2015), ethnographic observations

of checklist usage behaviors and usage context, consideration of human end-user and cultural factors, list

usability and phraseology (Asaf Degani, 1991) among others. My wife is a teacher and she shared with

me the ways in which she is required to submit lesson plans. She is free to create or retrieve the plans

from a source or colleague but they must meet particular criteria. This is similar to the ideas put forth by

the “Experience Design Framework” (EDF) which is a response to methods that are stricter. The authors

argue that frameworks such as the EDF can work to create a “principled context without being overly

prescriptive” (Jefsioutine, 2006).

Top Leadership Support: power, norms, expertise

Checklists are becoming more common in medicine as studies reveal dramatic impacts on rate of

infection and other outcomes (Gawande, 2009). “Americans… have upwards of 150,000 deaths following

surgery every year – more than three times the number of road traffic fatalities. Moreover research has

shown that at least half… are avoidable.” IN Michigan, a project which rolled out a checklist to prevent

central line infections and in 2006 the New England Journal of medicine published findings revealing a

66% drop in these infections statewide (Gawande, 2009). Similar dramatic gains are being seen

worldwide with more comprehensive surgical checklists being implemented. Despite dramatic evidence

like this, Professional norms and operations issues can remain major challenges in the implementation of

a checklist. Programs must be holistic and involve dedicated project managers with expertise as well as

top level leadership. For example, nurses may need to be empowered to break the norm of deferring to

surgeons and be required with absolute backing from top hospital leadership to point out when a surgeon

has omitted a step on the checklist. Top executives may need to personally conduct on-site observations

which may reveal issues like failure of operations staff to consistently provide vital surgical supplies to

operating rooms. Only top leadership can address these kinds of holistic system issues. Surgical kits are


being redesigned by manufacturers to include all necessary supplies packaged together and ready and

props are being introduced to provide forcing functions on behavior such as scalpel cases inscribed with

phrases like “ready for takeoff?”, a reference to the aircraft industry which pioneered the use of

checklists to deal with complexity (Gawande, 2009). List Design

I’ve discussed issues in implementation first above because I have personally tended to focus

more on the design of the checklist product and less on the context of use. There is a wealth of crucial

research on the design of the lists themselves and there is overlap between these two areas. List design

should be human-centered and user-friendly for example, in order to increase utilization (Asaf Degani,

1991).

Key terminology is discussed in list design requirements. Lists should be grounded in the

literature (McKeown, 2013), in many cases non-comprehensive containing only the easy to miss or “killer”

(critical) items and no longer than a single letter sized page, (Katherine Radeka, Whittier Consulting

Group, Inc., 2015) a guide to use should be included and the formatting should be logical incorporating

location and most crucially, timing of activities so they are not competing with other production activities

directly and the list can be completed consistently at a specific break point in workflow (Asaf Degani,

1991). The World Health Organization Surgical Safety Checklist pictured below (World Health

Organization, 2009) exemplifies some of these principles: Sections are titled by when they should occur, a

list of required participants in the check are specified, the whole thing fits on a single sheet of paper and

the list has been edited down through validation and testing to only the most critical or easy to miss

items.


Impact of the Literature on Design Checklist Decision Making

My research has pointed to some best practices I will try to incorporate in my design checklist as

follows below.

Design phases will each have their own list according to logical pause points.

There will be no more than ten items per list.

The deliverable for each phase will be a checklist item.

Activities will be drawn from widely accepted best practices.

Appropriate list review participants will be stated on the lists.

Each item will have a large checkbox to mark appropriately.

Each item will represent a critical or easy to miss item. Optional additional or specific activities

may be listed separately on the page.

Several options may be listed for an item with instruction to please circles those conducted.

A notes column will appear on the right side of the page clearly distinct from the checklist.

A diagram of all phases will appear on the lists with the particular phase emphasized.

A final “go” or “no-go” item to check may appear on some lists.

Date of list creation and/or revision will appear on all lists.

The initial lists in this paper will be considered prototypes as they have not yet been validated

with simulated or actual frontline users.

Lists will be updated to suit current user needs.

Literature Review: Systems Design

Design Methodologies used to create the Checklists

The checklists will be populated by tasks taken from the Software Development Lifecycle (SDLC)

(Turban, 2009) and other methodologies including project management, Interaction Design and User-

Centered Design (UCD). The complete range of tasks will be divided into separate shorter lists based on

the phases of the SDLC. This is a logical division as the SDLC phases “consist of sequential processes by

which information systems are developed” (Turban, 2009). My focus in this paper is on the Systems

Investigation (and Feasibility Study), Analysis and Design phases and these are the phases I will describe in

detail and create Checklists for. I will also include a checklist of general System Requirements which is

relevant to the Design phase.

The SDLC Waterfall and Iterative Waterfall Models

A traditional approach to the SDLC, the waterfall model provides a helpful framework for

understanding strategic sequencing of activities. This sequencing structure is essential for complex


projects and enhances the chance of success on almost all projects. Feasibility of project success is

investigated upfront. The Feasibility Study asks questions such as: Is there passionate buy-in from top

level leaders who can allocate and protect investments of time and money? Does the project align with

organizational and IT strategy?

The “Iterative” Waterfall model differs from a traditional waterfall in that it allows for quicker

movement through the phases with the understanding that the project will cycle around through the

phases repeatedly. There is no expectation for getting things exactly right the first time around. Iterative

models “define an initial list of user requirements, builds a prototype system, and improve the system in

several iterations based on user feedback.” (Turban, 2009) This approach speeds up development and

allows users to clarify needs they may not have been able to articulate upfront. Many of the projects I’ve

led have used this model. This looser structure is more feasible when project scale is smaller and free or

low cost web-based tools are chosen.

Iterative Waterfall Model of Systems Development. This paper focus on the phases of feasibility, analysis and design.

Achieving Discipline without Being Overly Prescriptive

Certain methods and activities of design are more useful than others for particular problems.

Many of the sources on design discuss problems with highly specific, restrictive, centrally imposed

methodology. Such top down prescriptive methods do not fit environments full of complexity, creativity,

unpredictability, expertise and individual temperaments and are usually rejected (Jefsioutine, Design

Frameworks, 2006). Dan Saffer argues that design is an applied art (Saffer, 2010) and other authors argue

that less prescriptive frameworks can create a principled context where key basic purposes are covered

using a range of tools and methods from various disciplines (Jefsioutine, Design Frameworks, 2006).


Organization of the Design Checklists

The rest of this paper contains systems design tasks descriptions and related information

correlated with the tasks contained in the actual checklists. The actual checklists follow the descriptive

sections. Description and checklist task numbering is correlated for referencing purposes. Tasks are

divided and named using phases from the SDLC. Page breaks occur between phases.


Systems Investigation and Feasibility Analysis

1. Initial Framing of the Problem: The initial stage in the SDLC is system investigation. Time is well

invested here in a preliminary “framing” of the system problem. Information systems are

embedded within a larger context of organizational context, procedures, training, etc. Zooming

out and establishing a broader holistic view of the existing system can help clarify what the actual

problem is that needs to be solved. Further framing in greater detail will occur again in the

Analysis phase.

2. Project Priority: The project should be prioritized according to its alignment with mission goals,

the scope of its potential impact and the stakeholders it will benefit.

3. Feasibility Study: The feasibility study is the primary task of the Systems Investigation phase. It

answers the “go-no go decision” e.g. “Should we move forward with this project?” In order to be

feasible and worthy of investment a new or modified system project must meet feasibility criteria

in the following four areas (Turban, 2009):

a. Technical Feasibility: The system must work with available hardware, software and

devices used currently and projected to be in place in the future per the IT strategy of the

organization (see Organizational Feasibility below).

b. Economic Feasibility: The system must meet time and cost constraints. What is the

budget for technology, training or related sources that could be utilized? Can existing IT

resources be used?

c. Organizational Feasibility: “A design strategy that doesn’t work for the overall organization’s

strategy is like a bad organ transplant: the host body will reject it (Saffer, 2010).” System must

align with organizational and IT strategy and meet legal and other constraints (Turban,

2009). Documentation including websites, business plans, constitutions, mission

statements, IT strategy plans etc. can be reviewed and should be confirmed by discussion

with appropriate personnel.

d. Behavioral Feasibility: System must engage stakeholders and users who generally fear

change and commonly sabotage or avoid new systems.

i. Did the Initial Concept Come from users? (Oregon State Computer Engineering

Dept., 2000)

ii. Staff Chart: Gaining access to a staff (organizational) chart is a good way to start

examining behavioral issues in this phase and in system analysis and design

phases. It can provide a framework for organizing information and identifying

stakeholders. Go over the chart with someone to be sure it is up to date and you


incorporate informal dynamics.

iii. Top Leadership Support and Modeling: Scott Anthony, David Duncan and Pontus

M.A. Siren state in their Harvard Business Review article on innovation that

“Every project… should have a senior executive sponsor or champion who

believes in it deeply” (Scott Anthony, 2014). They call this a “shepherding

function” essential to preventing new projects from being smothered by the

demands of organizational life (Scott Anthony, 2014). Case studies have revealed

that to foster the success of information related processes “people must be

aligned with a vision and… the alignment must be from the top down… leaders

must… lead by example.” (Plessis, 2007)

iv. Dual value proposition for the user: User engagement with a system often

requires that the system “add value to staff’s everyday working environment….

Successful enterprises ensure this dual value proposition.” (Plessis, 2007)

v. Trust: Users must trust the system.

vi. Dedicated system staff member(s) with time to devote: Dedicated staff roles are

an information system success factor as they can help insure the process is

managed consistently in a structured way. (Plessis, 2007)

vii. Time and Training: How much time is available for each project phase including

ongoing maintenance? Who will perform tasks? Will users devote time to give

input, undergo training? Do formal training processes or personnel in charge of

training in administrative or technology areas exist in the organization and will

training be supported by top leadership?

viii. What can we learn from past successes or failures? What would it take to make

this wildly successful?

https://hbr.org/search?term=scott+anthony

https://hbr.org/search?term=david+duncan

https://hbr.org/search?term=pontus+m.a.+siren

https://hbr.org/search?term=pontus+m.a.+siren


Systems Analysis 1 of 2

Deliverable: System Requirements Hypothesis

(This Hypothesis Combined with the Deliverable of Analysis 2 (Stakeholder Research Findings) leads

to the creation of the Design Brief with System Requirements which is the deliverable of the complete

Analysis phase (parts 1 and 2 combined). Analysis is actually one phase but I have broken it into two

phases for the sake of organization of tasks. They do not occur strictly sequentially and you move back

and forth between them.)

If the feasibility analysis has led to a “go” decision, the project can justifiably move into the analysis

phase. Generally, the closer the involvement of the designer(s) with users, the better the chances of

creating a system that will meet user needs. The systems analysis phase of the SDLC involves:

“examination of the business problem in more detail… processes involved [include]… gathering

information about the existing system…” and delivering a set of requirements for the new system.

(Turban, 2009)

1. Determining Your Design Approach: Design approach for a project should be consciously chosen

and the best approach varies by project. The best designers know when to use each approach so

their solutions are holistic. Three approaches are described by Dan Saffer in the book Interaction

Design (Saffer, 2010) including Rapid Expert, Activity-Centered and User-Centered Design. Rapid

Expert Design is how most design occurs given limited resources. If the designer has enough

experience and the problem is familiar enough or simple enough this may be the way to go and

involves less analysis work. Activity-Centered design focuses on tasks and does not zoom out to

consider the problem from a larger perspective. User-Centered Design involves users from the

onset with lots of interaction and co-creation. Users guide requirements and preferences while

the designer facilitates translation of user goals into the design of the system (Saffer, 2010).

Depending on where you decide to go on this continuum of approaches will inform your choice of

tasks from those described below.

2. Framing the Problem: Zooming out can help you clarify what the actual problem is that needs to

be solved. Systems are embedded within the context of processes, training and other realities.

Some problems have fuzzy borders, many stakeholders and no immediately clear solution.

3. Project Timeframe: Saffer recommends project planning by setting an end date. Then “work

backwards… blocking out time in chunks for the various portions of the design process. The plan

should be… posted somewhere where the team can see it… Key dates and deliverables should be


made known and agreed to by the team.” (Saffer, 2010). The plan will be revised but can help

clarify things.

4. Conducting Traditional Research and Analysis Independently: Research can be done

independently. Much of these research findings will need to be confirmed and brought to life by

discussion with stakeholders using the tasks in the Analysis 2 Phase.

a. Document Analysis: webpages, brochures, flyers, forms, procedures, manuals, receipts,

and other documents can be reviewed.

b. Analytics: Review system or site analytics on page visits, visitor flows, users, login

patterns, transaction statistics and receipts and other quantitative data.

c. Information requirements: Information requirements include information needed by

whom, when and in what format (Turban, 2009).

i. Information Inputs and Outputs Table (Input, Personnel, Process Activities,

Outputs) (upfront, recurring).

d. Workflow Analysis. Draw your own and have users draw them as well and improve on

yours. Discuss the drawings and consider emotional elements.

5. Understanding the Market (Subject Area): Trade journals and white papers as well as

investigation into other businesses in the same market can help a designer get their bearings

(Saffer, 2010). What are the best practices recommended? Is another organization has the

process just right and is willing to educate or partner.

6. Usability Evaluation of Current and Competitive Systems: An understanding of classic design

principles goes a long way toward spotting usability issues. The following are some key concepts

to factor as well as a description of heuristics based evaluation which alone can produce major

insights.

a. Heuristic based evaluation is used and is defined as the judging of user-interface

compliance with key usability principles which have been validated (Nielsen, 1995). The

ten principles include System Condition visibility, relationship between system state and

reality, error prevention, aesthetics and minimalist planning, user ability to recognize,

diagnose and correct for errors, authentication, help, swift processing for both

experienced and novice users, familiar language and consistent visual cues (Nicola

Convertini, 2004). The three day heuristic evaluation conducted in the article from the

University of Bari (Nicola Convertini, 2004) provides an example of rapid design which

delivered significant value. This has relevance given the frequent constraints placed on

the design process in areas including time, money, expertise and coordinated

organizational support for design work (Saffer, 2010).


b. Design Principles to Supplement the ten Heuristics:

i. Visibility, and Feedback.

ii. Minimal Learning Curve and Existing mental maps : Conceptual models of

products allow users to make ongoing decisions without reference to instructions

by transferring “old knowledge to the new object” (Norman, 2002)

iii. Affordance: Affordance is defined as “the perceived and actual properties for the

thing, primarily those … that determine just how the thing could possibly be

used… Affordance provides strong clues to the operations of things.”

iv. Constraints: Constraints occur when “the world restricts allowed behavior” and

thereby the amount of knowledge in the head required for a particular situation

is reduced (Norman, 2002).

v. Permissions: Users should not be able to delete or irrevocably damage a

document. User permissions should be granted, updated and revoked

accordingly.

vi. Minimal Written Instructions: The above design techniques can be applied to

create a user experience environment that requires minimal written instruction.

Most of the knowledge is “in the world”, creating an intuitive experience

(Norman, 2002).


Systems Analysis 2 of 2

Deliverable: Design Brief with System Requirements

(Requirements must include 1. Strengths and weaknesses of existing system, 2. Functions

new system must have to solve the business problem and 3. User information requirements. The

Design Brief is created using the deliverables of both Analysis sections.)

1. Stakeholder Research: Stakeholder Research Complements and Improves Independent Research

and Analysis. As described by Shelley Evenson of Carnegie Mellon School of Design “ethnographic

methods can account for the complexity of service elements that are onstage, backstage, visible

and invisible in the service experience” (Saffer, 2010).

a. Determine Important Stakeholders: Find organizational charts and create a list of key

stakeholders you can talk to and survey. Who are key clients, users, leaders, consultants,

passionate about this issue? Determine this so you can leverage the information.

b. Partner with another researcher: “A second pair of eyes, ears and hands is immensely

valuable for observing, listening and recording, and for discussing and analyzing the

research data afterwards.” (Saffer, 2010)

c. Test Plan: A Test Plan should be devised containing neutral questions as well as test

routes “through the product that test functionality” (Saffer, 2010). Cognitive and

behavioral measures like “time taken to complete a task or the number of errors or

deviations from a critical path” can be recorded (Jefsioutine, 2006).

d. Stakeholder Observation and Interviews: (Go to them, talk to them): Don’t make subjects

come to you. Get out of the comfort of your office and observe activities in the

environment they are typically performed in. Observational and interviewing strategies

include “Fly on the wall”, shadowing (with or without asking questions along the way)

directed storytelling, role play, interviewing those not involved in the process, asking for

a tour of a subjects desk/computer etc. Have subjects tell their own stories in their own

manner directly to designers so that nuances can be captured. Activities can also include

asking subjects to draw or collage the life cycle of their experience and explain their work

when done. Ask them to explain the drawing but don’t tell them yin advance that yuou

will do this asking.

i. See Appendix for Stakeholder Interview Questions

e. Write stuff down: Capture with a camera, pen and pad etc. either during research or

directly after.


f. A summative “Opportunity Report” should record observations and patterns showing

trouble spots and suggestions for improvement (Saffer, 2010) can be created to inform

the design going forward. Analyze user mental models of current task structure (Oregon

State Computer Engineering Dept., 2000).

2. Surveys. Surveys have been described in detail already in the section on evaluating outcomes.

Both of these surveys appear in the Appendix of this paper.

a. System Usability Scale (SUS)

b. Supplementary Questionnaire

c. Analyze survey results

3. Create a Design Brief Based on the Research Tasks Above: This document is an initial analysis of the

problem with initial suggestions for solutions, technical constraints, timetable, deliverables, goals,

major stakeholder contact info etc. It is produced from research, interviews, competitive analysis

and used iteratively for capture and communication of information and further questions. For

system design, this can be the vehicle for presenting the deliverable of the Systems Analysis

phase which are the System Requirements.

a. Does the Brief present information in easy to understand visual format?

b. The following two tasks come from the Checklist for User-Centered Design (Oregon State

Computer Engineering Dept., 2000):

i. Did real users review and evaluate the design brief/initial design?

ii. Did designers clarify user feedback to arrive at a significant majority opinion?

c. The System Requirements presented with the Design Brief: The requirements should

include the following three elements (Turban, 2009):

i. Strengths and weaknesses of existing system

ii. Functions new system must have to solve the business problem

iii. User information requirements


Systems Design

Deliverable: Technical Design Specification

Systems Design considers Logical Design and Physical Design. Logical design includes what the

system needs to do in abstract terms including “inputs, outputs, processing, controls, security and IS

roles”. Physical Design which includes how the system performs using specific hardware, software and

procedures. Also included are interface design and a blueprint of how everything works together

(Turban, 2009).

Logical Design

1. Consider all touch points: Consider all points at which users will interact with the system including the

actual product and all related services, advertisements, referrals, support experiences, follow up contact

points etc.

2. Employ Usability Heuristics and Design Principles: “There is nothing more annoying than talking to

someone who doesn’t respond” (Saffer 131). See Analysis section for details.

3. Personnel Roles: Information Systems and other personnel involved in the system operation have

been defined.

4. Procedures and controls

a. Permissions and security

b. Collaborative Process best practices: Precise role definition: According to Lynda Gratton’s

article in the Harvard Business Review, “Cooperation increases when the roles of

individual team members are sharply defined yet the team is given latitude on how to

achieve the task”, Face time: The fostering of informal relationships and accountability

(Erickson, 2007).

Physical Design

5. Tool selection and acquisition based on system requirements: Internet based software is more and

more common and is sometimes called “cloud computing” or Software-as-a-Service (SaaS).

Vendors host applications and customers have no control over the software and access it over a

network, usually the internet. (Turban, 2009) Most system administrators feel that increasing

reliance on cloud computing is inevitable and the majority of businesses are adopting cloud

computing. (Axelrod, 2013) “There’s no need to huddle around the same computer of send files

back and forth over email if you want to collaborate with other people” stated the website

howtogeek.com. (Hoffman, 2014)

a. Typical System Requirements Considerations Checklist. This is one of the checklists at the

end of this paper.


b. Add Specific System Requirements (deliveable of Analysis Phase) to General considerations

c. Vendor Rating Tool worksheet: Not part of this paper.

d. Market Research: The identification of software alternatives often involves research using

trade journals, consultants experienced in the area, surveying peers in similar companies

and web research. (Turban, 2009) . Industry Analysis conducted by companies like

Gartner, the world’s leading IT research and advisory company present information on

factors such as market share, fit for various environments, sector adoption rates,

functionality, infrastructure and others.

6. Create the Technical Design Specification Deliverable: From the logical and physical designs and

the steps below are completed (Turban, 2009).

a. Both logical and physical design are reviewed with all appropriate stakeholders.

b. Did users specifically notice that their suggestions had been incorporated? (Oregon State

Computer Engineering Dept., 2000)

c. Did users specifically say the new design was better? (Oregon State Computer

Engineering Dept., 2000)

d. Revise the Specification as needed.

e. All appropriate stakeholders approve the specifications and formally acknowledge that

they are being now “frozen” to prevent scope creep leading to runaway projects over

budget and/or deadline or abandoned entirely.

7. The Frozen Specifications are the Deliverable for the build/development phase.


Development, Testing, Implementation, Operation and

Maintenance Phases Will Not Be Explored in Depth

These subsequent crucial phases are not within the focus of this paper but I will summarize them

briefly to place the other phases in context.

Development

In the development phase, depending on tool selection choices there will be varied degrees of

coding and customization required. Even Software-as-a-Service (SaaS) and Off-the-Shelf applications

require customization. I have personally used many SaaS applications including Google Apps, Wufoo

forms, content management systems and email marketing software. Many of these applications are very

flexible, providing you the tools to create applications according to your specifications without much if

any actual programming.

Testing

Testing includes both technical and functional testing by developers and user testing. I will not

review this topic again as it has been elaborated already in the previous “Criteria and Methods of

Checklist Evaluation” section of this paper.

Implementation

Implementation can be conducted via direct conversion (all at once) to the new system, pilot

conversion or phased conversion (to allow for evaluation before full direct conversion). Training is very

important to the success of the system and can occur through various means.

Operation and Maintenance

These phases are essential and include technical fixes, updates and the addition of new

functionality. They often use the most resources of all in the lifecycle of the system.

The Design Checklists

I’ve created five design checklists for this paper corresponding to the phases of Systems

Investigation/Feasibility Analysis, Systems Analysis (Part One), Systems Analysis (Part Two), Systems

Design and System Requirements. The Lists appear below. They are prototypes in need of use, testing

and validation.


SYSTEMS DESIGN CHECKLIST Revised 3/16/15 by Adam Wilson

> INITIATION > SYSTEMS INVESTIGATION/FEASIBILITY > SYSTEMS ANALYSIS

1 > ANALYSIS 2 > DESIGN > PROTOTYPE-TEST > REFINE > IMPLEMENT > OPERATE > NOTES

1. The system problem has been framed to clarify where the root of the actual problem lies.

2. This project has been ranked a top priority against others based on mission goals, beneficiaries and scope of impact.

3a. This project has been deemed technically feasible.

3b. Resources are available which make this project economically feasible.

3c. This project aligns with organizational and IT strategy based on documentation and confirmation with appropriate personnel.

3di. The initial concept or motivation for this project came from users or users have voiced a willingness to make changes to support this project.

3dii. The organizational staff chart has been obtained and reviewed to identify various project stakeholders.

3diii. A top leader with adequate authority is passionate about the project and has agreed formally to shepherd it through all phases including research, design, build, test, iterate, launch, training, operation. Who?

Who is passionate about this? How to engage them?

3div. Stakeholders have definitely been able to see potential value to their own everyday work life from this project.

3dv. Users trust of information systems and the leaders of this project enough that they will trust the system with their valuable information resources.

3dvi. A dedicated, qualified staff member has been allocated by top leadership to take ownership of this project.

3dvii. Formal training processes and personnel exist and have been authorized to insure proper training on the new system.

3dviii. Past failures and successes have been shared by users and stakeholders and this project involves factors that predict it will succeed and can mitigate risks.

DELIVERABLE: GO OR NO-GO DECISION (please circle) 1. Will not proceed based on these criteria (list): 2. Looks promising, will be deferred for Review on: 3. May proceed, given the following additional criteria are met: 4. Will proceed beginning:



> INITIATION > INV./FEASIBILITY > SYSTEMS ANALYSIS 1 of 2 > ANALYSIS 2 > DESIGN >

PROTOTYPE-TEST > REFINE > IMPLEMENT > OPERATE >

NOTES

TRADITIONAL RESEARCH

1. Have a design approach been consciously chosen?

2. The system problem has been framed to clarify where the root of the actual problem lies.

3. Tentative Key dates and deliverables have been agreed to by the team and the tentative timeline has been posted.

4a. Key documents have been identified, collected and analyzed.

4b. Have you reviewed relevant quantitative information such as analytics, user logins or transaction statistics?

4c-d. A hypothetical workflow analysis chart and/or table of information inputs, people, process, and outputs has been created to go over with stakeholders.

5. Have you reviewed trade journals, industry analysis and competing businesses and identified best practices?

USABILITY and DESIGN

6a. Have designers conducted a personal heuristic evaluation of the existing system?

6b. Designers have evaluated the existing system using design criteria, identified issues and recorded ideas for their correction.

DELIVERABLE: System Requirements Hypothesis



> INITIATION > INV./FEASIBILITY > SYSTEMS ANALYSIS 1 > SYSTEMS ANALYSIS 2 of 2 >

DESIGN > PROTOTYPE-TEST > REFINE > IMPLEMENT > OPERATE > NOTES

STAKEHOLDER RESEARCH

1a. Have you discussed the organizational chart or similar

document to identify key stakeholders?

1b. A second researcher has been recruited for a second

pair of eyes and discussion of findings.

1c. A Test Plan has been devised containing neutral

questions for stakeholders and planned research

activities.

See stakeholder interview questionnaire.

1c. A test route through the product has been designed

“through the product in order to test functionality.

1d. Users have been observed in their own environments

with neutral questioning and recording with text, photos,

etc. The following activities have been conducted (please

conduct at least 2 and circle them at right).

Fly on the wall, shadowing, directed storytelling, role play, interviews, tour of computer and workflow etc., user workflow drawings explained afterward.

1f. A summative “Opportunity Report” has been created

from observations and identified patterns.

2a. The System Usability Scale (SUS) or alternate usability

questionnaire has been administered and scored.

SUS Supplementary Questionnaire available.

2c. Questionnaire results have been analyzed.

3. DELIVERABLE (OF ANALYSIS 1 & 2): Design Brief with System Requirements Stakeholder Research analysis has informed the Design Brief.

Traditional Research hypothesis have informed the Design Brief.

Information has been structured in a visual and easy to understand way in the Brief.

Strengths and weakness findings about the existing system are contained in the brief.

Functions the new system must have are included.

User information requirements are included.

Has a meeting been scheduled with appropriate stakeholders including real users to go over the Brief

Did designers clarify user and stakeholder feedback about the Brief to arrive at a significant majority opinion,

revise and agree on the system requirements and other elements of the Brief.



INV./FEASIBILITY > ANALYSIS 1 > ANALYSIS 2 > SYSTEMS DESIGN > PROTOTYPE-TEST > REFINE >

IMPLEMENT > OPERATE >

NOTES LOGICAL DESIGN

1. All touch points at which users will interact with the system including related services have been mapped holistically.

2. Usability Heuristics and Design Principles have been applied to each interface and aspect of the system.

Information Systems and other personnel involved in the system operation have been defined.

Appropriate user permission profiles have been created.

Security measures both behavioral and technical have been designed for information backup and access.

Collaborative issues such as precise role definitions, face time and accountability have been designed.

PHYSICAL DESIGN

System Requirements from the Analysis phase have been added to the general system requirements table.

See table in Appendix.

Market research has identified software alternatives.

Potential vendors have been rated using Requirements table Consider using the Vendor Rating Tool worksheet.

DELIVERABLE: FROZEN TECHNICAL DESIGN SPECIFICATIONS Both logical and physical design have been reviewed with all appropriate stakeholders including users.

Did users specifically notice that their suggestions had been incorporated?

Did users specifically say the new design was better?

Specification have been revised as needed.

All appropriate stakeholders have approved the final specifications and formally acknowledged they are being

“frozen” to prevent scope creep.


Generic System Requirements

Checklist Taken from Introduction to Information Systems: Supporting and Transforming Business (Turban, 2009)

Software Under Consideration: _____________________

Project specific requirements have been added to the generic requirements below.

1. Functionality

2. IT Strategy Alignment

Usability

Availability of Quality Documentation and Training Resources

Vendor Reputation, Industry Analysis

What do competitors use?

Cost. Free Trial?

Integration with existing applications

Necessary Hardware, Software and Networking Resources and compatibility:

Implementation, Updates

Security

usable for other processes

Long term product availability

Revised 3/16/15 by Adam Wilson


Reflection

Surprising Finding

The single most surprising thing I begin to see in the 12 sources and other research on this

project is the power of simple, precise interventions to dramatically impact extremely complex situations.

I was pleasantly surprised About the Checklists I’ve been using for Administrative Work

My prior work in administration has been highly checklist driven. Although I had not yet studied

checklist best practices the lists I created for various aspects of my job follow some of them. They include

easy to miss and critical items, fit on one page, are quickly reviewable and ordered sequentially and/or

logically, were used collaboratively by administrative staff to track and communicate among team

members and were online to facilitate sharing and frequent updates (Katherine Radeka, 2015).

Comprehensive instructions were hyperlinked to items to facilitate access without overloading the

checklist itself. The lists were built in Google Sheets and the Design Checklists at the end of this paper

may eventually be migrated there as well. My office management checklists helped make me a very

reliable and consistent administrator and I would like to expand that kind of competency into the slightly

less routinized and more creative space of design.

The Checklists in the Paper are Prototypes

These checklists have not been trialed and validated with front line users either in a simulation or

in a real situation and revised as needed. They are merely prototypes.

Submitting the Checklists to Design Scrutiny

While the design checklist should be subjected to the best practice standards of creating checklist

they should then again be subjected to the best practice standards of design. Under such analysis I

observe that I prefer online checklists which can be shared and accessed remotely. I suspect the next

iteration of them will involve migrating them to Google Sheets.


References

Aline Chevalier, M. Y. (2002). We site designs: Influences of designer's expertise and design constraints.

International Journal of Human-Computer Studies, 57-87.

Allen, D. (2002). Getting Things Done: The Art of Stress-Free Productivity. Penguin Books.

Asaf Degani, E. L. (1991). Human factors of flight-deck checklists: The normal checklist. Retrieved from

NASA Technical Reports Server (NTRS): http://ntrs.nasa.gov/search.jsp?R=19910017830

Axelrod, G. (2013, October 23). 47 Stats You Need to Know About the Google Apps Ecosystem. Retrieved

from bettercloud.com: http://blog.bettercloud.com/google-apps-stats/

Brooke, J. (1986). SUS - A quick and dirty usability scale. Farley, UK: Redhatch Consulting, Ltd.

Coleman, M. (2015). Process. Retrieved from megancoleman.com:

http://www.megancoleman.com/process/

davidallengtd.com. (2014). Getting Things Done Daily Planner. ataglance.

Erickson, L. G. (2007). Eight Ways to Build Collaborative Teams. Retrieved from Harvard Business Review:

http://www.harvard-

samsung.net/hbspCourse/hmm11/team_leadership/base/resources/EightWays_BuildCollaborativ

eTeams.pdf

Gawande, A. (2009). The Checklist Manifesto. New York: Metropolitan Books.

Geracie, G. (2010). Take Charge Product Management. Actuation Press.

Hoffman, C. (2014, February 23). How to Collaborate on Documents Over the Internet. Retrieved from

howtogeek.com: http://www.howtogeek.com/183176/how-to-collaborate-on-documents-over-

the-internet/

Jefsioutine, J. K. (2004). Designing in the E-Society: The Experience Design Framework. Proceedings of the

IADIS International Conference (pp. 1103-1107). Avila, Spain: e-Society. Retrieved February 1,

2015, from

http://www.mgt.ncu.edu.tw/~ckfarn/doc/conference/IADIS%20ES2004_Vol2.pdf#page=415


Jefsioutine, J. K. (2006). Design Frameworks. Encyclopedia of Human Computer Interaction. (C. G. (ed),

Ed.) University of Central England, UK: IGI Global. Retrieved February 1, 2015, from

http://common.books24x7.com.ezproxy.depaul.edu/toc.aspx?bookid=14703

Katherine Radeka. (2015). A Checklist for Designing a Checklist. Retrieved from

leantechnologydevelopment.com:

http://leantechnologydevelopment.com/system/files/Checklists-Letter.pdf

Katherine Radeka, Whittier Consulting Group, Inc. (2015). A Checklist for Designing a Checklist. Retrieved

from leantechnologydevelopment.com:

http://leantechnologydevelopment.com/system/files/Checklists-Letter.pdf

Lewis, J. R. (1995). IBM Computer Usability Satisfaction Questionnaires. International Journal of Human

Computer Interactions, 57-78.

McKeown, B. J. (2013). Assessing and manageing risk with people with physical disabilities: the

development of a safety checklist. Health, Risk and Society, 162-175.

Nicola Convertini, A. M. (2004). Usability Models on Institutional Portals: A Case Study at The University of

Bari., (pp. 1099-1102).

Nielsen, J. (1995, Janurary 1). 10 Usability Heuristics for User Interface Design. Retrieved from

nngroup.com: http://www.nngroup.com/articles/ten-usability-heuristics/

Norman, D. A. (2002). The Design of Everyday Things. New York: Basic Books.

Oregon State Computer Engineering Dept. (2000). Checklist for User-Centered Design. Retrieved from

web.engr.oregonstate.edu: https://web.engr.oregonstate.edu/~pancake/cs252/checklist.html

Palmius, J. (2007). Criteria for measuring and comparing information systems. Proceedings of the 30th

Information Systems Research Seminar in Scandinavia.

Plessis, M. d. (2007). Knowledge Management: what makes complex implementations successful? Journal

of Knowledge Management, 91-101.

Saffer, D. (2010). Designing for Interaction: Creating Innovative Applications and Devices. Berkeley: New

Riders.

Scott Anthony, D. D. (2014, December). Build an Innovation Engine in 90 Days. Harvard Business Review.


Turban, R. K. (2009). Introduction to Information Systems. Hoboken: Wiley and Sons.

usability.gov. (2015). System Usability Scale (SUS). Retrieved from usability.gov:

http://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html

Warning, P. (2009). Information Seeking and Stoppping Among Undergraduate Interns. Proceedings of the

2009 INternational Conference on Knowledge Management. Hong Kong: S.K.W.

World Health Organization. (2009). Surgical Safety Checklist. Retrieved from who.int:

http://www.who.int/patientsafety/safesurgery/checklist/en/


Appendix A

System Usability Scale

Instructions: Use after the respondent has had an opportunity to use the system being evaluated, but before any debriefing or discussion takes place. Respondents should be asked to record their immediate response to each item, rather than thinking about items for a long time. All items should be checked. If a respondent feels that they cannot respond to a particular item, they should mark the center point of the scale. Strongly Strongly Disagree agree

1. I think that I would like to use this system frequently

2. I found the system unnecessarily complex

3. I thought the system was easy to use

4. I think that I would need the support of a technical person to be able to use this system

5. I found the various functions in this system were well integrated

6. I thought there was too much inconsistency in this system

7. I would imagine that most people would learn to use this system very quickly

8. I found the system very cumbersome to use

9. I felt very confident using the system

10. I needed to learn a lot of things before I could get going with this system

Scoring the SUS SUS yields a single number representing a composite measure of the overall usability of the system being studied. Note that scores for individual items are not meaningful on their own. To calculate the SUS score, first sum the score contributions from each item. Each item's score contribution will range from 0 to 4. For items 1, 3, 5, 7, and 9 the score contribution is the scale position minus 1. For items 2,4,6,8 and 10, the contribution is 5 minus the scale position. Multiply the sum of the scores by 2.5 to obtain the overall value of SU. SUS scores have a range of 0 to 100. © Digital Equipment Corporation, 1986.


Appendix B

Stakeholder Interview Sample Questions Taken from Interaction Design (Saffer, 2010)

1. Who are you and what is your role in the org

2. Why is this project important to you? To the org? (internal context)(look for unstated

goals/agendas)

3. What would make a successful project? (metrics)

4. Has anyone ever tried to address this problem before? (Why?)

5. What doesn’t this project cover? (Why?)(scope)(What are constraints, boundaries can’t cross?)

6. If we could only do only one thing with this project, what would that be? (Why?)

7. How could this project affect your day-to-day life? (Organizational expectations, part of life at…)

8. Are there any issues about this project I should be aware of?

9. What are the risks in doing this project? What could make it fail?

10. What are your competitors doing in this space?

11. “What would customers do (instead) if all the traditional competitors went away?

12. Who else should I talk to about this project?

I’ve added these questions taken from a Checklist for User-Centered Design (Oregon State Computer

Engineering Dept., 2000)

1. What are your work goals related to the project?

2. How do you currently structure those goals into tasks?

3. Why do you do it that way?

4. How do you wish you could do it?


Appendix C

Stakeholder Questionnaire Items Supplemental to the SUS

The following questions are not directly addressed by the SUS. They could be part of a separate evaluation questionnaire with follow-up interactions. I have limited them to 25 per the recommendations by SUS article author John Brooke (Brooke, 1986). They are designed to be answered using a Likert scale from 1-5 strongly disagree to strongly agree.

Regarding expanding software choices for users and use on multiple devices.

1. The system is my preferred way to perform the tasks it is designed for.

2. The system has worked well on all my devices.

From A 1995 Computer System Usability Questionnaire developed by IBM (Lewis, 1995).

1. I can effectively (and efficiently) complete my work

2. Whenever I make a mistake I recover easily and quickly.

3. The system was intuitive or Instructions were visible or easily retrievable

4. List the most negative aspects: (3 open ended fields to complete)

5. List the most positive aspects: (3 open ended fields to complete)

The Experience Design Framework “includes attitude measurements and emotional response” as well as social and pleasurable elements at the top of a “hierarchy of user needs” (Jefsioutine, 2006).

1. The system can help divide workload to the right people

2. I would definitely recommend this system to others 3. Working with this system would support positive relationships with others. 4. This system could help users work together well as a team. 5. This system could support improved relationships between different departments.

The article Criteria for Measuring and Comparing Information Systems (Palmius, 2007) contributes these questions related to organizational and knowledge management impacts of the system.

1. The system will help the organization capture significant knowledge embedded in individuals.

2. The system makes this knowledge easy to access and use by the appropriate people.

3. Processes are in place for leveraging this knowledge to create value. 4. People are regularly using the processes provided by the system for knowledge capture. 5. The system provides strong ROI. 6. The system significantly enhances the competitive advantage of the organization. 7. The system significantly enhances the satisfaction of external customers.

Additional Questions regarding trust and sustainable operation and support.

1. I am very comfortable that critical information in the system is safe.

2. Updating the system with information to keep it very current will be quite feasible. 3. We have the right people to maintain the system in good working order long term. 4. Problems with the system are addressed rapidly to my satisfaction. 5. My satisfaction and/or effectiveness with the system would be very much enhanced by these

changes or enhancements (open ended question):

Adam Wilson AP Final Project REVISED FINAL DRAFT

Documents

Transcript of Adam Wilson AP Final Project REVISED FINAL DRAFT