Cozi Usability Test Report - User Experience Design ...€¦ · The following report summarizes a...

Cozi Usability Test Report

Kimberly Fulton, Justin Hamacher & Carlos Marin Burgos • December 10, 2010

Family life. Simplified.

EXECUTIVE SUMMARY 3

INTRODUCTION 5

Product Description 5

Study Purpose 5

Objectives & Problem Statements 5

STUDY DESCRIPTION 6

Participant profile 6

Methodology 6

Team roles 6

Test environment & equipment 6

Materials 6

Methods and procedures 7

Tasks overview 8

FINDINGS AND RECOMMENDATIONS 10

Successes 10

Opportunities for improvement 10

Interesting participant suggestions 15

STUDY EVALUATION 16

What worked 16

What could be improved 16

Additional areas for exploration 17

APPENDICES 18

A: Participant screener & sessions schedule 19

B: Background questionnaire form 20

C: Orientation script 21

D: Consent form 22

E: Participant task guide 23

F: Moderator task guide 29

G: Post-task questionnaire form 32

H: Post-test questionnaire form 33

I: Background questionnaire answers 34

J: Data collection table 35

3Kimberly Fulton, Justin Hamacher & Carlos Marin Burgos

Executive summary

The following report summarizes a four-participant usability study focused on the Cozi web client. Two main objectives guided this study:

1. Evaluating the usability of the Cozi web client in the context of the new user experience.

2. Investigating the mental models new users form as they first begin to interact with and learn about Cozi.

A review of Cozi’s consumer demographics helped us develop a set of criteria intended to represent the typical Cozi user. We recruited four participants based on this criteria, making sure to restrict ourselves to individuals who had no prior experience using Cozi. In total, four participants were tested. They com-pleted 19 tasks, beginning at the Cozi marketing website and continuing through the four main product areas: calendar, to do lists, shopping lists, and journal. Participants were instruct-ed to practice thinking-out-loud as they worked through their tasks. Data was collected in the form of task completion rates, error rates, Likert-scale ratings, and responses to open-ended interview questions. Morae usability testing software was used to capture audio and video recordings of the participants and their computer screens as they worked.

The test results revealed five main success areas:

1. Participants understood what Cozi was just from view-ing the marketing website homepage.

2. Regardless of the method they employed, all the par-ticipants were all able to successfully input a given set of repeating appointments.

3. Half the participants found the “Weekly routines” page when they first started filling out the calendar, despite the fact this page is hidden under a drop-down menu.

4. Participants rated the ease of creating reminders 6.5 out of 7 (7 being best).

5. Despite the fact that the calendar is easily the most com-plex feature in Cozi, it was named by two of the four par-ticipants as being the easiest part of Cozi to use, while a third participant named a specific aspect of the calendar as the easiest.

In addition to highlighting successes, the study also made evi-dent several opportunities for improvement. In reviewing and analyzing the test data, we identified a set of nine issues and formulated a series of associated recommendations:

1. Participants were confused about how to create a new to do list. Recommendations for solving this issue include using more explicit labeling and making potential ac-tions that users can perform more apparent.

2. Participants experienced confusion when creating ap-pointments that repeated on a monthly basis (e.g. on the first Friday of every month). Mitigation of this issue can be achieved through a simple rewording of the as-sociated repeating appointment pop-up menu.

3. Participants did not understand that, in addition to the appointment title, they could also type appointment details into the “Type appointment” field at the top of the calendar. Recommendations regarding this issue in-clude making “What can I type?” more discoverable and, when the “Type appointment” field is active, in-cluding some sort of text-box watermark that says: “Type appointment title and details here”.

4. Participants’ mental models regarding to do lists in-volved a connection between tasks and dates or the cal-endar. In order to avoid certain points of confusion and to increase user satisfaction, we recommend embracing this mental connection by supporting an actual rela-tionship between the to do list feature and the calendar.

5. Participants did not see the error message when it appeared on the “About your household” page. This is-sue can be rectified by moving the error message next to the entry field where the error occurred.

6. Participants could not figure out how to make list sub-headings. This issue can be resolved by adding a tool tip that explains this capability.

7. Participants did not understand why they would enter their mobile phone numbers into the “About your household” page. A simple explanation added to the link text and the associated mobile number entry page can help users understand the purpose behind provid-ing Cozi with this data.

8. Participants did not understand why they would enter their postal codes on the “About your household” page. Similar to the mobile phone numbers, the addition of a simple explanation can alleviate this issue and help us-ers understand the benefit of entering this information.


9. Participants had a hard time seeing how to move for-ward from the shared journal webpage preview. Recom-mendations for addressing this issue include making the wizard bar and “Yes, share our journal!” button more noticeable.

Although this study was conducted with a small set of only four participants, the results clearly demonstrate that there are sev-eral issues users might face in their initial experience with using Cozi. In accordance with Cozi’s iterative design method, we recommend addressing the issues outlined above and follow-ing up with additional usability tests in the future.


PRODUCT DESCRIPTIONCozi is an online organizer designed specifically to help busy families “simplify their lives.” Intended to be a single hub that the whole family can access, Cozi offers a feature-rich calendar as its centerpiece tool, in addition to shopping lists, to do lists, and a family journal. In support of these features Cozi has developed a number of mobile access solutions, including the ability to send shopping lists or schedules via text message, or the option to call a toll-free number and have them read aloud. A mobile client (m.cozi.com), iPhone app, and Android app are also available, providing on-the-go access to the four main areas of the product. Despite the fact that the company itself is a small start-up with limited resources, the Cozi team is constantly working to update and improve their product. They operate under an ad-based revenue model that allows them to offer their software to consumers for free. In order to generate more ad impressions and achieve their goal of becoming a profitable business, Cozi is focused on not only signing up new users, but increasing 30-day retention rates (the number of new users who, 30 days after creating accounts, are still actively using the product).

STUDY PURPOSEBecause the acquisition and retention of new users is so crucial to the company’s success, evaluating usability in the context of the new user experience was the primary focus for our study. Another factor we took into consideration was Cozi’s ongoing usability testing program, which meant that the product had already undergone a substantial number of tests in the past, and would continue to undergo more in the future. In order to maximize the usefulness of our own study, we decided to inves-tigate the mental models new users form as they begin to learn about and interact with Cozi. This decision was guided by our initial meeting with Cozi’s Vice President of User Experience, Jan Miksovsky; he highlighted mental models as something he was interested in investigating, and noted that he had yet to do so in his own studies. By kicking off this exploration ourselves, our study can help guide the company in future usability tests. The foremost purpose of our study, however, was to sim-ply introduce ourselves to the process of conducting usability tests, and to learn as much from the experience as possible. Thus, our study was somewhat atypical compared to more pro-fessional studies; we optimized it in such as way as to not only gather useful data, but to ensure that each team member had sufficient opportunity to experience and learn about all of the different elements of usability testing.

Introduction

OBJECTIVES & PROBLEM STATEMENTS

OBJECTIVE 1: Assess the usability of Cozi in context of the ini-tial new user experience.

1. Do users notice and/or click on the elements intended to help them start getting set up?

2. Calendar

a. Do participants successfully discover and use the calendar settings to begin filling out their calendar?

b. Can they successfully add a school calendar?

c. Can they locate desired dates and information?

d. Can they successfully use the natural language ap-pointment entry field?

e. Do they discover reminders?

3. To do / shopping lists

a. Can participants easily create a new list?

b. Can they assign a list to a specific family member?

c. Can they identify how to send lists as a text message?

d. Can they easily organize and edit lists and list items?

4. Journal

a. Can they create a new journal entry with a photo?

b. Can they set up their journal as a shared webpage?

OBJECTIVE 2: Explore the mental models new users form as they first begin to interact with Cozi.

1. How do new users’ mental models develop?

2. Do they understand that a single Cozi account is intend-ed to be shared among family members?

3. Do they understand how Cozi works with text messages?

4. Are they concerned about the privacy of their data?

5. Do they understand where their data is being stored?

6. Do they feel like the payoff of Cozi is worth the effort?

7. Do they feel uncertain about anything as they begin to interact with the product?


METHODOLOGY

TEAM ROLES

• Moderator: Interacted with the participants and admin-istered scenarios, tasks, and questionnaires. Each of us acted as moderator at least once.

• Note-Taker: Captured data and notes from the tests. Each of us acted as note-taker at least once.

• Tech Manager: Ensured the equipment and Morae soft-ware was set up and running properly. We intended to be collectively responsible for this role, but certain issues prevented us from completely fulfilling this goal. Justin ended up being the primary tech manager.

• Project Manager: Delegated to other team members and ensured participants and the workspace were arranged. We were all collectively responsible for this role.

TEST ENVIRONMENT & EQUIPMENTOur tests were conducted in a private studio (see Figure 1 on the following page for the room layout), which ensured a neu-tral, distraction-free environment. The studio was located near our participants’ homes, thus making it easier for them to take part in our study. Equipment included:

• Testing computer: Participants were provided a PC laptop computer running Windows 7, Internet Explorer 8, and Morae Recorder. The laptop’s built-in webcam captured audio and video recordings of the participants as they worked through their tasks.

• Observing computer: We used a MacBook Pro installed with Bootcamp in order to run Morae Observer. A large monitor was connected to facilitate easy viewing by our entire team.

MATERIALSAdditional materials needed to facilitate our study included:

• Background questionnaire: Collected background infor-mation about the participants (Appendix B).

• Orientation script: Was read aloud to each participant in order to explain testing procedures (Appendix C).

• Consent form: Obtained written permission to gather audio and video recordings of the participants, and explained that all information collected would remain confidential (Appendix D).

Study description

PARTICIPANT PROFILESCozi’s user demographic encompasses families of all shapes and sizes, including married couples as well as divorced cou-ples or single parents. The common thread is that they’re all busy juggling work, kids’ school and activity schedules, social engagements, and all the other little things necessary to keep a family running smoothly. Technology plays a part in these fami-lies’ lives; at minimum, they have internet access at home and use it on a regular basis. Within this overarching demographic are certain charac-teristics that clearly stand out as representing the typical Cozi user. We used this information to guide our selection of partici-pants: 42% of Cozi families have two kids; 81% include a mar-ried couple; 90% have one+ children between the ages of 5–15; and 78% have kids who collectively spend four+ per week in extracurricular activities. Of Cozi’s primary users (the people who sign up and create Cozi accounts): 48% work full time; 62% are between the ages of 35–44; and 85% are female. Moms often act as the chief managers of their families, so it’s little surprise that they would be the ones most frequently creating and maintaining Cozi accounts. However, for many moms, one of the benefits to using Cozi is that it helps them coordinate family management with their husbands more eas-ily. This means that dads’ buy-in to using Cozi can greatly influ-ence the product’s retention and active rates, making the dad user category one that shouldn’t be neglected. Thus, we made sure to test Cozi with the same number of dad participants as mom participants. We recruited four participants in total:

P1: MOM (WIFE OF P2), ARTIST, 35–40One 8 year old daughter with one extracurricular activities.PC/Google Chrome user, with an Android smartphone.Spends “too much” time online, uses Google Calendar

P2: DAD ( HUSBAND OF P1), INDUSTRIAL DESIGNER, 35–40 One 8 year old daughter with two extracurricular activities.PC/Google Chrome user, with an Android smartphone.Spends about 15 hours online per week, uses Google calendar.

P3: MOM, ARTIST/GRAPHIC DESIGNER, 35–40One 8 year old daughter with two extracurricular activities.Mac/Safari user, with an iPhone.Used to spend about 25 hours per week online, but less now.

P4: DAD, LANDSCAPE FOREMAN, 35–40One 5 year old daughter with one extracurricular activity.Mac/Safari user, with an older Nokia cellphone.Spends about an hour online each day.


• Participant task guide: Contained 19 tasks and associ-ated follow-up questionnaires (Appendices E, G and H).

• Moderator task guide: Contained 19 tasks and their as-sociated follow-up interview questions. (Appendix F).

• Data Collection Form: A simple spreadsheet used to col-lect and organize test data (Appendix I).

METHODS & PROCEDURESThe following methods were used to observe and analyze par-ticipant behaviors as they interacted with the product:

• Background questionnaire •Morae recordings •Thinking out loud •Post-task/post-test questionnaires •Open-ended, post-task/post-test interview questions

Our usability study was conducted with four participants in a private office studio. Prior to arriving for the study, participants were asked to complete a background questionnaire via email. This questionnaire covered information about the participants that we deemed relevant to their perception of and interaction with Cozi. Each session lasted about 90 minutes in total. Participants were greeted at the studio door and led inside, where the moderator situated them in front of the testing computer and provided them with a task guide. They were then given an ori-entation, which was read aloud by the moderator from a predetermined script. The orientation included filling out a consent form, as well as instruction and practice in the Think-ing-Out-Loud method. The moderator then began taking the participants through a set of exercises designed to produce data relevant to our study questions. Because the marketing website (cozi.com) is where many users first begin interacting with Cozi—and thus where they first begin forming their associated mental models—it was the starting point for our testing tasks. From here, partici-pants continued through a series of tasks within the product itself. These tasks covered the four main feature areas of Cozi: the calendar, to do lists, shopping lists, and journal. By con-ducting our study across the marketing site and the four princi-pal product areas, we were able to acquire a more complete illustration of the participants’ mental models. Tasks were completed on a PC laptop computer set up with a built-in webcam and Morae usability testing software. The Morae software was used to capture audio and video re-cordings of the participants, as well as their on-screen activity and keyboard and mouse input data. As participants worked through their given tasks, they were asked to practice thinking out loud. This technique, combined with in-depth interview questions, allowed us to gain insight into participants’ thought

processes. The in-depth interview questions were administered throughout the test, occurring immediately after pertinent tasks were completed and the experience was still fresh in par-ticipants’ minds. We also conducted brief post-task question-naires that asked participants to rank the ease with which they completed the preceding task or tasks. These questionnaires were only given when we deemed them necessary and appro-priate, and not after every single task. At the end of the session, participants were asked to com-plete a post-test questionnaire evaluating their overall reaction to the product. A debriefing session followed, providing the moderator with the opportunity to ask follow-up questions. For their involvement, we offered participants a token of appre-ciation in the form of Cozi swag and free babysitting services.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .FIG. 1: Testing environment setup (not to scale)

Participant

Moderator

Notetaker /Tech Operator

Notetaker


task problem statements* data collected collection method

1

Poke around the marketing

website until you’ve gained

an understanding as to what

Cozi is.

1.1 Mental model development

1.2 Understanding of how

accounts are shared

1.5 Understanding of where data

is stored

1.6 Payoff vs. effort of using Cozi

1.7 User uncertainties

• What users read/click on

• Quotes

• Thinking out loud

• Morae recordings

• Probing questions

2Sign up for Cozi and look over

the family homepage.


2.1 Discovery/use of hints


• Quotes




3

Fill out the “About Your

Household” page using the

provided information.


1.3 Understanding of how Cozi

works with text messaging

1.4 Privacy concerns

• Quotes • Thinking out loud


4Poke around the calendar

page, decide where you’d

go first.




• Quotes




5

Enter the provided series

of repeating appointments.



2.2a Discovery/use of calendar

settings


• Error rate

• Task completion

• Quotes

• Overall reaction




• Questionnaire

6

Add the listed school

calendars.

2.2b Ease of adding school

calendars


• Error rate

• Task completion




• Observation & note-taking

• Questionnaire

7

Find a timeslot to schedule the

given appointment.

2.2c Locating desired dates and

information in the calendar


• Error rate

• Task completion





• Questionnaire

8

Enter the aforementioned

appointment using the “Type

appointment” field.

2.2d Usability of the natural

language appointment

entry field


• Error rate

• Task completion





• Questionnaire

9

Create a reminder for this

appointment.

2.2e Discoverability of

appointment reminders


• Error rate

• Task completion





• Questionnaire

TASKS OVERVIEW

* Refer to the Objectives & Problem Statements section for unabbreviated problem statements. The numbers mark which part of the section to refer to (ex: 2.2a refers to objective 2, calendar, problem statement a).


task problem statements* data collected collection method

10

Create a new to do list with

the provided data.

2.3a Ease of new list creation • What users click on

• Error rate

• Task completion





• Questionnaire

11

Assign this list specifically to

your spouse.

2.3b Ease of assigning to do lists

specific family members

• What users click on

• Error rate

• Task completion





• Questionnaire

12

Identify a way to alert your

spouse you’ve created this list.

2.3c Discoverability of “send list

as text message” feature


• Error rate

• Task completion





• Questionnaire

13

Create a new shopping list

with the provided data.

2.3a Ease of creating a new list • What users click on

• Error rate

• Task completion





• Questionnaire

14

Make this the first list. 2.3d Organizing /editing lists • What users click on

• Error rate

• Task completion





• Questionnaire

15

Group the kids’ items apart

from the other items.

2.3d Organizing /editing lists • What users click on

• Error rate

• Task completion





• Questionnaire

16

Delete the indicated items. 2.3d Organizing /editing lists • What users click on

• Error rate

• Task completion





• Questionnaire

17

Look over the journal page. 1.1 Mental model development



• Quotes





• Questionnaire

18

Create a new entry with the

provided data.

2.4a Ease of journal entry creation • What users read/click on

• Error rate

• Task completion





• Questionnaire

19

Share your journal as a

webpage.


1.4 Privacy concerns

2.4b Ease of journal webpage

setup


• Error rate

• Task completion

• Quotes






• Questionnaire


Findings and Recommendations

SUCCESSES

1. All 4 participants understood what Cozi was after viewing the homepage of the marketing website.

2. 2 of our 4 participants found the “Weekly routines” page, which is a good place to start when filling out the calendar for the first time. The fact that half our users found this page isn’t bad considering it’s hidden in a drop down menu.

3. Regardless of the method the used—filling out the “Weekly routines” page or simply entering appointments directly into the calendar itself—all 4 participants were able to success-fully input the given set of repeating appointments.

4. On average, participants rated the ease of creating appoint-ment reminders a 6.5 out of 7, with 7 being the best.

5. The calendar, which is the most complex feature in Cozi, was named by 2 of our 4 participants as being the easiest part of Cozi to use, while a third participant named a specific aspect of the calendar as the easiest.

FINDING 1Participants had difficulty creating a new to do list.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

SUPPORTING DATA

Results from task 10: When asked to create a new to do list with a given title and set of tasks, 3 of the 4 participants typed the list title into the task entry field in the default “Shared To Do” list, rather than typing in the list title field or simply creat-ing a new list (see Fig. 2). The one participant who did not do this partially completed the task by successfully entering the given task items into the default “Shared To Do” list (he did not, however, change the list title. He probably forgot about this part of the directions; we should have reminded him). Of the 3 participants that had trouble: P4 eventually fig-ured it out after noticing the other lists on the page; P1 eventu-ally figured it out after seeing the “David To Do” list, partially completing the task by successfully entering the task items into this list; and P3 got so confused she was unable to even par-tially complete the task.

Previous usability tests: Similar confusion has occurred in past usability studies, such as those from 12-17-09 and 11-12-09. While these studies are fairly old, to do lists haven’t changed significantly since, so the findings still apply.

Ease ratings: Compared to most of the other tasks in our study, the to do list tasks rated lower on the “worked as expected” scale. On average, our participants rated these tasks 4 out of 7 (7 being best). Out of the 3 participants who had trouble, they gave these tasks an average rating of 3.3 out of 7.

Quotes: “ Did I create a list? I don’t know. I was assuming that if I added

an item that that was a list, but now I can’t go into it. I’m hav-ing a hard time figuring out how to add things to the list.” –P1

“ I don’t know if this list is going to work as good as the list I plastered to the kitchen cabinet with a pen right by it.” –P1

“ I’m wondering how I open the list to add things.” –P3

RECOMMENDATIONS

Change the default list titles: Add “List” on the end of “Shared To Do” and the other default list titles. This will help users un-derstand that the list pane is not a folder of lists, but a list itself.

Reword “Click to add an item”: Change the hover text “Click to add an item” to say “Click to add a task” This will further rein-force the proper mental model regarding how the lists function.

Apply the Visibility Principle: “Systems are more usable when they clearly indicate their status, the possible actions that can be performed, and the consequences of the actions performed” (Lidwell, 202). This can be applied to empty lists by including text to the effect of: “Your Shared To Do List is currently empty. Click to add a task.”

Add explanatory introductory text: Similar to the Cozi calen-dar, include new user introductory text that explains how to use the to do list feature.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .FIG. 2: As shown below, participants entered the list title, “For dinner tonight”, in the task entry field rather than the title entry field.


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .FIG. 3: The repeating appointment drop down menu as it currently appears.

FINDING 2Participants had trouble creating appointments that repeated on a monthly basis.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

SUPPORTING DATA

Results from task 5: 3 out of the 4 participants got confused when asked to create an appointment that occurred on the 1st Friday of every month. After selecting the “Repeats” checkbox in the appointment edit dialogue window, all three participants clicked on “week” in “Repeats every week on Friday” to bring up the associated pop-up menu. Because none of these partici-pants had a first Friday selected, the example given in the menu said, for instance, “month (on the 4th Friday)” (see Fig. 3). Participants 3 and 4 eventually figured it out because, de-spite their doubts, they selected the month option. P1, however, was creating appointments from within the calendar itself, and actually exited the appointment edit dialogue to return to the calendar and make sure she had a first Friday selected.

Quotes: “ How would I do this? ... OK, so I have to go back, I have to go to the first Friday of this month.” –P1

“ It said fourth, that’s why I was confused.” –P3

RECOMMENDATIONS

Reword the pop-up menu: Reword the menu so it’s clear the text in the parentheses is just an example and can be changed. So, for instance, the menu could say, “month (example: on the 4th Friday)”. This will help users understand that they can se-lect month, then decide whether they want 1st, 2nd, 3rd etc.

FINDING 3Participants did not understand they could type appt. details into the “Type appointment” field.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

SUPPORTING DATA

Results from tasks 5 and 8: 2 out of the 4 participants did not realize that, in the “Type appointment” field, they could type appointment details in addition to the title. By default, without any details added, the participants’ appointments were as-signed to “all” and spanned the entire day. To compensate for this, P1 would double click on the appointment after it ap-peared on the calendar, opening the edit appointment dialogue in order to go back and edit her entry. However, once she fig-ured out that she could double click on a day to bring up the edit appointment dialogue, and thus create a new appointment that way, she simply stopped using the “Type appointment” field. Later, when she had to use it for task 8, she entered in the date and time, but not the attendees. She went back and edited the attendees via the same strategy she employed before—clicking on the appointment in the calendar and bringing up the edit appointment dialogue. P4, on the other hand, started out using the “Type appointment” field in task 5, but when he didn’t get the results he wanted as a consequence of not includ-ing appointment details, he eventually found the “Weekly rou-tines” page and used that instead. Later on, in task 8, he used the “Type appointment” field perfectly, without clicking on “What can I type?”. With the other participants, P2 only used the “Type appointment” field in task 8, and by clicking on “What can I type?” he understood that he could enter appoint-ment details. P3 did not click on “What can I type”, but used the “Type appointment” field properly on her own.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .FIG. 4: The “Type appointment” entry field, as currently implemented at the top of the calendar, in its active state.


Quotes: “ I wish I could just edit it when I first started, so I don’t have to go back in...” –P1

“ Dinner at Grandma’s, all day... I don’t want to do that.” –P4

RECOMMENDATIONS

Textbox watermark: When the “Type appointment” field is ac-tive, include some sort of textbox watermark that says, “Type appointment title and details here.” Once the user begins typ-ing, this message would disappear.

Apply the proximity principle: “Elements that are closer to-gether are perceived to be more related than elements that are farther apart” (Lidewell, 160). As currently implemented, “What can I type?” is located at a distance from the cursor (see Fig. 4), which is the user’s main area of focus. Moving “What can I type?” closer to the cursor will make it easier to find, and its relevance to the user’s current actions will be more apparent.

FINDING 4Participants’ mental models regarding to do lists involved a connection between to do list tasks and dates and/or the calendar.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

SUPPORTING DATA

Results from tasks 10–12: 3 of the 4 participants made com-ments or exhibited behavior that suggested a mental connection between tasks and dates and/or the calendar. P1, for example, clicked on the to do list snapshot at the top of the calendar, and tried to enter a calendar appointment in “Shared To Do”.

Previous usability tests: Participants have exhibited this mental connection in previous usability tests as well. On 11-11-2010, the participant wanted to be able to add task due dates. The par-ticipant from the 5-13-2010 test also wanted this feature, in ad-dition to wanting a reference calendar within the to do list area. In an iPhone app test from 8-5-2010, the participant thought tasks from “Shared To Do” would appear on the calendar.

Quotes: “ Can I connect this to a date?” –P2

“ My first instinct would be to go to the calendar and set up a list.” –P3, when asked to create a new to do list

“ Lists are confusing. Do they link to a day or time?” –P3

“ I am not sure if I can sync to do lists with specific days of the week?”–P3

“ A place for ‘complete by’ and ‘completed on’ would be nice.” –Facebook fan posting to the Cozi wall about to do lists

RECOMMENDATIONS

Embrace this mental model: Embrace this mental connection between tasks and dates by supporting an actual relationship between to do lists and the calendar (i.e. something more than the to do list snapshot that appears at the top of the calendar, see Fig. 5). This relationship could include allowing a designation that places selected tasks in the calendar itself, and vice versa.

Add a “due date” feature: Add a “due date” feature in to do lists, allowing users to connect tasks with specified dates.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .FIG. 5: The to do list snapshot as it currently appears at the top of the calendar.

FINDING 5Participants did not notice error messages on the “About your household” page.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

SUPPORTING DATA

Results from task 3: 3 of the 4 participants made errors when typing in their information on the “About your household” page. 2 of these 3 did not initially realize they had made an error, and thought that their data had been successfully saved after clicking the “Done” button. This happened because when par-ticipants scrolled to the bottom of the page and clicked “Done”, the page popped back up to the field where the mistake was made. However, the accompanying error message was above that (see Fig. 6 on the following page), near the top of the page where the participants couldn’t see it. Had we not pointed out their errors, participants 2 and 3 would have navigated away from the page without making cor-rections. Without the participants realizing it, their data would not have been saved.

Quotes: “ Aha, it’s red... yeah, that would be confusing, because usually when it does that it also adds red text above it or something that says ‘you need to fix this thing.’” –P2


RECOMMENDATIONS

Quick fix: Ensure that, in all browsers, the page pops up to the top where the error message is, rather than to where the error was made. This will make the error more obvious.

Apply the Proximity Principle: Following the Proximity Prin-ciple, as defined on page 11, move the error message so that it appears next to the field where the error has occurred.

Apply the Forgiveness Principle: “Designs should help people avoid errors and minimize the negative consequences of errors when they do occur” (Lidwell, 88). If users try to navigate away from the page when their data hasn’t been saved due to an error, include a pop-up message that notifies them of this fact. A sim-ilar pop-up occurs when errors aren’t involved; a consistent treatment should be applied here.

FINDING 6Participants could not figure out how to create list subheadings.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

SUPPORTING DATA

Results from task 15: When asked to group certain items in a shopping list apart from the other items without creating a new list, 0 of the 4 participants used subheadings to accomplish this task. The only action the participants took was to simply rear-range the items so that the indicated ones were on the bottom.

Previous usability tests: In a previous usability test from 12-3-2009, the participant was also unable to figure out how to create a list subheading. This situation was different from the present study, however, because the test was being conducted on the Cozi 2.0 version of the web client. This means there was actually a tool tip pane located to the far left that explained the creation of subheadings. Despite this, the user was still unable to com-plete the task, stating, “This seems a little obscure.”

Quotes: “ I just moved (the list items) around... I don’t see a way to make any other designation.” –P1

RECOMMENDATIONS

Apply the Visibility Principle: Following the Visibility Princi-ple as defined on page 9, clearly indicate the creation of sub-headings as an action that can be performed.

Create new user introductory text: Like the calendar, create new user intro text that includes a tip on creating subheadings.

Create a dismissable tool tip: Create a dismissable tool tip that would appear when users add an item or task for the first time.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .FIG. 6: The “About your household” page with an error. If the user is scrolled to the bottom of the page, it is unlikely that they’ll be able to see the error message when it is located so far away from its associated error.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .FIG. 7: The to do lists, as currently implemented, with a subheading and tasks beneath. Subheadings are created by typing in all caps.


FINDING 7Participants didn’t know why they would enter their phone numbers on the “About your house-hold” page.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

SUPPORTING DATA

Results from task 3: 2 of the 4 participants had no correct ideas regarding why they would enter their mobile numbers on the “About your household” page. The participants that did had rel-evant experience from using Google calendar.

Quotes: “ I have no idea at this point... other than... security reasons, maybe?” –P3, when asked why she thought she would enter mobile phone numbers.

“ Um... so I don’t forget them?”–P4, when asked why he thought he would enter mobile phone numbers.

RECOMMENDATIONS

Include an explanation: Change the link to say, “Enter mobile phone numbers to receive messages and appointment remind-ers,” and include additional explanatory text on the mobile number entry page.

FINDING 8Participants thought the postal codes were for marketing purposes, not maps.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

SUPPORTING DATA

Results from task 3: 3 of the 4 participants thought entering postal codes would be for marketing purposes. Regardless of whether or not this is actually true, postal codes provide a ben-efit to users in that they allow Cozi to provide maps when users enter locations for their calendar appointments.

Quotes: “ I think that’s probably for the surveys* and for them to gather marketing information.” –P2

*At the time, Bounce was running surveys in the 300x250 ad space.

RECOMMENDATIONS

Explain the benefit of providing a postal code: So users un-derstand the benefit of entering their postal codes, add text nearby that explains how postal codes are used to provide maps.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .FIG. 8: The “Enter mobile phone numbers” link (top), as it appears on the “About your household” page. Clicking on this link takes you to the “Edit mobile numbers” page (bottom). Neither the link nor the edit page offer any direct explanations as to why users would want to enter their mobile numbers.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .FIG. 9: The postal code entry field as it appears on the “About your household” page. As you can see, there is no explanation offered as to why users would want to provide this information.


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .FIG. 10: The shared journal webpage preview page, as it currently appears.

INTERESTING PARTICIPANT SUGGESTIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1. Add the ability to copy/past an appointment in the calendar.

2. Add the ability to change the default time for reminders.

3. Add some form of color-coding to month view.

4. In the school internet calendars, add the option to just type in a school or district (perhaps including an auto-fill fea-ture), rather than forcing users to hunt through a long drop down menu.

5. Add some sort of scheduler to the calendar—”a function that will be able to plan availability” (see page 41, P1, task 7).

6. When conflicting appointments are entered into the calen-dar, create an error message to notify the user.

7. Make it so that the “page up” and “page down” keyboard keys scroll the calendar.

8. Add the ability to create multiple reminders (such as a day ahead of time, then 30 minutes ahead of time) for the same appointment.

9. Add the ability to change the day or frequency with which the family journal newsletter goes out.

FINDING 9Participants had a hard time seeing how to move forward from the shared journal webpage preview.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

SUPPORTING DATA

Results from task 19: 2 of the 3 participants who went through the journal webpage wizard had trouble moving forward from the preview page. P1 didn’t see the wizard bar at the top of the page, and not seeing that there were any additional actions she had to take, thought she was done with the task. P2 didn’t notice the “Yes, share our journal!” button, and instead was clicking above that on the next step in the wizard (P4 briefly tried to do this at first as well). Unable to move forward, he consequently decided to start over from the beginning, hitting the back but-ton in his browser. He figured it out on his second go-around.

Previous usability tests: A previous usability study conducted on 9-21-09 yielded the same issue that P1 experienced: the user missed the wizard bar entirely, and thought she was done with the process upon reaching the preview page.

Quotes: “ I didn’t see this up here, that there was a bar for this tutorial thing.” –P1

“ I’m kinda getting stuck here... Aha! I have to click ‘yes share our journal.’” –P2

RECOMMENDATIONS

Make the wizard bar more noticeable: Make the wizard bar more noticeable, making sure that it doesn’t blend in with the browser chrome.

Make the button more noticeable: Make the “Yes, share our journal!” button more noticeable by reducing the visual com-petition between it and the wizard steps above.

Make the next step in the wizard clickable: Like the previous three wizard buttons, make the “invite” button clickable when users are on the preview page (it can remain inactive when us-ers are on earlier pages). This will help ameliorate situations like those that occurred with P2 and P4, where they tried to click on the “invite” button to advance forward.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .REFERENCES:

Lidwell, William, Kritina Holden, and Jill Butler. Universal Principles of Design. Gloucester, Mass: Rockport, 2003.


Study evaluation

WHAT WORKEDOverall, our study proceeded quite smoothly, and we didn’t ex-perience any significant problems. Several factors contributed to our success:

Working in a team: Working in a team allowed us to take a “divide and conquer” approach to implementing our study, in-creasing our efficiency and effectiveness. We were also able to take turns between moderating and note-taking, helping us each stay present and focused during what would have other-wise been a very long and tedious procedure.

The participants: Our participants were well selected, and they offered plenty of useful, spontaneous commentary. Having two moms and two dads provided some interesting data points, as did having two participants who were a married—they made comments regarding how they thought their spouse would in-teract with Cozi, and it was interesting to see how their predic-tions played out.

Having a private testing space: Using Justin’s private studio for testing provided a familiar space to which we had unlimited access, allowing for adequate preparation and troubleshooting. It was also located close to where our participants lived, thus making it easier for them to take part in our study.

Having an on-team expert: Kim, being an employee at Cozi, was familiar with the content of the site and was able to move fluidly through areas the rest of the team might have had trou-ble with. She was also able to provide useful background infor-mation, such as previous usability studies, that helped in our study design and data analysis.

WHAT COULD BE IMPROVEDThere were a number of measures we could have taken to create a more efficient test. The following represents the main issues we identified after conducting our test and subsequent analysis:

User fatigue and morale: Most of our tests took about an hour and a half to complete, leaving participants fatigued towards the end of the session. This issue could be avoided in future studies by running shorter tests, or splitting the test into two sections and allowing the participants a short break in between. A similar opportunity for improvement lies in splitting er-ror-prone tasks apart from one another and spreading them more evenly throughout the test. This would help prevent users from becoming unnaturally frustrated after experiencing a train of sequential errors when attempting a series of difficult tasks.

Test team fatigue: We conducted all four of our usability tests on the same day, working from 9 in the morning until about 6:30 in the evening. Consequently, towards the latter half of the day, we were all quite tired, and perhaps not as alert with our latter participants as we were with our earlier ones. This can be avoided in future studies by spreading our tests out over the course of two or more days, and using the time in between to start working on the data analysis. It should be noted, however, that having all our tests on the same day did make it easier to focus our activity, and it was nice to get it all over with and have the remaining time to focus on solely on data analysis.

Tracking user cognition: There were instances throughout our tests where we could have done a better job tracking the par-ticipants’ thought processes—for example, they would some-times forget to think out loud, and we would similarly forget to remind them to “keep talking.” There were also occasions where users made an interesting comment, but we failed to probe for additional information. These issues can be addressed by, obviously, being more alert and better reminding participants to keep talking when they fall silent, and by taking more liberty in asking additional probing questions when it’s beneficial to do so. Since this was our first testing experience, we may have been a bit hesitant to let the conversation stray from the scripted questions in our moderator packet. However, there could be great value in al-lowing a more free-flowing moderator/participant discourse.

Questionnaire clarification: In creating our post-task question-naires, we included three Likert rating scales. In an effort to en-courage participants to pay more attention to their scoring, we alternated the polarity of our scales (see appendices G and H), having the “good” end of some scales on the right, while others were on the left. However, because the same post-task ques-tionnaire was used throughout the test, and because there were only three rating scales (meaning two shared the same polari-ty), it was clear that—based on their comments and other scale ratings—participants eventually went into auto-pilot mode when filling out the questionnaires, and ceased to notice the switched polarity of the scales. Consequently, they ended up circling the opposite number from what they truly intended. We accounted for this in our data table, but this issue could have been avoided entirely if we had kept the polarity consis-tent across all three scales. This was not, however, an issue in our post-test questionnaire, most likely because it had new questions and was only administered once.


Positioning of the moderator: It would have been helpful if we had positioned the moderator within the camera’s field of view. This would have made it easier to analyze the conversation be-tween moderator and subject within the Morae software.

Consecutive failures: Our final problem area was rather signifi-cant and lead to considerable issues with our data collection. This issue was consecutive user failures, or “error trains”. Hav-ing the success of a task be contingent on the completion of a previous task can create “error trains”, cases where multiple tasks cannot be completed successfully due a failure in a prior task. This leads to loss in both data and time. There are two pos-sible recommendations for countering this problem. The first would be to make sure that any task on which other tasks de-pend is always completed, either by coaching the user on what to do once they hit a dead end, or by having the moderator complete the task for them. The second recommendation would be to move the troublesome tasks to different places within the sequence of the test, having the user return to that area of the test a number of times rather than having interrelat-ed tasks occur in succession. Doing this relates back to the mo-rale issue mentioned earlier, and would hopefully prevent users from abandoning investment in the test due to a feeling of “just not getting it.”

ADDITIONAL AREAS TO EXPLORE

Other features and interactions: Cozi is a rich product with many components, so there’s still much left to explore in terms of other features and interactions. Cozi’s mobile solutions, for example, would make an excellent area of exploration. This is particularly pertinent considering Cozi just released a new Android app, and since mobile is becoming an increasingly im-portant aspect of the software marketplace.

Advertising: An interesting investigation could be developed around the advertising within Cozi: evaluating how success-fully it has been integrated, and whether or not it negatively impacts users’ opinions of the product and their likelihood of adoption.

Localization: Perhaps a test could be designed that would be helpful in supporting Cozi’s localization efforts for Asian mar-kets. For instance, a test could be conducted with a participant who is a recent immigrant—they would speak English as their second language and have a different cultural perspective.

Accessibility: While accessibility might not be a major priority for Cozi at this point in time, it could still be an interesting field to explore. Improvements to usability for users with certain limitations or disadvantages can help other users as well, and testing with such participants could reveal compelling issues that might otherwise not be uncovered.

Eye tracking: Conducting a series of usability tests using an eye tracker could help reveal new insights. The eye tracker could, for instance, be useful in evaluating the effect of advertising in the site, or in understanding how users navigate and scan through the various calendar views.


Appendices

TEST MATERIALS

A: Participant screener & sessions schedule 19

B: Background questionnaire form 20

C: Orientation script 21

D: Consent form 22

E: Participant task guide 23

F: Moderator task guide 29

G: Post-task questionnaire form 32

H: Post-test questionnaire form 33

RESULTS

I: Background questionnaire answers 34

J: Data collection table 35

Task 1 35

Task 2 36

Task 3 37

Task 4 38

Task 5 39

Task 6 40

Task 7–9 41

Task 10–12 42

Task 13–16 43

Task 17–19 44

Debriefing 45


1. How may children do you have?

a. One . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (continue) b. Two . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (continue) c. Three or more . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (continue) d. None . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (terminate)

2. Do you have any children under the age of 16?

a. Yes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (continue) b. No . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (terminate)

3. Do you have any children who currently attend school?

a. Yes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (continue) b. No . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (terminate)

4. How often do your children, collectively, participate in extracurricular activities?

a. 1-2 times a week . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (continue) b. 3-4 times a week . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (continue) c. More than 5 times a week . . . . . . . . . . . . . . . . . . . (continue) d. Never . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (terminate)

5. Are you or your spouse currently working?

a. We both work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (continue) b. One of us works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (continue) c. Neither of us work . . . . . . . . . . . . . . . . . . . . . . . . . (terminate) d. I am single and I work . . . . . . . . . . . . . . . . . . . . . . . . (continue) e. I am single and I don’t work . . . . . . . . . . . . . . . (terminate)

6. Outside of your regular work schedule, how many appoint-ments or activities do you typically have in a week? This can be clubs, workouts, volunteering, social engagements, etc.

a. 1 a week . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (terminate) b. 2–4 a week . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (continue) c. More than 5 a week . . . . . . . . . . . . . . . . . . . . . . . . . . . (continue) d. None . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (terminate)

7. Do you have an email address?

a. Yes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (continue) b. No . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (terminate)

8. How often do you use the internet?

a. 1–6 times a week . . . . . . . . . . . . . . . . . . . . . . . . . . . . (terminate) b. Every day . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (continue) c. Multiple times every day . . . . . . . . . . . . . . . . . . . . . (continue) d. I don’t use internet . . . . . . . . . . . . . . . . . . . . . . . . . (terminate)

9. How often do you see your child/children?

a. Every day . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (continue) b. 4–6 times a week . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (continue) c. 2–3 times a week . . . . . . . . . . . . . . . . . . . . . . . . . . . . (terminate) d. Once a week . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (terminate) e. Once a month . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (terminate) f. Once a year. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (terminate) g. Never . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (terminate)

10. Have you tried or used any family-oriented online organizers in the past two years? If so, which one/s?

(If they have used Cozi, terminate)

APPENDIX A: PARTICIPANT SCREENER & SESSION SCHEDULE

Participants: Four Floaters: N/A Incentive: Babysitting services & Cozi swagImportant details: • Have participants complete the background

questionnaire prior to test in order to save time • Participants do not need to bring anything. All

hardware and software will be provided. • Supply beverages and snacks!

Schedule:Saturday, November 20, 201010:00 am - 11:30 pm : Participant One12:00 pm – 1:30 pm : Participant Two01:30 pm – 2:30 pm : Lunch & review session02:30 pm – 4:00 pm : Participant Three04:30 pm – 6:00 pm : Participant Four


Thank you for participating in our study. Please answer the following questions:

1. What is your and your spouse’s occupations?

2. Of the age ranges listed below, which do you fall under? (circle one)

a. 25–29 b. 30–39 c. 40–49 d. 50–59

3. How many children do you have, and how old are they?

4. How many extracurricular activities are your children, collectively, currently participating in?

5. What type of computer and operating system do you use at home and, if applicable, at work? How about your spouse?

6. Which web browser (such as Internet Explorer 8, FireFox, Safari, etc.) do you use the most?

7. Do you own a mobile phone, and if so, what type of phone is it? What about your spouse?

8. Do you use, or have you used in the past, an electronic calendar (such as Outlook, Google Calendar, iCal, etc)?

If so, which one? This can include both computer- and phone-based calendars.

9. What do you typically use the internet for (e.g. email, shopping, reading the news, etc.)? Of your total time spent online,

approximately what percentage is spent on each activity?

10. On a typical day, about how much time do you spend online?

APPENDIX B: BACKGROUND QUESTIONNAIRE FORM


APPENDIX C: ORIENTATION SCRIPT

Welcome to our usability study. Thank you for agreeing to participate.

In order to maintain consistency I will be reading to you from this script, which is also being read to the other participants.

We are testing a live website to see how it might be improved. You represent the typical user. You are here to offer us your feedback on the usage of the site. Please keep in mind that we are not testing you, we are testing how easy the site is to use.

Before we begin I want to let you know that you will be recorded while you work.

(Point to camera on laptop)

We are doing this in case we miss something during the study and want to go back and review it. Your name will not be associ-ated with the recording, so you will remain anonymous. You are a volunteer and are free to leave the test at any time. Before we begin, we would like you to read and sign this consent form.

(Give participant consent form)

Let me know if you have any questions.

(Once participant has completed the form...)

We will begin our study by giving you some tasks to perform, then we will ask you some questions afterwards.

If you get stuck performing a task and need to ask for help, I may not be able to help you, because we want you to try and figure out things on your own. The purpose of the study is to see how users would react in a real situation. However, if at any point you feel like you would call customer care, just let me know.

One thing you will be asked to do during the test is to “think out loud” while you are completing the tasks. I want you to pretend you are in this room by yourself and tell me everything that is going on inside your head. Specifically, tell me what you are doing, what you’re thinking about doing, and why you are doing it. If you are looking for something specific, I want you to tell me what you are looking for. If you have any opinions, please tell me those as well. This may seem challenging, but it is important to tell me how you’re feeling and what you’re thinking while performing the tasks. If you forget to think out loud, I will remind you.

Thinking out loud can be an unusual request, so let’s practice. I want you to pretend you are walking through your house and counting how many windows there are. Tell me everything that you’re thinking as you do this.

(Participant practices thinking out loud)

Do you have any questions before we begin the test?


APPENDIX D: CONSENT FORM

Cozi Usability Study Participant Consent Form

The purpose of this document is to obtain your consent to participate in this study and your agreement to keep any information that you learn in this study confidential.

PURPOSE & BENEFITSThe purpose of this study is to better understand the needs of new Cozi users so that the company can improve the design and usability of the product. This study is not testing your performance, but rather how Cozi, the application you will be using today, will perform. Your participation and feedback will help the company improve the performance of Cozi for new users as well as hundreds of thousands of current users.

PROCEDURESAs a participant in this usability study, you will be asked to perform a set of tasks using the Cozi website. After completing these tasks, you may be asked to answer a few questions about your experience, and to fill out a brief questionnaire.

ANONYMITY AND CONFIDENTIALITYToday’s session will be visually and aurally recorded to allow us to review the information we collect. The tape will not be broad-cast publicly or used for any purpose unrelated to evaluating the application. In order to ensure both anonymity and confidential-ity, your name will not be associated with the recordings or any of the data we collect today. Remember, you are free to refuse participation in the study and to leave at any time without penalty.

PARTICIPATION AGREEMENTAs a participant in today’s study, you agree that any ideas that arise as a result of today’s interaction are property of Cozi.

Please print your name, the date, and sign below if you agree with the statements above.

I, the undersigned, hereby give permission for today’s usability session to be visually and aurally recorded for the purposes described. I agree to abide by all terms set out in this consent form.

Name. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Date. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Signature. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Experimenter: print your name, the date, and sign below.

Experimenter. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Date. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Signature. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .PAGE 1

Your friend sent you a link to cozi.com with a short message saying you should check it out. You follow her suggestion and go to the site to see what it is.

Task 1: Poke around the site, and once you’re confident that you’ve gained an understanding about what the site is for, close your browser window.

Please do not turn to the next page until instructed to do so by your moderator.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .PAGE 2

Task 2: Sign up for Cozi using the following email and password: [email protected] / cozi123. When you reach your family homepage, poke around to get your bearings. Alert the moderator when you’re done.


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .PAGE 3

Task 3: Click on “Tell Cozi who’s in your family” under “Welcome to Cozi!” Use the name David/Anne for your spouse and [email protected] as their email address. Enter Emily and Ryan as children, with [email protected] as Emily’s email address. Do not click the Save button until instructed to do so by the moderator. Alert the moderator when you’re finished.


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .PAGE 4

Task 4: Click on “Calendar” on the left hand side. Take a moment to poke around and get familiar with the area. Alert the moderator when you’re done.


APPENDIX E: PARTICIPANT TASK GUIDE


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .PAGE 5

Your family is always busy with a variety of regular, repeating activities.

Task 5: Find a way to enter the data below into the calendar as efficiently and easily as possible.

a. Everyone, Dinner at Grandma’s, every Th., 5:30–7:00 pm

b. You, Workout, every Wed. and Fri., 7:00–8:00 pm

c. Spouse, Volunteering, every Th., 7:00–8:30 pm

d. Emily, Work, every Fri. 6:30–10:00 pm and every Sun. 12:00–4:00 pm

e. Emily, Yearbook, every Wed. until May 5th, 3:00–5:30 pm

f. Ryan, Boyscouts, every other Mon., 3:00–4:30 pm

g. You and your spouse, Lunch date, 12:00–1:00 pm, every first Fri. of the month


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .PAGE 6

<Post-task questionnaire, see appendix G>

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .PAGE 7

Emily attends the International Community School in the Lake Washington School District, and you would like to add important dates from this school to your Cozi calendar. For Emily, you’re particularly interested in adding appoint-ments for 10th grade students, drama club, and days of no school. You and your spouse are involved in the PTA, and you’d like to add important dates for this as well.

Task 6: Find a way of adding the above information by clicking on “School / Inter-net Calendars” under the Set Up tab.



. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .PAGE 8


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .PAGE 9

Your relatives, Scott and Jackie, will be visiting from out of town Feb 8–11. They’re available on any of those days for dinner with you and your spouse. You’d like to take them to your favorite restaurant, Zig Zag Café, for dinner at 7:00.

Task 7: Find an evening that you and your spouse are available to go out to dinner with Scott and Jackie, and that Emily is available to stay home with Ryan. You would be willing to skip a workout in order to schedule this dinner.


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .PAGE 10


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .PAGE 11

Task 8: Appropriately revise your calendar to schedule this dinner. Use the “type appointment” entry field to enter in the dinner appointment.


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .PAGE 12


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .PAGE 13

Task 9: Find a way to help you remember this dinner a week prior to its occurrence.



. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .PAGE 14


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .PAGE 15

Let’s say it’s the day of your scheduled dinner with Scott and Jackie, and you’d like your spouse to help take care of a few things in preparation.

Task 10: Navigate to the To Do lists and create a new list called “Dinner with Scott and Jackie” with the following items: Make reservation at Zig Zag Café for 7:00; Rent some DVDs for the kids; Tidy up the car.


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .PAGE 16

Task 11: Assign this list specifically to your spouse.


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .PAGE 17

Task 12: Identify a way to alert your spouse you’ve created this list. You only need to identify the action/s you would take; you do not need to actually perform them.


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .PAGE 18

<Post-task questionnaire, see appendix G. Will apply to tasks 10–12.>


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .PAGE 19

You decide you’d like to have Scott and Jackie over to your house for appetizers before heading to the restaurant. You also need to get something that the kids can have for dinner while you’re out.

Task 13: Go to shopping lists and create a new list titled “For tonight” and enter these items:

Grapes, Crackers, Pepperoni Pizza, Ice cream, Bottle of red wine, Brie


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .PAGE 20

Task 14: Make this list your first shopping list (ahead of Groceries and the others).


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .PAGE 21

Task 15: Group the kids’ items, pizza and ice cream, apart from the other items and under a heading called “kids”.


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .PAGE 22

Task 16: You remember that you already have a bottle of red wine and some crack-ers. Subtract these items from the list.

Please do not turn to the next page until instructed to do so by your moderator.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .PAGE 23



. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .PAGE 24

Let’s say your family recently returned from a vacation to Hawaii. Ryan had a lot of fun taking surfing lessons, and you got a really great photo of him.

Task 17: Navigate to the Journal and take a brief moment to look over the page. Alert the moderator when you’re done.


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .PAGE 25

Task 18: Create a new entry using the photo on your desktop, RyanSurfing.jpg, and the following text: “We had a great time in Hawaii! Ryan finally got to take surfing lessons like he’s always wanted. He’s a natural.”


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .PAGE 26

Task 19: Set up your family journal to be shared as a webpage.


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .PAGE 27


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .PAGE 28

<Post-test questionnaire, see appendix H.>


APPENDIX F: MODERATOR TASK GUIDE

Your friend just sent you a link to cozi.com with a short message saying you should check it out. You follow her suggestion and go to the site to see what it is.

T1 Poke around the site, and once you’re confident that you’ve gained an understanding about what the site is for, close your browser window. (This is so participants can’t see the website when answering the following question)

Q1.1 Pretend you’re describing Cozi to your spouse who has never heard of it before. How would you explain it to him/her?

Q1.2 Is there anything about Cozi that you’re currently un-certain of?

Q1.3 Based on what you’ve seen so far, is Cozi something you would be interested in for your own family? Is there anything that would make you hesitate to sign up? (Does Cozi seem like something that would help users manage their family life more easily? Or does it seem like too much work to actually be beneficial?)

T2 Sign up for Cozi using the following email and password: [email protected] / cozi123. When you reach your family homepage, poke around a bit to get your bear-ings, then decide where you would go first. Do not actually navigate away from the page until instructed to do so by your moderator.

Q2.1 Where would you go first from here? Why?

Q2.2 Is there anything about Cozi that you understand dif-ferently now that you’re signed in?

T3 Click on “Tell Cozi who’s in your family” under “Welcome to Cozi!” Use the information below to fill out the page. Do not click the “Save” button until instructed to do so by your moderator. Alert the moderator when you’re finished. Your spouse is David/Anne. His/her email is [email protected]. Your children are Emily and Ryan. Emily’s email is [email protected] (Do users read the text at the beginning? Do they even look at it?)

Q3.1 What do you think the purpose is for entering in this information?

Q3.2 What sort of relationship do you think other mem-bers of your family are intended to have with Cozi once you’ve signed up for it?

Q3.3 Regarding the “enter mobile phone numbers” link, what do you think the purpose of entering your phone numbers would be? (Do participants understand how Cozi works with phones? Is their answer based on what they’ve learned about Cozi so far, or are they just guessing?)

Q3.4 What about the field for your postal code? (Did users notice/read the text regarding postal codes on the homepage?)

Q3.5 How would you feel about actually filling out all this information about your family? (Do participants have privacy concerns?)

(Tell user to click save. They are now back on Home)

Q3.6 From here, where would you want to navigate to next, and why? (Regardless of the participants’ answers, we’ll instruct them all to go to the same place next)

T4 Click on “Calendar” on the left hand side. Take a moment to get familiar with the area, and decide where you would go first from here. (Do users look at the welcome text or the highlighted Phone and Set Up buttons?)

Q4.1 What would you do first from here? (Would they click on the welcome text or the highlighted Phone and Set Up buttons?)

Q4.1 Is there anything about the calendar that’s unclear at this point?

Your family is always busy with a variety of regular, repeating activities.

T5 Add the following appointments to your calendar. (Do us-ers discover/use “Weekly Routines” or “schedules” areas?)

a. Everyone, Dinner at Grandma’s, every Th, 5:30–7:00 pm

b. You, Workout, every Wed and Fri, 7:00–8:00 pm

c. Spouse, Volunteering, every Th, 7:00–8:30 pm

d. Emily, Work, every Fri 6:30–10:00 pm, every Sun 12:00–4:00 pm

e. Emily, Yearbook, every Wed. until May, 3:00–5:30 pm

f. Ryan, Boyscouts, every other Mon., 3:00–4:30 pm

g. You and your spouse, Lunch date, 12:00–1:00 pm, every first Fri. of the month


(Written questionnaire, then...)

Q5.1 How was it entering in all that information?

Q5.2 Were you unclear on anything?

Q5.3 What else can you do in order to start filling out your calendar with pertinent appointments or events? (Do participants notice or find the set up tab/school and internet calendars / schedules? Schedules in particu-lar are not very discoverable)

Emily attends the International Community School in the Lake Washington School District, and you would like to add impor-tant dates from this school to your Cozi calendar. For Emily, you’re particularly interested in adding appointments for 10th grade students and days of no school. You and your spouse are also involved in the PTA, and you would like to add important dates for this as well.

T6 Find a way of adding the above information by clicking on “School / Internet Calendars” under the Set Up tab. (Can participants easily add all these calendars and assign to the proper attendees? Note: Probably not)

(Written questionnaire)

Your brother- and sister-in-law, Scott and Jackie, will be visiting from out of town from Feb 8th to the 11th. They say they’re available on any of those days to go out to dinner with you and your spouse. You’d like to take them to your favorite restaurant, Zig Zag Café, for dinner at 7:00 pm.

T7 Find an evening that you and your spouse are available to go out to dinner with Scott and Jackie, and that Emily is avail-able to stay home with Ryan. You would be willing to skip a workout in order to schedule this dinner. (Do participants have any issues navigating to distant dates / locating infor-mation?)

T8 Appropriately revise your calendar to schedule this dinner. Use the “type appointment” entry field to enter in the din-ner appointment. (Do participants have trouble using the natural language of the type appointment field?)

(Written questionnaire for, collectively, tasks 7–8)

T9 Find a way to help you remember this dinner a week prior to its occurrence. (Do users discover reminders?)

(Written questionnaire)

Let’s say it’s the day of your scheduled dinner with Scott and Jackie, and you’d like your spouse to help take care of a few things in preparation.

T10 Navigate to the To Do lists and create a list called “Dinner with Scott and Jackie” with the following items: Make res-ervation for four at Zig Zag Café at 7:00; Rent some DVDs for the kids; Tidy up the car.

T11 Assign this list specifically to your spouse

T12 Identify a way to, using Cozi, alert your spouse that you’ve created this list. You only need to identify the action/s you would take; you do not need to actually perform them. (Do they find the send list as text message feature?)


You decide you’d like to have Scott and Jackie over to your house for appetizers before heading to the restaurant. You also need to get something that the kids can have for dinner while you’re out.

T13 Go to shopping lists and create a NEW list titled “For to-night” and enter these items: Grapes, Crackers, Pepperoni Pizza, Ice cream, Bottle of red wine, Brie

T14 Make this list your first shopping list (ahead of Groceries and the others).

T15 Without creating a separate list, group the kids’ items, piz-za and ice cream, apart from the other items. (Do they dis-cover how to drag /drop items, create list subheadings?)

T16 You remember that you already have a bottle of red wine and some crackers. Take these items off the list (If all the they do is check them off, then ask them to get rid of the items entirely. Don’t use the words delete or remove).


T17 Navigate to the Journal and take a brief moment to look over the page. Alert the moderator when you’re done.

Q17.1 How would you describe the Cozi journal and how it works to someone unfamiliar with it? Is there any-thing about it that’s unclear at this point?

Let’s say your family recently returned from a vacation to Hawaii. Ryan had a lot of fun taking surfing lessons, and you got a really great photo of him.

T18 Create a new entry using the photo on your desktop, Ryan-Surfing.jpg, and the following text: “We had a great time in Hawaii! Ryan finally got to take surfing lessons like he’s al-ways wanted. He’s a natural.”


Your mother-in-law is always wanting to hear the latest news about your family, but she gets so much email that the stories you send her often get overlooked. She scoffs at the idea of joining a social networking site.

T19 Set up your family journal so that your mother-in-law can read your entries.


Q19.1 Is there anything you would add to or change about your description of the Cozi journal from before? Is there anything that’s unclear?

Q19.2 If you were keeping a Cozi journal for your family, how would you feel about sharing it? Which sharing method would you prefer? (Would participants feel safe sharing their journal? Would they have privacy concerns? Which of the sharing methods would they be most likely to use?)

(Written questionnaire concerning overall experience)

DEBRIEFING

Q20.1 If you created a Cozi calendar and wanted your spouse to be able to view it, what would you tell them to do? (Does the participant understand their family would share a single Cozi account, and that his/her spouse doesn’t need to sign up for Cozi themselves in order to log in?)

Q20.2 Let’s say you set up Cozi from your home computer, and enter an appointment into your Cozi calendar. You’d like to view this appointment from your work computer. How would you do this? (Does user un-derstand that their data is stored online and can be accessed through any internet connection?)

(Other questions as moderator sees fit)


APPENDIX G: POST-TASK QUESTIONNAIRE FORM

Please answer the following question for the task/s you just completed.

<Task number/s will be printed here>

Using the following scales, how would you rate your experience performing the previous task/s: (circle a number below)

Very difficult Very easy1 2 3 4 5 6 7

The task was exactly what I expected

The task was not at all what I expected

1 2 3 4 5 6 7

I am dissatisfied with how the task worked

I am satisfied with how the task worked

1 2 3 4 5 6 7

Comments:


APPENDIX H: POST-TEST QUESTIONNAIRE FORM

4. What was the easiest feature to use on the site?

5. What was the most difficult feature to use on the site?

1. Using the following scale, rate your overall experience using Cozi (circle a number)

It was very difficult to use

It was very easy to use

1 2 3 4 5 6 7

2. Using the following scale, rate your overall experience using the calendar (circle a number)

It was very difficult to use

It was very easy to use

1 2 3 4 5 6 7

3. Compared to other electronic scheduling tools you may have used—Outlook, Entourage, iCal, etc—rate your experience using the Cozi calendar (circle a number. If not applicable, leave blank)

It was much easier to use

It was much more difficult to use

1 2 3 4 5 6 7


APPENDIX I: BACKGROUND QUESTIONNAIRE ANSWERS

PARTICIPANT 1 (MOM) PARTICIPANT 2 (DAD) PARTICIPANT 3 (MOM) PARTICIPANT 4 (DAD)

Q1. Participant’s and their spouse’s occupations

Participant: Mom, artist

Spouse: Freelance industrial designer, recent grad

Note: Wife to P2

Participant: Freelance industrial designer

Spouse: Visual artist

Note: Husband to P1

Participant: Artist / graphic designer

Spouse: Bakery owner

Participant: Landscape foreman

Spouse: Fashion designer

Q2. Age range

35–40 35–40 35–40 35–40

Q3.Number of kids and their ages

One 8 year old daughter One 8 year old daughter One 8 year old daughter One 5 year old daughter

Q4. Kids’ activities

Fiddling, swimming Fiddling, swimming Soccer, hiking Dance

Q5. Computer and OS

Home: Vista home

Work: Windows 7

Home: Vista home

Work: Windows 7

Home: Mac OSX

Work: Mac OSX

Home: Mac OSX

Work: N/A

Q6. Most used web browser

Google chrome Google chrome Safari Safari

Q7. Cell phones

Participant: HTC Hero

Spouse: HTC Hero

Participant: Android Phone

Spouse: Android Phone

Participant: iPhone

Spouse: iPhone

Participant: Older nokia

Spouse: Smartphone

Q8. Other digital calendars used

Google calendar Google calendar iCal N/A

Q9. Internet use

30% communication30% social networking20% Adobe programs20% watching movies/shows

40% communication10% social networking50% Adobe & software programs10% watching movies/shows

40% communication20% social networking20% Adobe & software programs20% watching movies/shows

40% communication30% social networking10% web applications20% watching movies/shows

Q10. Time spent online / week

“ Too much if you include checking email on phone.”

“ Probably 15 hours and when I am working on a computer project even more.”

“ It was probably about 25 hours when I helped with my husbands business, but less now that I am at home more often.”

“ Not very much, perhaps an hour a day.”


APPENDIX J: TEST NOTES / QUESTIONNAIRE RESULTS

PARTICIPANT 1 PARTICIPANT 2 PARTICIPANT 3 PARTICIPANT 4

COMPLETION COMPLETED COMPLETED COMPLETED COMPLETED

NOTES

• Looked through hover panel first, then the top right promotion area. Down the right column to the Live Simply links. Then noticed the awards.

• PC: “Ooo, that looks interesting.”

• NC: Noticed Bounce survey ad; said she doesn’t like when people profile her.

• Spent more time looking around than would have if wasn’t doing a test

• NC: “I hate surveys. I don’t want to do that.” Turned off by ads, especially the Bounce survey ad.

• Feels cautious: it’s just trying to sell something

• “One website where we can communicate with each other.”

• Immediate impression: “Just looks like a shopping site.”

• Too much information: overloading if not actively looking for all these things.

• NC: “There’s so much information on the page, it’s hard for me to know exactly what this is right off the bat. If I came across this just browsing the internet, I’d probably just close it.”

• “To do lists... I have no idea what that means....”

• NC: Turned off by ads• Too digital, not the type of

person that would use this• Noted it’s free• Didn’t look at other hover

panels: “Online calendar kinda thing.”

Q1. How would you explain Cozi?

“ A website to appeal to parents who maybe aren’t on the same page... a service to help people get connected together in a family.”

“ Well, from what I gather, it looks like a way to organize our chores and appointments and easily communicate with one another. Share shopping lists and stuff.“

• She seemed to feel uncertain about what exactly Cozi was, but she actually did understand.

• “It might be... it looks like a way to help organize our everyday tasks or schedules. It seems like there might be more resources. Different ways of organizing...”

• “Some online database for helping you organize your family’s... sorry, I don’t know.”

“ Some sort of thing that I can store appointments and information in. Maybe I can shop on it.”

Q2. Is there anything you are currently uncertain of?

“Nope, not yet.” “ I want to learn more about how you set it up to see if it is worth the time.”

“ I mean... yeah... what would be the benefit of me using this site? When I could just be linking up my life with iCal or Google Calendar, or my own personal computer.”

“No.”

Q3. Would you use Cozi? Is there anything that would make you hesitate?

“ It seems like just another thing to sign up for. I’ve already taken the time to get myself situated with Google calendar... this is yet another thing to pump a bunch of information into that my husband and I may or may not actually use. I would set it up, but he wouldn’t actively check it.”

• “Maybe. This may sound really bad, but if it was easy for my wife to do, and then send me something to my phone so I could just check off something, than yeah, I would do it.”

• “If everyone has to go and make their own account, I probably wouldn’t do it.”

• “My wife is really in to this stuff, which is why I used her as an example.”

“ I wouldn’t because if I have to go to too many different places to organize, I feel like I wouldn’t do it. Unless it was somehow already synced up with my email or computer somehow.”

“ I don’t use computers very much and it seems like it might be a bit of a drag to continually update it.”

TASK 1: Ascertain what Cozi is by browsing the marketing website.




ERRORS 1 0 0 0

NOTES

• ERROR: Hit sign up without entering password.

• “It would be cool if it linked to my Google calendar.”

• NC: Noted the Bounce survey ad: “I hate that type of stuff ... I don’t like it when people profile me.”

• PC: “App for android... Oh, I have an android, yay.”

• Doesn’t know what the messages area is, starts to think Cozi is somehow about social networking for parents.

• “Messages, I don’t know what that it is... some sort of social networking thing?”

• Initially looked to upper right for sign up button.

• Would look at privacy policy quickly to make sure he won’t get spammed.

• NC: Almost wants to go away from the site when confronted by yet another survey ad.

• After reading privacy policy agreement: “I’m assuming if I had children they’d have to somehow have access to this site so that they could also view our schedules?”

• “I’m looking mainly to the upper left first.”

• PC: “I like the way they put their tabs on the left hand side.”

• NC: Regarding the family homepage: “I feel like there’s so much, so many words on this page... I would much rather things be in tabs or something. I don’t know where to go first.”

• In the welcome text, looked at the blue text and didn’t really pay attention to rest: “I didn’t even read the black text, I went straight for the blue.”

• “It’s clearly organizational.”• PC: “I’m glad there’s no

pop-ups.”

“I’m not reading this (the terms) at all.”

Q1. Where would you go from here (family homepage), and why?

Would click on “Tell Cozi who’s in your family” link in the intro text.

• “ I would look at the left sidebar links first.”

• “ Then I might go in to settings, and try to turn off the surveys.”

• “ I would go in and try to hook it up to the calendar.”

Would click on “Tell Cozi who’s in your family” link in the intro text because “I’m just curious how it’s all organized.”

• Would go to calendar first from homepage: “I’m not ready to tell Cozi who’s in my family just yet.”

Q2. Is there anything you understand differently now that you’re signed in?

“I’m getting the sense that they’re trying to make a networking site for parents.”

“ It is pretty much as I thought it would be.”

“ Yeah, it just seems like a way of organizing your family schedule.”

“ Sort of what I thought it would be.”

TASK 2: Sign up, browse the family homepage.




ERRORS 0 1 1 0

NOTES

• Initially confused as to what “Adult 2” meant. Read intro text and realized it must mean spouse.

• NC: “I’m starting to think I don’t like Cozi.”

• Confused by term “Adult 2”• ERROR: Misentered email,

didn’t notice error text b/c was scrolled to bottom of page and couldn’t see error text at top. “Aha it’s red... yeah, that would be confusing because usually when it does that it also adds red text above it or something that says ‘you need to fix this thing.”

• NC: Again, turned off by Bounce survey ad.

• Didn’t see Cozi was free, but thought it might be.

• Didn’t notice / read text at top.• “I’m curious about the privacy.”

Took a while skimming the Privacy Policy.

• BUG? Entered info, checked privacy policy... then all info was gone when she returned to “About Your Household” page.

• ERROR: Didn’t see Adult 2 entry field, entered spouse in the first child slot, then entered kids underneath.

• Same problem with error message as P2. Misentered email, didn’t save at first.

• BUG? Saved page and was taken to shopping lists?

• Got an error notice like P2 and 3, but unlike them, noticed one of his entry fields had turned red and then corrected the error in it.

Q1. Purpose for entering all this info?

“Just loading up our profile.” “ So that they can be referenced in lists or on the calendar, then if... seems like Emily is old enough to have a phone, maybe she could get stuff sent to her.”

“ It’s some online database that helps track things... yeah, I don’t know...”

Spam, or perhaps for joining a network of other parents. Maybe entering this info so can send appt. reminders.

Q2. Relationship rest of familywould have with Cozi?

• “I’m assuming that it’s just for informational purposes. I’m not sure if this is directed for networking for the kids as well.”

• “Or maybe kids who are a little bit older... they can access this site for some reason. Be part of the family planning and whatnot.”

“ I’m just assuming that, Ryan, if he’s old enough, well anybody on here ... would have access to at least see the calendar. I’m hoping there’d be a way to allow or block the kids from changing things.”

“ I don’t think anybody else would use it. But if I were the main person in my family posting all the events... maybe they’d have a reason to check it to see what’s going on.”

Understood that both he and his family could update things on Cozi.

Q3. Why enter cell numbers?

“ Possibly so that we can get text messages... reminders. Like with Google.”

“ I’m assuming it’s for syncing up Outlook or calendars, sending lists, getting feedback from the lists whether things are done or not.”

“ I have no idea at this point... other than... security reasons, maybe?”

“Um... so I don’t forget them?”

Q4. Why enter postal code?

“ For social networking maybe... hook us up with other Ballard families.”

“ I didn’t even look at it... to have a location, and I think that’s probably for the surveys (Bounce ads) and for them to gather marketing information.”

“ Probably for marketing reasons.”

“ Let me find it... that would be for getting things mailed to me, maybe even a gathering information thing that Cozi is doing.” (marketing)

Q5. How would you feel about actually filling this info out?

• Still thinks Cozi is somehow a networking tool. This makes her feel hesitant—she already has Facebook.

• Hesitant because of profiling (Bounce survey ad). “I hate being profiled. Depends on how invasive the advertising is.”

• “Um, it depends on... before I completed tasks 2 and 3 I would have wanted to learn more about it. I would look it up or do a Google search.”

• “If they seemed reputable, I would sign up.”

“ Assuming it’s secure... I have no reason to not trust them at the moment. I’d be online looking for user reviews, for anybody saying anything negative about it. I haven’t seen anything on here saying I need to pay for it. That would be a concern too.”

Feels uncomfortable, wouldn’t do it.

Q6. Where would you go next from here?

• “This is where I would probably get stuck... it’s a decision at this point as to whether or not to continue. At this point if I don’t know anybody else who’s on here... I’d be hesitant to add more stuff. I’d just stop.”

Settings. To try to turn all the ads off, or at least turn the surveys off.

• BUG? Landed on calendar after clicking done on “About your household” page (should have landed on homepage?)

• “I guess I’d want to click on a day to see what that looks like.”

Not sure if he would keep poking around or leave the site entirely.

TASK 3: Fill out the “About your household” page using the provided information.



NOTES

• Read intro text.• “I hate watching tutorial

videos, but my husband is good about watching them.”

• “I like to figure things out on my own.”

• Would go straight to internet calendars because wants to see if she can sync with her Google calendar.

• Skipped intro text to see month/day views.

• Would go to calendar month view first thing.

• Didn’t see today button, used mini cal to find current day.

• BUG? Calendar defaulted to month view instead of week (should have been week). So didn’t see intro text.

• “I have no idea what this remind us thing is... send me an email maybe? Maybe that’s why my phone number was there, so that they can send me texts? That’s my new guess.”

• “The numbers seem kind of big to me. I guess that’s not a bad thing. But I would rather see the numbers smaller and the events bigger.”

• PC: “I like how clean the days look.”

• Didn’t see arrows to go month to month. Hovered over mini cal... “I guess have to go to the little calendar up here, and the arrows don’t show up unless I’m over it.”

• Went straight to day view and set up a fictional appointment to start testing things out.

• BUG? Calendar defaulted to month like with/ participant 3, so didn’t see welcome text.

Q1. What would you do first from here?

“ Ooo internet calendars, maybe I can link it to my Google! I wouldn’t even look at the tutorial video, I’d go straight to internet calendars.”

“ I would go to month and see what that looked like instead of the weekly one”

“ I would go to month and try to add stuff to a day inside it.”

“ Maybe look at the whole year on a calendar.”

Q2. Is there anything about the calendar that’s unclear at this point?

“ Nope, it all seems pretty standard.”

“Nope, it is clear.” “No...” “Seems like a calendar.”

TASK 4: Click on Calendar, then look around and decide where you would go first from there.


TASK 5: Enter the provided series of repeating appointments into your calendar.



ERRORS 1 2 3 3

NOTES

• ERROR: To enter first repeating appointment, went to the To Do Lists via list at top of calendar. Entered “Dinner at Granny’s” into shared to do. Went back to cal. when didn’t see way to add date/time.

• Didn’t find weekly routine page• Didn’t understand that can

type out all the appt. info in Type Appt. field (typed name, not details). Kept typing name, then double-clicking appt. on calendar to add more info.

• Didn’t find how to make multi-day repeating appt, made two separate repeating appts.

• “So it’s like a Google calendar.”• “I’m always trying to find

shortcuts.”• Wanted to copy/paste appt.• Wishes “All” wasn’t default:

always having to uncheck it.• “I always like month view best.”• “Seems simple if you’re already

familiar w/ calendars, but if not you might need the tutorial.”

• When entering “Every 1st Friday” appt., only option she saw under the menu was every third Friday. The option to select 1st/2nd etc. doesn’t show up until you’ve selected “Month (every 3rd Friday).” Didn’t want every third Friday, so started over and recreated appt. on a 1st Friday.

• Now reads calendar intro text, might watch video.

• Didn’t find weekly routine page, entered appts. by double clicking on day.

• Thought there weren’t any half hours on time picker—screen brightness was poor so couldn’t see. Ended up just typing in.

• Finally saw half hours on third appt. entry

• Found multiple days option• Was setting reminders for each

appt. Would look for reminder settings to change default time.

• Likes month view best, but wishes it was color-coded

• Typed attendee name into appt. name to make up for lack of color-coding in month view

• “I don’t know where I’d go first... I might try to do this type appt. area?”

• Found weekly routine page.• ERRORS: Entered first two

appts. as AM, not PM. Noticed later and corrected.

• ERROR: Thought she had to use picker to enter time. When entered time herself, just entered “7”, which Cozi doesn’t understand (need am/pm). “That would be the easiest thing to do, if I could just type a 7.”

• NC: Using time picker is slow.• BUG? “I wonder if I can tab...

nope, no tabbing. That’s not very convenient either.” Usually tabbing works?

• “I don’t know if I have to push enter or done... I’ll just push enter because it’s closer.”

• “Oh, my dinner moved down, that’s the one I started with.... I have no idea why it did that.”

• Emily’s work happened twice a week at diff. times. Wanted to enter in the same slot, but had to do separate ones.

• “Then that moves to the top... so it’s organizing by day. I’m finally seeing what it’s doing...”

• Had trouble with every 1st Friday appt. ”It said fourth when I scrolled down, that’s why I was confused.”

• Clicked on day to enter appt. but “nothing happened” (didn’t double click). Got confused.

• Tried using type appt. field but appt. didn’t show up right away. When it did, was all day because only typed appt. and not time.

• NC: “Now I’m getting frustrated...”

• “This might take a long time guys.”

• Took him a long time to find weekly routine page..

• ERROR: On weekly routine page, because appts. reordered after he entered it, edited wrong appt. when he went back to correct attendees in the last appt. he created.

• At first, got confused in how to select 1st Friday (because drop down said 4th Friday), but selected it anyway and it then said 1st Friday.

• Said unclear at first but simple to figure out.

EASE 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7

WORKED AS EXPECTED 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7

SATISFACTION 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7

Q1. How was it entering all that info?

“Just like any other calendar” “Seems like most calendars.” NC: “Very tedious... I don’t do that any more. Most things I’m just copying and pasting them, not entering from scratch.”

“ More info than I would ever put in a computer calendar.”

Q2. Was anything unclear?

“ Very similar to Google Calendar.”

“No.” “ It all seemed pretty straightfor-ward. At first I wasn’t sure how to put in ‘every 1st Friday,’ but everything else seemed fine.”

NC: ”I don’t do this often. It was a little hard, yes, to figure out.”

Q3. What else can you do to start filling out your calendar?

“ I would probably go to find out if you can have different calendars... like on Google, I have my bill pay calendar, my purple house calendar... b/c otherwise when I look at it all together, it’s overwhelming.”

“ Appointments can be added it seems.”

“ I noticed this type appoint-ments... I might try that. I have no idea what the settings do, maybe there’s some way you can have this site link up with... I don’t know.”

Would possibly add things to the to do list.


TASKS 6: Add the indicated school calendars by clicking on “school/internet calendars” under the set up tab.


COMPLETION COMPLETED COMPLETED COMPLETED NOT COMPLETED

ERRORS 0 0 0 3

NOTES

• Didn’t immediately notice the highlighted set up tab.

• ”Add a childcare schedule... Oh, ok, so you can have different calendars.” (wrong)

• Assigned PTA to Emily (task was probably just unclear).

• Didn’t easily notice highlighted set up tab.

• NCx2: Dislikes ads.• BUG? Had to click continue

twice?• NC: Not happy, seems

complicated.

• “I have no idea where the school internet calendars are.”

• Was confused, couldn’t find set up tab at first. “I don’t know why I didn’t see that.”

• PC: “Looks like I can customize my colors, which is always fun.”

• Thought she’d go into activity/schedule... but then saw school calendars.

• “I know the public school calendars are online these days, maybe it syncs with them. That would be nice for blackout dates and holidays so you don’t have to go in and type them yourself.”

• Typing the school name right away would be easier, rather than looking through drop down for it.

• When on “Whose calendar should events from ICS appear on?” page, in reference to the button at the end: “Subscribe to calendar... I guess I wish it said ‘add to calendar’ because I’m not sure exactly what that button means.”

• Did not go to set up quickly, did not go to school schedule first.

• NC: Confused• NOTE: Might have been

confused by task question, may want to throw out this data.


TASK 7–9: Find a timeslot to enter the given appointment. Revise your calendar to schedule this appointment. Use the “type appointment” entry field. Create a reminder (a week ahead) for the appointment from task 8.



ERRORS 0 0 0 0

NOTES

• Looked for secret “figure it out for you” feature. “I’m wondering if one of the features of the website is to figure out whether or not there’s an awesome day we can all do this... a function that will be able to plan availability. But I’m not seeing anything like that.”

• Noticed she made a mistake entering one of the appts. in task 5.

• Wanted the page up / page down buttons to scroll the calendar.

• Easily found proper dates, but wondered if typing into “type appt.” would be faster way of finding the days.

• NOTE: This task was easier for this participant to complete because task 6 was not successfully completed, and thus the calendar did not have as many appts. as intended.

COMPLETION COMPLETED COMPLETED PARTIALLY COMPLETED COMPLETED

ERRORS 1 1 1 0

NOTES

• ERROR: Initially assigned appt. to All because that’s default.

• Expected error message for schedule conflict, because she hadn’t yet removed the workout appt.

• Wondered what black selected day arrows meant, but figured it out.

• Noticed and clicked on “What can I type?”

• ERROR: Used a hyphen after attendee names; Cozi doesn’t understand this so appt. was assigned to All and mis-named.

• ERROR: Typed wrong date into type appt. field (2010 instead of 2011). Then was taken to 2010, “I don’t know where the rest of the stuff is.”

• NC: “When I try scrolling it moves too fast.”

• Initially had trouble finding type appointment field

EASE 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7


SATISFACTION 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7

COMPLETION COMPLETED COMPLETED COMPLETED NOT COMPLETED

ERRORS 0 0 0 0

NOTES

• Accomplished very easily (because familiar with Google calendar?)

• After seeing his appts. in week view, again mentions how he’d like month view to be color-coded.

• “I was trying to add multiple reminders and was unable to do so. For me, a reminder a day ahead and 30 minutes ahead of time would be helpful.”

• NC: “A reminder 30 minutes ahead is not sufficient. There must be more than one reminder.”

• Did not find reminders. Made a new appt. to act as a reminder. Inexperience with this sort of software is perhaps the reason he didn’t think to look for reminders?

EASE 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7WORKED AS EXPECTED 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7

SATISFACTION 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7

TASK

7TA

SK 8

TASK

9TA

SKS

7–8

TASK

9


TASKS 10–12: Create a new to do list with the provided data. Assign this list specifically to your spouse. Identify a way to alert your spouse you created this list.


COMPLETION PARTIALLY COMPLETED PARTIALLY COMPLETED NOT COMPLETED COMPLETED

ERRORS 1 0 1 1

NOTES

• PC: “This is actually getting interesting... this might be handy.”

• Didn’t actually create a new list• ERROR: Entered list title as

item in “Shared To Do” list. Thought “Shared To Do” was a like folder of lists, rather than a list itself.

• “Did I create a list? I don’t know ... I was assuming that if I added an item that that was a list, but now I can’t go into it. I’m having a hard time figuring out how to add things to the list.”

• “I’m blanking on this one.”• Eventually went to David To Do

(because the list in the task was for David). “I guess I just type a task in here, now I get it.”

• NC: “I don’t know if this list is going to work as good as the list I plastered to the kitchen cabinet with a pen right by it.”

• “Can I connect this to a date?”• Thinks to do lists are somehow

connected to calendar / works like calendar... wants dates!

• Didn’t actually create a new list, just entered items into “Shared To Do” without changing list title (forgot that was part of the task?)

• ERROR: Same error as Participant 1.

• “My first instinct would be to go to the calendar and set up a list.”

• Subsequently could not figure out how to enter items; didn’t know how to “open” the new list she made so she could edit.

• “I’m wondering how I open the list to add things to it.”

• “I am not sure if I can sync to do lists with specific days of the week?”

• Double clicked around, was lost, could not find correct area.

• “I tried! I’m not sure if I did it. I want to know if it worked. Can I come back later?”

• ERROR: Same misunderstand-ing as Participants 1 and 3.

• Entered items correctly, underneath incorrectly entered list name.

• Figured it out eventually after seeing other lists.

COMPLETION N/A COMPLETED NOT COMPLETED COMPLETED

ERRORS 0 0 N/A 0

NOTES

Already completed because in previous task, put list items in default David To Do list rather than creating a new list.

• Changed assignment dot to spouse’s color, but title still said “Shared To Do”.

• Confused by this, but then quickly figured out how to edit the title.

• Still very confused from earlier misunderstanding in task 10, didn’t even know where to start for this task.

• “At this point I would be calling to tell him to put it in here.”

• Completed in task 10

COMPLETION COMPLETED NOT COMPLETED COMPLETED NOT COMPLETED

ERRORS 0 0 0 0

NOTES

• PC: “I think this might actually be better than rememberthe-milk.com.”

• Noted phone button immediately, but was confused by task wording, so didn’t actually identify it as the answer to the task.

• Not clear that there was any way to do this.

• Likely was unclear because task is poorly worded. We should have edited the task by this point, but did not.

• Identified send list as text message feature, but was uncertain that this was the answer to the task.

• Likely was unclear because task is poorly worded.

• “I have no clue.”

• Task is poorly worded (word “alert” seems to throw people off?)

EASE 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7


SATISFACTION 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7

TASK

10

TASK

11

TASK

12

TASK

S 10

–12


TASKS 13–16: Create a NEW shopping list with the provided data. Make this your first list. Group the kids’ items apart from the other items. Delete the indicated items.


COMPLETION PARTIALLY COMPLETED COMPLETED NOT COMPLETED COMPLETED

ERRORS 1 0 0 0

NOTES

• ERROR: Was in to do lists, not shopping lists.

• Figured out how to rename list, unlike with the to do list tasks.

• Doesn’t like making lists for things: “A lot of work, I don’t think I’d do this.”

• “My mind is shutting down.”• Didn’t actually create a new

list, just renamed default list

• “ Lists are confusing. Do they link to a day or time?”

• Entered items into default Groceries list instead of creating a new list.

• If she can text this to herself that would be nice (you can actually do this)

• Completed perfectly!

COMPLETION COMPLETED N/A NOT COMPLETED COMPLETED

ERRORS 0 0 0 0

NOTES

Figured out immediately, but was still in to do lists rather than shopping.

• Already completed this task because renamed default “Groceries” list rather than creating a new list

Was not able to reorganize the lists.

• Wanted to drag to reorder the lists (on the home list screen, not the organize lists screen).

COMPLETION PARTIALLY COMPLETED PARTIALLY COMPLETED NOT COMPLETED PARTIALLY COMPLETED

ERRORS 0 0 2 0

• Rearranged the list items so the kids’ items were together at the bottom; did not create a list subheading.

• “I just moved them around... I don’t see a way to make any other designation.”

• Rearranged the list items so the kids’ items were together at the bottom; did not create list subheading.

• “ Maybe I am supposed to put a note for David to view the list. I do not know how else he is supposed to see it.”

• “ It does not seem like there is a way to organize from here. Do you want me to try it again, or...?”

• Rearranged the list items so the kids’ items were together at the bottom; did not create list subheading.

• NOTE: This task sets participants up to partially fail. No real way for them to know how to create subheadings.


ERRORS 0 0 0 0

NOTESCompleted very easily. Completed correctly. “ I assume it is saving since there

is no save button.”• PC: Liked “cross-off thing”

EASE 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7

WORKED AS EXPECTED 1 2 3 4 5 6 7 1 2 3 4 5 6.5 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7

SATISFACTION 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7

TASK

13

TASK

14

TASK

15

TASK

16

TASK

S 13

–16


TASKS 17–19: Look over the journal page. Create a new journal entry using the provided data. Share your journal as a webpage.


Q1. How would you describe the journal to somebody unfamiliar with it?

• “It’s like a blog probably. I’m assuming you’ll be able to put pictures & words, maybe video.”

• PC: “A monthly newsletter. That’s kinda cool, if you actually do it.”

• “Social media.”• “Status update on Facebook.”• Already has two or three things

like this, wouldn’t use this.

• Understood what it was• “It seems like it is an open

forum so anyone who goes in there could share.”

• Understood what it was• But noted too much text to

understand the journal with just a brief look.


ERRORS 0 0 0 0

NOTES

• Completed easily• BUG? Photo wouldn’t load in

journal entry. Did load as the family photo, user was puzzled.

• BUG? Canceled something, then taken to shopping list?

• NC: Again, dislikes the ads.

• She was able to create the journal entry successfully.

• “I’m going to click on the family newsletter to see what it is.”

“Kind of like sharing on Facebook. I get it”

COMPLETION COMPLETED COMPLETED NOT COMPLETED COMPLETED

ERRORS 1 0 1 0

• PC: “Monthly family newsletter, I really like that idea.”

• Trouble finding webpage option• “Are the pictures going to

show up on Google?”• PC / NC: “Protect keywords

like your kids names... Oh! That’s nice. But that might be frustrating for people... people get really confused about stuff.”

• Didn’t see how to continue from preview page: “I didn’t see this up here, that there was a bar for this tutorial thing.”

• Journal photo wasn’t loading still, appeared as family photo. “I’m confused.”

• Seems tired by this point. Tried to skip ahead to “invite” in journal set up wizard.

• On preview page, clicked on next wizard item rather than the “Yes share our journal” button. Then started over b/c clicking on the wizard item wasn’t taking him forward.

• “I’m kinda getting stuck here...Aha! I have to click yes share out journal.”

• Didn’t seem to like the extensive journal setup.

• Didn’t find the webpage sharing option.

• Wondered if there was a way to make specific entries shared/not shared (there is).

• For a newsletter, would like to change the day or frequency with which it goes out.

• Wanted to skip ahead to “invite” in the journal set up wizard, just like participant 2.

• Dads seems distinctly uninterested in journaling.

• NOTE: Did not actually set up the journal webpage because the browser wouldn’t scroll so he could click next button at bottom of page (so couldn’t move ahead in the set up wizard). Probably could have clicked buttons in wizard bar, though? This participant’s survey data for this task may not be as reliable because he had an incomplete experience.

Q1. Anything you would change about your prior description? Is anything unclear?

“ It seems like Cozi is trying to take all the popular social things that we already use and make them more family friendly.”

“ Can I customize it to send only certain things to certain people?”

• “I’m assuming you can change the day that the newsletter goes out, or the frequency. I wonder about the attachments with photographs, if a certain size in the email will exceed what their inbox can hold.”

N/A, couldn’t go through the whole journal task because of browser malfunction.

Q2. How do you feel about sharing? Which method would you prefer?

“ I don’t really like either of those options... I’m not clear if it would searchable. I like the privacy option on blogspot, that it’s invite only.”

“ Not so sure I like it. Privacy is important.”

• “Maybe some of the journals I want to share with everyone, but some I just want to share w/ certain family and friends. I hope there’s a way to do that.”

• “I like little prompts when I enter a site, something that might say ‘Hey, somebody’s journaled’.”

N/A, couldn’t go through the whole journal task because of browser malfunction.

EASE 1 2 3 4 5 6 7 1 2 3 4 5.5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7

WORKED AS EXPECTED 1 2 3 4 5 6 7 1 2 3 4 5.5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7

SATISFACTION 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7

TASK

17

TASK

18

TASK

19

TASK

S 18

–19


DEBRIEFING


Q1. If you wanted your spouse to view your Cozi calendar, what would you tell them to do?

“ I would tell him to check his email because I probably invited him. I wouldn’t tell him to go to the site, because he would just get confused. He might end up setting up a whole ‘nother family calendar.”

Would send her a link and tell her how to do it if she had a question.

• Would expect to share it with her spouse using a link, possibly.

• “I have no idea, I guess there must be a link share? I didn’t see it. I don’t remember.”

• T o share with spouse, would simply show it to them himself.

Q2. You set up Cozi on your home computer and enter an appt.How would you view it at work?

“ I would either just have the app on my phone... or have it synced somehow. I dunno. Probably an app on my phone.”

On my phone or computer. “I’m assuming I can just go online and check it out”

I would need to find a computer to log into to do this because we don’t have them at work.

OVERALL EASE 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7

EASE OF CALENDAR 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7

EASE OF COZI V. OTHER TOOLS 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4 5 6 7

EASIEST FEATURE?

Calendar Calendar Quick add for events To do list

MOST DIFFICULT?

Lists: “They can get confusing.” Journal To do lists Calendar

OTHER NOTES

• Liked the “everything in one place” aspect of it.

• Said might even use it herself... her impression of “just another thing to do” diminished as she went through the tasks.

• Completed entire test in about an hour. The other three participants all took about an hour and a half. Her prior experience using Google calendar probably made Cozi much easier to learn.

• Thought it would be useful but wanted to only use one system, not 5 different online calendars and organizers and papers and notes.

• Did not like ads and was the type of person who might leave the site before even complet-ing registration due to this.

Was suspicious of the privacy and marketing of the site initially, but left feeling like it might be a handy tool for organizing things.

Said he probably would not use something like this since he finds it easier to just talk with his wife and store info in paper or offline.

Cozi Usability Test Report - User Experience Design ...€¦ · The following report summarizes a...

Documents

Transcript of Cozi Usability Test Report - User Experience Design ...€¦ · The following report summarizes a...