Usability Evaluation of thetwo most popular iOS PDF Reader

Project Overview

More and more books are published in digital forms known as eBooks. There are several good eBook Reader applications available on the iOS App Store. According to the amount of downloads Acrobat Reader is the most popular one, but is it also the most efficient one?

The purpose of this usability case study was to compare the free Acrobat Reader with an alternative PDF application. After searching on the iOS App Store, PDF Expert has been chosen as a good option, because of its rating and positive feedback comments.

UX Case StudyNovember 2015 - Januar 2016

Goals

The primary research goals of this project was to:

• evaluate and compare the usability of the two PDF applications

• measure the efficiency in time of task completion

• get participants subjective experience and satisfaction

• compare participants subjective feedback with the efficiency of the task completion

Before we started with the test phase, we formulated a null hypothesis and an alternative hypothesis.

h0 : PDF Expert is equally efficient as Acrobat Reader

h1 : PDF Expert is more efficient than Acrobat Reader

Hypthesis

My Contribution

Responsible Activities

Test Participants Recruitment

Usability Test Plan

Coding Scheme

Test Moderation

Test Data Analysis

Usability Lab

Google Forms (Questionnaires)

Observation XT

Used Design Tools

Role : UX Designer

The outcome of this usability case study shows that PDF Expert is more efficient than Acrobat Reader. The results of the two-sample t-test illus-trate that there is a significant difference between Acrobat Reader and PDF Expert. The wilcoxon-test had similar results. The results of the post test questionnaires did not have any significant patterns. Collected data produced indications of where to focus our efforts. The identified problem areas were analysed which produced a list of usability viola-tions concerning constraints, feedback and visibility.

Outcome

Test Environment

The usability tests was conducted at the usability lab of Uppala Univer-sity, located at the faculty Ekonomikum. The lab consists of an observa-tion room and a test room which are detached by an one-way mirror so that the participants cannot see the oberservers.

The observation room is equipped with a computer and the requried applications for observing the usability test were preinstalled. The Ob-servation XT software was later used for analysing the recorded data.

We set up the test room, where the usability test takes place, with two cameras and one microphone. The iPad was connected via a DVI cable in order to record the screen. Moreover, the iPad was secured on the ta-ble to ensure a proper view on the tablet.

Recruiting Criteria

A statistical hypothesis test about a population or distribution is usually done with a random sample that represents a subset of the population. The students at Uppsala University constitute the frame population in this study. They are a representative group for the target population.

Nevertheless, we had a selection process because we wanted to elimi-nate experience bias - the participants should not have any experience with an iOS PDF application - and we tried to have an equal gender ratio. The age range was between 18-35 years.

Procedure

General Information

The usability tests involved two different roles:

• A moderator, whose task was to lead and support the participant throughout the test tasks and surveys.

• Two observers, whose task were to take notes of exceptional inci-dents and to make sure that the video recording was complete.

Test Procedure

The sessions began with the moderator giving the participant an intro-duction about the project and the data collection. The participant was asked to sign the participation consent form. Before entering the test room, the moderator showed the participant the observer room and in-troduced the observers. In the test room, the moderator explained the test setup and handed over the test papers to the participant.

Test Structure

Each application had it’s own test paper. This allowed us to switch the test orders to eliminate bias. Each test paper also included at the end a questionnaire about the application’s usability and the participant’s experience with it.

Methods

• Observation and video recording of the tablet screen and of the par-ticipant’s hand movement.

• The participants were requested to think aloud their thoughts

• Standardised usability questionnaire given after testing

Data Collection

Data Analysis

• Statistics by comparing task times for each PDF applications: Types of statistics: Q-Q Plots, T-Test and Wilcoxon signed-rank test

• Quantitative data in graphs from the questionnaire

• Video analysis

The result is based on a total of 14 observations (8 women and 6 men). Throught the results section, we will refer to Acrobat Reader as AR and PDF Expert as PDF E.

Results

Task Median Mean Minimum Maximum Std. DevAR: Find Chapter 57,06 79,86 11 293 76,38PDF E: Find Chapter 28,2718 39,36 10 91 28,89AR: Find Headline 8,19 12,28 6 60 13,53PDF E: Find Headline 6,021935 8,07 1 14 6,82AR: Highlighting 30,28 54,76 14 219 58,71PDF E: Highlighting 32,5429 49,14 8 174 46,13AR: Add Note 57,05925 32,64 11 145 32,66PDF E: Add Note 28,665 35,36 8 135 31,13AR: Undo 19,78175 34,58 7 128 33,78PDF E: Undo 9,14514 12,36 4 26 8,08

This above table refers to the total time it took for the participants to complete each task for each application. The median in all the test cases for PDF Expert, except for highlighting, is lower than Acrobat Reader. The mean is slightly different, but again for all the test cases, except for adding a note, PDF Expert has a lower mean than Acrobat Reader. If we look at the standard deviation, PDF Expert’s standard deviation is high but Acrobat Reader’s is higher for all the test cases.

Descriptive Statistics

Student’s T-Test

A t-test was applied to determine if there are any significant difference between the applications. The pre-condition to use a t-test is that the pop-ulations should be normally distributed. We used the graphical method Q-Q plots to examine whether the test data is normally distributed. The Q-Q plots results illustrate that all variables are normally distributed.

Quantitative Data

The quantitative data were analysed using a student’s t-test and the wil-coxon signed rank test.

Results from the paired two-samples test:

Sig. (2-tailed)

Pair 1 AR: Find Chapter - PDF E: Find Chapter 0,119Pair 2 AR: Find Headline - PDF E: Find Headline 0,323Pair 3 AR: Highlighting - PDF E: Highlighting 0,428Pair 4 AR: Add Note - PDF E: Add Note 0,837Pair 5 AR: Undo - PDF E: Undo 0,043Pair 6 Total AR - Total PDF 0,015

Results from the paired two-samples test correlations:

Correlation Sig. (2-tailed)

Pair 1 AR: Find Chapter - PDF E: Find Chapter -0,220 0,449Pair 2 AR: Find Headline - PDF E: Find Headline 0,083 0,779Pair 3 AR: Highlighting - PDF E: Highlighting 0,911 0,000Pair 4 AR: Add Note - PDF E: Add Note -0,024 0,936Pair 5 AR: Undo - PDF E: Undo -0,126 0,667Pair 6 Total AR - Total PDF 0,487 0,077

Wilcoxon Signed-Rank Test

According to the Q-Q plots, the study sample has a normal distribution.Yet it is dubious given its size thus a wilcoxon signed-rank test was done.

Results from Wilcoxon test statistics:

Asymp. Sig. (2-tailed)

AR: Find Chapter - PDF E: Find Chapter 0,245AR: Find Headline - PDF E: Find Headline 0,158AR: Highlighting - PDF E: Highlighting 0,470AR: Add Note - PDF E: Add Note 0,594AR: Undo - PDF E: Undo 0,030Total AR - Total PDF 0,005

Qualitative Data

• I think that I would like to use this system.

• I found the system unnecessarily complex.

• I thought the system was easy to use.

• I think that I would need the support of a technical person to be able to use this system.

• I found the various functions in the system were well integrated.

• I thought there was too much inconsistency in this system.

• I would imagine that most people would learn to use this system very quickly.

• I found the system very cumbersome to use.

• I felt very confident using the system.

• I had to learn a lot of things before I could get going with this system.

• Overall, I am satisfied with the ease of completing the tasks

• Overall, I am satisfied with the amount of time it took to complete the tasks

Post-Test Questionnaires

SUS Score

The system usability scale (SUS) is a reliable method to measure the us-ability of two applications and it can be used on small sample sizes. It consists of a 10 item questionnaire with 5 response options: from strong-ly agree to strongly disagree. The response options were scaled from 1 to 7. The grades were graded according to the below grading scale.

Descriptive Statistics

The results in the table below were calculated from the total score of each question. The questions are renamed to Q# and refer to both ques-tionnaires.

Rating Scale: 1=Strongly disagree | 7=strongly agree

Question ARMedian

PDF EMedian

ARMean

PDF EMean

ARStd. Dev.

PDF EStd. Dev.

Q1 4 5 5 5 1,237179148 0,8112726208Q2 3 3 3 3 1,486332289 1,222660198Q3 5,5 5 6 5 1,205853073 1,171515676Q4 1,5 1 1 1 1,222660198 1,012674777Q5 5 5 6 5 1,444553458 1,171515676Q6 2,5 3 2 3 1,386891988 1,23097771Q7 5 5 5 5 1,012674777 1,245399698Q8 2 4 2 3 1,624116714 1,069044968Q9 4,5 5 6 5 1,708322963 0,9715336078Q10 2 2 2 2 1,287696884 1,399708424Q11 5 5 6 5 1,423204203 1Q12 5 5 6 5 1,878992348 1,309307341

The median values are mostly the same, so are the mean values. Hence, there is not a distinctive difference between the Acrobat Reader and PDF Expert. However, the standard deviation values show some significant disparity. PDF Expert has a lower standard deviation in all the questions except for Q7 and Q10.

The SUS score of the post test questionnaires:

System SUS Score

Acrobat Reader (AR) 70.01PDF Expert (PDF E) 68.64

The table shows that there is no substantial difference between the par-ticipants liking. The table also shows that the softwares according to Figure 3 is “OK” and also very close to “Good” in the adjective ratings. In other words, this SUS-score states that Acrobat Reader is consider to have acceptable usability and that PDF Expert is just below what is con-sidered to be acceptable.

The standard deviation for PDF Expert is high but they are higher in all cases for Acrobat Reader. That means that the results are spread out. The high standard deviations is probably caused by the sample size. However, the interesting fact is that Acrobat Reader has a much higher standard deviation in all cases. We assume that this is due to the Ac-robat Reader’s slightly worse response for touch gestures. PDF Expert has a better touch response. But this be further analysed using software benchmark tests.

The results of the two-sample t-test showed that there is a significant difference between Acrobat Reader and PDF Expert. In table 1 you will find the significance-value of 0.015 which is well below 0.05. This em-pirical evidence is enough to reject the null hypothesis with a signifi-cance level of 5%

h0 : PDF Expert is equally efficient as Acrobat Reader

h1 : PDF Expert is more efficient than Acrobat Reader

Analysis and Conclusion

If one considers the individual task pairs it can be distinguished that only one of these is significant. Pair number 5 “AR: Undo - PDF E: Undo” has the significance-value of 0.043. Pair number 1 “AR: Find Chapter & - PDF E: Find Chapter” is close to be significant with a value of 0.119. Re-garding correlation, pair number 3 “AR: Highlighting - PDF E: Highlight-ing” has a correlation of 0.911 and the significance-value of 0.000 and is the only test pair that has significant correlation. None of the other pairs is close to being significantly correlated.

The wilcoxon-test had similar results and showed that there is a signif-icant difference between “Acrobat Reader - PDF Expert” and the indi-vidual task “AR: Undo - PDF E: Undo”. If we look in table 3 we can see that it is quite the same but slightly different significance-value of 0.005 respectively 0.030.

Since both tests showed similar results, it is reassuring to say that our sample size was not a major problem. However, it would be unreason-able to assume normal distribution. At least for some of the Acrobat Reader variables, given that they contain outliers and large standard deviations. A boxplot had therefore been a good graphical method to get an additional clarity. Some critics can be directed to our sample. It would be unreasonable to assume that it is a random sample, given that all participants were found in Ekonomikum. It would also be un-reasonable to assume that the results can be generalised to the entire population. Some reservations existed due to the fact that there were no specific background factors of the sample (gender, age etc.). The partic-ipants also tested the softwares in fixed orbit, some participants started with Acrobat Reader and some with PDF Expert.

Considers the post test questionnaires, it can be distinguished that the answers are similar to each other. Regarding the descriptive statistics table it can be distinguished that the standard deviation is bigger in

general for Acrobat Reader, which is also the case in the t-test. This may be because our sample is so small. It may also be that some users with less experience with tablets or programs in generell have easier to take in Acrobat Reader, when PDF Expert has a wider range of advanced fea-tures. Regarding the SuS score is can be distinguished that it is not a big difference. Acrobat reader landed on 70 and PDF expert 68, (rounded score). This may indicate that the results of the t test and Wilcoxon test did not reflect the actual truth. It can also be due due to the similarities of the two softwares, that they both are launched and established prod-ucts.

After discussing and analysing our test results, we gained an insight into the softwares advantages and disadvantages from an efficiency perspec-tive. We studied these differences and discovered a number of usability violations. These discoveries form the key findings in this evaluation and is presented below.

Recommendations for Improvements

In this chapter, the findings of the major usability violations will be dis-cussed and rated according to the below severity rating table. Further recommended solutions possibility will be given. The usability viola-tions is based on participants data and was identified in the video re-cordings after analysis of the statistics.

Severity Description

1 Could be fixed2 Should be fixed3 Must be fixed

Show the Right Page Number

One of the major problem in Acrobat Reader that most the participants struggled with the discorrespondence of the page number. As shown in figure below Acrobat Reader is showing “Page 151” on the slider but on the document it says “Page 133”. This is not the case in PDF Expert as shown. This is a small fix that can be made by Adobe and would avoid confusion and substantially increase the findability of the pages.

Usability Violation: Visibility, Feedback Severity: 3

Acrobat Reader

PDF Expert

Placement of Chapter Button

Another problem faced during finding a chapter in Acrobat Reader was finding the “Chapter Shortcut” button. The button is positioned in the bottom-right corner of the interface which led to a longer finding time. Many of the participants rather went for a workaround than finding the “Chapter Shortcut” button. In Comparison to PDF Expert the “Chapter Shortcut” button is placed in top-right corner which gives the function according to the test results a better visibility.

Usability Violation: Visibility Severity: 2

Acrobat Reader

PDF Expert

Searching in Acrobat Reader

An annoying problem in Acrobat Reader that some of the participants faced during finding a chapter was while using the search function. When the search function is active then no other actions can be made. The user needs to exit search mode by explicitly clicking the “Cancel” button before being able to do anything else. This led to confusion and frustration, and help from the moderator was needed. A recommen-dation for this problem would be to prompt a confirmation message if user is done with searching as soon as the system detects that user tries to do another interaction.

Usability Violation: Constraint, Feedback Severity: 2

Acrobat Reader

Undo in Acrobat Reader

Undoing comments and highlights was something that was not clear in Acrobat Reader. The “Undo” button is placed on a position that is far away from the area where the last interaction was made. Also the button is only in the written text form. The common icon for undo is not shown at all. This problem could be solved by either placing the button on the popup button list, for example next to the “Clear” button, or changing the button from text to icon. As a side note, the “Clear” functionality just removes the highlighted part and “Undo” functionality is undoing the last action.

Usability Violation: Visibility Severity: 2

Acrobat Reader

PDF Expert

Notes with Predetermined Text in Acrobat Reader

One small feature that Acrobat Reader uses, is that it has a predefined text (“viktigt” swedish word meaning important) when you add a note. This led to confusion for the majority of the test participants because the predetermined text is not a placeholder and it cannot be removed. The removal of the predetermined text or having at least converting it to a placeholder would rapidly increase the usability of this function.

Usability Violation: Visibility, Feedback, Constraint Severity: 3

Acrobat Reader