White Paper on the Usability of Publisher Wildfire Info Graphic

8/2/2019 White Paper on the Usability of Publisher Wildfire Info Graphic

1/9

Report on the Usability of Publisher Wildfire Infographic

Publisher Team 2: Mark Johnson, Sarah Lynn, Pierce Trey, Morgan Weir, CrystalZarate

Introduction

The purpose of this document is to detail the methods and results of a usability test onthe effectiveness of two infographics regarding wildfires, which may be found in

Appendix A. Both were designed to increase readers knowledge regarding wildfiresuppression, but the data in one was presented in a gains frame# while the secondpresented the same data in a losses frame. The goal of this usability test was todetermine the efficacy of these infographics on readers. The test was designed usingmethodological triangulation and will be further detailed in the methods section of thisdocument.

MethodsEach group member tested one gains and one losses frame of the infographic for a totalof ten participants. These selections were not random, but based upon people groupmembers knew (for instance, roommates). To protect anonymity, participants wereassigned a number which was written at the top of all their test material. They weregranted an unlimited amount of time to view the infographic assuming that when viewedin regular life, viewers will not have a time restriction. Additionally, the amount of timespent viewing the infographic may yield insight toward its effectiveness and appeal. This test was analyzed using methodological triangulation which uses threeindependent data platforms to collect data then analyze the results. Through this, theeffectiveness of the infographic was determined. This group used surveys, talk-aloud-protocol, and interviews as the triangulation points. The experimental methodsimplemented in each of these are detailed below.SurveysSurveys were conducted before and after viewing the infographic. The pre-surveyconsisted of two parts; the first was composed of questions regarding participantsbackgrounds (Appendix B) while the second inquired about their current viewsregarding wildfire management (Appendix C). The post-survey was completed afterreading the infographic and asked the same questions as part two of the pre-survey.Because the surveys were completed on a scale of negative three to three, it waspossible to sum participant responses on both surveys. The sum of the responses willhereafter be referred to as the score. Knowledge gain was measured by calculating thebetween the pre- and post-surveys.

Talk-Aloud ProtocolThe talk-aloud portion of the test occurred simultaneously with reading the infographic.This measurement required viewers to tell the testers what they were thinking andwhere they were looking whilst viewing, much like thinking aloud. Assuming people donot usually do this while reading, participants were asked to complete a practice talk-


2/9

aloud with a different infographic before viewing the infographic being tested. Thepractice infographic is attached as Appendix D.

InterviewInterview data was the final data point collected from this usability test. The interview

was conducted after the post-survey and the questions focused on how the infographicaffected the participant and what the readers felt could be improved. The interview wasconducted one-on-one in a casual setting. The interview questions are attached asAppendix E.

ResultsThe overall goal of this usability test was to determine the effectiveness of twoinfographics. The effectiveness was determined by participant feedback regarding thedesign as well as knowledge gain. The concluding result was that the infographics

designs were insufficient in attracting viewers and being credible. The results also showthat while participants did acquire knowledge from the infographic, it did not motivatemost to take further action.

Figure 1. Interview Responses.

Credibility and Motivation to ActionFigure 1 displays interview responses when participants were asked the following threequestions:


3/9

Did you find the information credible and presented in a manner whichencouraged you to take the matter seriously?

Did the infographic motivate you to take future action? Do you feel you learned anything?

As seen in Figure 1, 75 percent of participants indicated that they learned informationbetween the pre- and post-surveys. However, 60 percent of participants felt theinfographic lacked credibility and 70 percent of participants were not motivated to takefurther action.

Figure 2a. Knowledge Gain for Losses Infographic. Note that participants seven andten did not change any answers between the pre- and post-survey and thereforemeasured no knowledge gain.


4/9

Figure 2b. Knowledge Gain for Gains Infographic.

Knowledge GainFigures 2a and 2b display the knowledge gain of each participant. This data wascompiled by comparing the data from pre-survey and post-survey questions as detailedin the methods section of this document. As seen in Figure 2a, the losses frame,participants two and four displayed slight changes in knowledge while participant sixgained a significant amount. However, participants seven and ten did not display any

change in score indicating no knowledge gain. Despite this, the average result was apositive knowledge gain. Figure 2b shows that although all those who viewed the gainsinfographic showed a change in knowledge, the average result was negative. In bothframes, the average knowledge gain was very small indicating most participants did notgain a significant amount of knowledge.


5/9

Figure 3a. Average Effectiveness Rating of Prevention Techniques (Gains Frame)

Figure 3b. Average Effectiveness Rating of Prevention Techniques (Loss Frame)EffectivenessThe final section of the surveys contained a series of five statements recommendingpreventive actions against wildfires. Participants were asked to rate the effectiveness ofthese actions on a scale of negative three to three (three being highly effective). Figures3a and 3b display the number of each statement and the average participant rating of it.


6/9

As seen in Figure 3a, the gains infographic, all of the participants felt that the actionswere at least somewhat effective both before and after viewing the infographic. Theeffectiveness ratings were low, averaging between one and two. In the post-survey,questions two and four had the same rating as in the pre-survey. Questions one andthree decreased in effectiveness after viewing the infographic while question five

increased.

According to Figure 3b, in the pre-survey participants viewing the losses infographicfound statements two, three, and five effective. However, question one was neutral andquestion four was slightly ineffective. In the post-survey questions three, four, and fivedisplayed the same effectiveness rate as seen in the pre-survey. Question twodecreased in effectiveness, while question one made a slight increase.

Figure 4a. Average Likelihood of Performing Prevention Technique (Gains Frame)


7/9

Figure 4b. Average Likelihood of Performing Prevention Technique (Losses Frame)Likelihood of Taking Preventative ActionIn addition to stating prevention techniques, the final portion of the surveys askedparticipants how likely they would be to perform the action. The overall purpose of theusability test was to determine if people are motivated to take action against forest firesafter viewing the proposed infographics. As displayed in Figure 4a, most participantswho viewed the gains frame responded that they would be the most likely perform the

actions described in statements two, three and five. However, statements one and fourare not likely to be performed by viewers. There was no change in the responsesbetween the pre- and post-survey regarding questions three and five. Questions oneand two increased in likelihood while question four decreased.

Figure 4b displays the same likelihood of performing the prevention technique when itwas presented in the losses frame. On average, participants found themselves likelyperforming the prevention techniques in statements two, three, and five. As in the gainsframe, viewers were unlikely to perform the indicated prevention technique described instatements one and four. After viewing the infographic, responses to statements threeand four remained the same. Responses to statements two and five indicated an

increased likelihood of performing the prevention technique. Statement one had themost significant change increasing in likelihood from an average rating of -1.2 to 0.

AnalysisIn general, the results provide information about aspects of the infographic that shouldbe changed rather than positive attributes of the infographic. They present an overallneed for change in the infographic content. Using methodological triangulation of datacollection, some conclusions can be drawn regarding the infographic.


8/9

Many participants changed only a few answers between the pre- and post-surveys,suggesting that the infographic had little or no effect on them. However, 60 percentreported knowledge gain from the infographic, as seen in Figure 1. Average knowledgegain measured marginally higher from the losses frame than from the gains as shown in

Figures 2a and 2b. Despite the reported knowledge gain, 60 percent of viewers foundthe information not credible, also seen in Fig. 1. This might suggest that the material iselementary or not well referenced.Neither frame appears to have a significantly greater influence on the participants thanthe other; they both proved equally ineffective. As noted before, the losses frameproved to convey more knowledge than the gains frame (Figures 2a, 2b). Also, theparticipants that viewed the infographic with the gains frame found the preventiontechniques described in the final portion of the survey more effective than those thatviewed the losses infographic. However, those that viewed the infographic in the lossesframe were overall more likely to perform the actions.

As seen in Figure 1, the infographic was not successful in motivating the participants topursue further action. There were complaints regarding the credibility of the informationand the lack of specific wildfire data which may be part of the reason participants werenot motivated toward further action. As motivating viewers was the primary goal of theinfographic, this indicates the infographic would benefit from adding more motivatingelements as well as additional data.

The talk aloud protocol information gathered from the usability test produced littlehelpful information. General information such as approval of the colors used in theinfographic and expressing boredom with the infographic was collected from this datapoint.DiscussionThis infographic needs many revisions. The flow of the visual layout seems to begenerally acceptable, but the content of the document needs to be brought to a higherlevel, as it was perceived more often than not as common sense material. Thepreventative techniques presented on the infographic were perceived by the testsubjects as being ineffective and unimportant. The presentation of the information wasresponded to as being not serious enough and even described as something you wouldpresent to a childboth because of the physical design, and the simplicity of thepreventative techniques. The framing was shown to be irrelevant, as there was nosignificant change in data between frames. Both frames failed to produce a significantamount of knowledge gain when the answers of the post-survey were compared withthose of the pre-survey. The data from the losses frame showed a small improvementover the gains frame in terms of knowledge gain, but the difference was not enough toindicate a clear lead, especially considering the small size of the data set.Due to the lack of effectiveness of this infographic, it is recommended that the currentinfographics not be used for a large scale study. The changes required to increase the


9/9

effectiveness of these documents to a useful level may be more effort than it is worth; itwould be better for entirely different graphics to be used. The physical design of theinfographics have good aesthetic appeal (though it was considered too cartoony by afew participants, which correlated to a decreased credibility of the data in their minds),but it failed to convey any useful information. In addition to the need to revise the

infographics, some of the survey questions need to be revised. Some questionssupplied for the study (appendix B) were phrased in an unclear manner, and a numberof participants expressed confusion over them and found it difficult to assign anumerical value to their answer (and mentioned that their answers may have changedbetween the surveys because they were not quite sure how the question was posed). Inorder for the survey to be effective, these questions need to be revised. The talk-aloud protocol used in this study failed to provide any information relevant tothe framing of the infographic, and mostly just confirmed that people tend to read top tobottom, left to right. Since the layout of the information did not change between thegains and losses frames, this data on layout provides no helpful insight into the framing

of data used.

It is important to keep in mind that the infographics tested in this study were designedby students with no experience in design or the framing of information. This lack ofprofessional design experience could explain why there was not much data pertinent tothe framing of information. In order for a large scale study on framing to be conducted,the infographics need to be properly framed. This study also sampled much too small ofa population for the data to provide definitive evidence for the effectiveness of framing.

White Paper on the Usability of Publisher Wildfire Info Graphic

Documents

Transcript of White Paper on the Usability of Publisher Wildfire Info Graphic