Comparative evaluation of Norwegian airways flight booking system graphicaluser interfaces

Project group 11

Joakim Larsson∗

School of Information andCommunication Technology,

KTH Royal Institute ofTechnology

Stockholm, Sweden

Robin Tillman †

School of Computer Scienceand Communication,

KTH Royal Institute ofTechnology

Stockholm, Sweden

Alex Wennberg‡

School of Information andCommunication Technology,

KTH Royal Institute ofTechnology

Stockholm, SwedenJuho Kastemaa§

School of Information andCommunication Technology,

KTH Royal Institute ofTechnology

Stockholm, Sweden

Abstract

The usability of webpages as a research topic has been around fora good amount of time. Effectiveness as well as satisfaction haveboth been studied in the area of usability.

This study examines the understandability of the Norwegian Air-line’s flight booking system interface. It is a comparative study,examining the newly released interface in relation to the interfacebeing replaced. To narrow the study down we have focused on theunderstandability of the interface which is used to search for avail-able flights.

The project was conducted by letting 12 persons test the two inter-faces. They were divided into two groups, one which was taskedwith testing the old version, and one that was tasked with testingthe new version. They were all given the same task, searching fora flight between two predetermined airports, for a company of twoadults, a child and an infant. After finishing the tasks the users filledout a questionnaire and then partake in a short interview.

The project concludes that no statistical differences in understand-ability between the two interfaces were found, but we have identi-fied a few changes that could be made to prevent user errors whileusing the system. We recommend Norwegian to help users withspelling errors that might happen as they type in the destinations,give them further visual feedback on the size of the company thatthe user is booking a ticket for, and make the static airport sugges-tions more discoverable.

∗joakim7@kth.se†robint@kth.se‡alexwen@kth.se§kastemaa@kth.se

1 Introduction

Norwegian Airlines launched a new, redesigned user-interface (UI)for their website in early autumn of 2015. At first it was a betaversion alternative to the standard UI, and was added to Norwe-gians subdomain. With the both versions of the Norwegian web-site available, a possibility to do a comparative study between thethen currently used UI (see Appendix B) of the Norwegian flightbooking system and the new beta release UI (See Appendix A) ap-peared. The study was limited to investigate the flight booking sec-tion, which is one of the main functions on the Norwegian Airlineswebpage.

The aim of this comparative usability study, was to investigatewhether the initial steps of the flight booking functionality in thenew release of the Norwegian airways flight booking system wasmore understandable than the previously used UI. The goals wereto investigate the end user satisfaction and the usability factorcalled understandability using qualitative interviews and quantita-tive questionnaires. The target group for this study were a part ofNorwegian booking systems end users, which in this study werestudents at the Royal Institute of Technology (KTH), aged between20 and 30 years.

The effectiveness as well as the efficiency both have a strong impacton the satisfaction of the user when using an online booking system[Matzler et al. 2006]. Furthermore, the perceived satisfaction canbe related to the truthfulness of the system [Flavian et al. 2006].When using a flight booking system the user expects both a trustful,efficient and an effective system which should be both easy andsafe to use. The measured usability factor of this study will beunderstandability. This study was limited to the initial steps of thebooking process, processing basic flight information such as dates,destinations and passengers.

Few studies of usability evaluation of airline booking systems havealready been done. Hasan [2010] stated in his research that there isa lack of usability evaluation in Web-Based Online Airline BookingSystems. In his case study Hasan [2010] came up with a proposedmodel to evaluate Malaysia Airline, Royal Jordanian Airline andEmirates Airline systems. Chen, Paul, Kibaru, Ma, and Saparova[2015] used think aloud interviews to discuss the challenges en-countered during iterative testing of an insurance company’s home-page prototypes in order to understand what elements influencesusers preferences of designs the most. The study found that con-

tent, layout and visual appearance had the most influence to theuser experience.

1.1 Research question

For this project we compared Norwegian newly released websiteand Norwegian old website in an effort to study the understandabil-ity of both system from the users perspective. Our research questioncan be worded as ”are the initial steps of use in the new release ofthe user-interface of the Norwegian airways flight booking systemmore understandable than that of the old booking interface it hasreplaced?”

2 Method

To limit the scope of this study we have focused on evaluatingthe understandability of the two flight booking User Interfaces ofthe Norwegian Airlines webpage. More specifically we decidedto evaluate the understandability of the interface that is used whenfinding a ticket as well as the search tool which the user uses to finda flight which matches their needs. This has been done through userstudies where participants have been gathered to perform a prede-termined task with the systems, and answer both a questionnaire aswell as participate in a short interview afterwards. The tests wereconducted in a room with two personnel present; one who led thetest and the interview, and one who took notes.

Norwegian Airlines webpages current interface for searching forflights can be seen in Appendix A. The interface is large, centeredand distinguished from the background by being shown as an ele-ment floating above the webpage. It initially shows two search barswhere you type in the two airports that you want to fly between.When those two terms have been typed in, further options appearsfor determining the search. There are boxes for determining if theflight is for a one-way ticket or a return ticket, a calendar for defin-ing the dates for the flights, textboxes for determining the size ofthe company that is flying and further text boxes defining whetherto show the cheapest flight or a flight as close to the desired flyingdate as possible.

The old version of the interface that has been replaced by the cur-rent version is pictured in Appendix B. This interface is confinedto a column on the left hand side of the webpage. Unlike the newversion the interface is not distinguished from the rest of the web-page, and every element of the interface is visible at all times. Thindrop down boxes are used to handle all text input, resulting in aninterface which is smaller and much more compact than in the newversion.

Norwegian phased in the new version of the webpage during theevaluation session of the old version. This resulted in the old web-page being made unavailable after two people had evaluated it. Tobe able to continue evaluating the webpage we used an archive ser-vice1 to find an old archived version of the Norwegian webpage,and used a version saved 2015-08-252. The archival service hasntbeen able to properly save an element of the interface. The popupcalendar renders as a static image, covering parts of the user in-terface. To make the archived version more closely resemble thewebpage that we intended to evaluate we chose to remove the cor-rupted elements using Chrome developer tools. This means that theold version that most users evaluated lacked a feature which wouldhave been present in the live version of the web interface. Further-more, one of the users was accidentally given a version archivedin 2014-11-21, a version where one of the airports which the task

1https://archive.org/web/2https://web.archive.org/web/20151025114135/http://www.norwegian.com/uk/

involved was missing. This led to the user being confused about themissing airport during the test session, which might have impactedtheir perception of the interfaces understandability.

Since the two UIs are similar, using one system before the othermight cause a learning effect large enough to impact the outcomeof the study. To reduce this impact, a between group design wasused when modeling this study. By assigning users a single systemto use we try to reduce the impact of previous immediate experiencewith a similar system would have on how the user perceives the testsession. However, we are also losing out on the insight that theusers could reach when comparing the two systems. To get theusers to interact with the interface to as high a degree as possible,a task was designed to try to both use large parts of the interface,while still trying to mimic the use of a casual use session.

Participants were chosen through the contact networks of the au-thors, and they got to choose timeslots using Doodle. Two partic-ipants were, however, recruited through asking people in the sur-roundings of the laboratory. All participants chosen were studentsof KTH, did not partake in the evaluation methods course, and werewithin the age span of 21-27 years old. A total of twelve partici-pants were recruited.

The participants were divided into two groups of six people each,and each group were in turn assigned one of the two UIs of thewebpage. Our intention was to have everyone who signed up fortimes on the first day to use the old system, but due to the old web-page being made unavailable in between two evaluation sessionswe had change that part of the plan. This resulted in a test orderwhere the old system was tested out by the first two participants,the next two participants used the new system, three participantsused the old system, four participants used the new system and thelast participant used the old system.

To examine if our choice of groups had skewed the average experi-ence of using systems like these amongst the test groups, we askedeach participant to estimate their own experience with online ticketbooking systems, as well as their previous experience with Norwe-gian Airlines webpage. The participants that used the new versionreported a lower level of experience with ticket booking systemsthan the group using the old version of the webpage, which couldhave impacted the results of this study.

The evaluation sessions were held in KTHs multisensory studio, asecluded room equipped with computers and a myriad of measur-ing devices. Each participant was shown the way into the room,and presented to the three or four group members present in theroom, see Figure 1. A single sheet of paper was read aloud, withthe participant being able to read as well, with instructions of theevaluations session, see Appendix D. Two researchers, the modera-tor and the observer, were located in the study room with the studysubject. The moderator conducted the interviews and the observertook notes nearby during the user session using Google forms. Oneof the researchers was located outside of the room and his task wasto measure the time taken during the whole study session. The par-ticipant was asked to perform a single task of booking a flight be-tween Birmingham and Barcelona at December 15th for two adults,a child and an infant. Afterwards the participant was asked to fillout a questionnaire, see Appendix C, followed by a short interview,see Appendix C. The user interviews were also voice recorded forlater use.

3 Results

In this section the result of two analysis is presented. The section3.1 presents the analysis of the data from the questionnaire, fol-lowed by the section 3.2 with the result from the short interviews.

Figure 1: Setup of experiment in the Multisensory Laboratory atKTH

3.1 Quantatative results

Based on the experiments described in method section of this re-port, one quantitative analysis was performed using the IBM SPSSStatistics 23 software package.

A Mann-Whitney U test was run to determine if there were differ-ences in understandability between the new version and the old ver-sion of Norwegians initial step in their booking system. The data isillustrated in Figure 2, which show the frequency along the ordinalscale. Distributions of the understandability for both versions werenot similar, as assessed by visual inspection. The understandabilityof the new version (mean rank = 33.39) and old version (mean rank= 39.61) were not statistically significantly different, U = 760, z =1.552, p =.121, using an exact sampling distribution for U [Dinneenand Blakesley 1973]. The same procedure was run for each subpartof the booking system (choosing destination, date, and number oftravelers) but didnt show any statistically significant result.

Figure 2: Comparison of mean values between the both versions

Even though the tests didnt show any statistically significant differ-ences between the two versions, the data still shows both versionsto be very understandable, with the new version tending to be moreunderstandable which is illustrated in Figure 3. However the stan-dard deviation of the new version was 0.76 and for the old version1.20, which makes it difficult to make any assumptions based onthese results.

3.2 Qualitative results

From the qualitative part of the study both specific and general opin-ions regarding both interfaces was required. In this sections the re-sults from the discussion part of the experiments will be presented.

Figure 3: Frequency of the understandability of initial step in Nor-wegian booking system

3.2.1 Old interface

Regarding the old interface, positive feedback on the possible waysof specifying airports for the task was provided from three partic-ipants (P01, P07 and P11). The feedback specifically said that ”Ifound it comfortable to specify destinations.” (P11) and ”It was easyto type to choose Birmingham as a destinations.” (P07). The feed-back also stated that by being given the airport codes (IATA airportcode) it was even easier to choose destinations for the task (P11).One participant found it to be useful to be able to choose destina-tions from a dropdown menu, rather than typing the airport (P01).Hence, both methods of choosing destinations (typing and choos-ing from the dropdown menu) was appreciated. General feedbackon the old interface included comments like ”It was hard to makemistakes.” (P02) and ”The system was self-explaining.” (P11). Fur-thermore it was commented that the alternatives in the booking pro-cess was clear (P05) and that it was easy to orient to the bookingfield of the webpage in order to complete the task to book a flight(P06).

The negative criticism regarding the old interface was mostly spe-cific to some parts of the interface. Four out of six participants eval-uating the old interface had negative opinions regarding the specifi-cation of one-way flights in the task. This confusion could be tracedto two factors: firstly it was unclear to one participant how to spec-ify a one-way ticket (P07), and secondly three participants foundthe lack of feedback from the interface when specifying a one-wayflight to be confusing (P01, P02, P06). These three users all men-tioned that when specifying a one-way ticket, the option for choos-ing a return date remained available in the booking, even though thetrip was one-way only. One participant summarized his/hers opin-ion as follows: ”When I choose ’One-way’, the date for choosing areturn date should be removed.” (P01). Furthermore, one user men-tioned that when specifying Barcelona as a destination by typingthe airport code (BCM), more than one option was given which wasconfusing (P07). One user would have wanted an option to choosean area of dates, rather than one specific date for the flight, in orderto get a better overview of available flights (P05). Lastly, one userfound the calendar hard to use, as well as the day/month/year com-bination being confusing (P11), and one user also mentioned thatthe website was overall ugly (P06).

3.2.2 New interface

Positive feedback on the new interface could, just as for the oldinterface, be put into two categories: airport specifications and gen-eral. Four out of six users (P03, P04, P09, P12) had some sort

of positive feedback regarding the specification of airports for thetask. Comments were of the sort as follows: ”Good search func-tion.” (P03), ”Everything was easy. Especially specifying where totravel from, and where to.” (P09) and ”It was easy to specify des-tinations, especially when searching for the airport codes.” (P12).More general positive feedback regarding the new interface saidthat the ”clear and clean” and therefore easy to use (P10), and thatthe interface was intuitive (P08).

There were some negative feedback regarding the airport specifica-tion in the new interface as well. One user did not understand thepurpose of the line with pre-determined airports in UK given by thesystem (P09). The same user was confused when finding severalairports with the same name, which lead to confusion whether theparticipant had finished the task or not. Two participants were con-fused when specifying the children and infants for the flight of thetask. The first participant thought that he/she was supposed to typethe age of the children, when the interface actually asked the userto type the amount of children (P03). The participant thought thatthis misunderstanding was due to earlier experiences with differentsystems. Another found that one were forced to first add a childrenbefore defining the number of infants for the flight (P04). Further-more, the user suggested that two buttons would have been more in-tuitive: one for children and one for infants. Lastly, one participantwould have wanted information if the specified flight included anytransfers, or not, and also would have wanted the option to specifydirect-flight only (P12).

4 Recommendations

When typing in the name of one of the airports one of our userstyped the name incorrectly, resulting in the text box not showingany suggestions for completing the typed word. The user did notnotice that they had typed in the name incorrectly, which made theuser stop and think, in confusion, as to why the airport they werelooking for wasnt available in the drop down box. One way to in-terpret why this error happened would be to assume that the userslipped while typing the name of the airport. According to DonNorman, slips are common errors who just sort of happen whenhumans interact with things [Norman 2013]. Since the interfaceprobably will be used by human beings, it is safe to assume thatsome users will be slipping while interacting with the interface. Toreduce the impact of these errors, thus smoothening the interactionwith the interface, we recommend Norwegian to provide some kindof way for users to detect slips like misspellings. An example ofthis could be the interface running a kind of spell checking on whatis being typed into the text fields, providing a red underline if thetyped string cant complete into any existing destinations. Further itcould check the typed string against similar strings, providing sug-gestions for strings that the user might have wanted to put into thetext field.

There are text shortcuts beneath the text box handling the input ofthe first search term that can be used to quickly type in the namethey signify. These hyperlinks were never used by any of our testsubjects during the evaluation sessions, even though one of themcould be used for one of the search phrases they were tasked to putin. This could be an indication that, for users, this feature is hard todetect. If the feature could be even more clearly grouped togetherwith its textbox, perhaps by encasing the pair inside a boundingsquare, or made to be a bit more attention grabbing, perhaps theelements discoverability could be improved.

At least two of our users erroneously put in the age of the child theyhad in the group they were booking the flight for, in the field wherethe number of children would be written. Perhaps it is justified toprovide a visual representation of the company alongside the num-

bers in the boxes defining the size of the company. By providingbasic representation of human beings to the side of the elementswhich define the size of the company it could be made clearer tothe user the size of the company they currently have defined.

Weve also noticed that the old version of the webpage had our usersconfused about the date defining element. When they ticked the boxindicating that they wished to book a one way ticket, they expectedthe part of the UI that lets you choose the date for your return ticketto be hidden. This has been fixed in the new version of the web-page, where having the one-way flight ticket option ticked causesthe return flight element to be hidden.

5 Discussion

We cant seem to notice any significant difference between Norwe-gians new and old web interfaces in this evaluation. Both groupsdeemed the websites they had used as very easy to understand,which seems to indicate that the interfaces both display good un-derstandability in relation to the group that have used them. Whilethe test subjects indicated that they easily understood how to usethe interfaces, the test sessions didnt flow as freely as they couldhave had, as the users encountered problems while conducting theirtask.

The questionnaires used to gather the quantitative data used in thisstudy contained few, relatively generalized questions. If a largernumber of more detailed questions were asked, perhaps we wouldbe able to see a notable difference in the two interfaces. If a futurestudy would evaluate the two systems we have studied using similartest groups, using more questions could generate a different resultthan the one produced in this study.

When we conducted the evaluation we did not record the screen ofthe user sessions and chose to take notes instead. This resulted indata being lost, as we were unable to provide the level of detail inour notes that a recording captures. Having lost this informationto base our evaluation on can have impacted the evaluation in anegative way, and there could be problems with the system that havenot been discovered in this evaluation. However, during the studysession the observer was able to observe what happened during thestudy, but he couldnt see everything that happened precisely on thescreen, therefore the screen recording might come in hand in furtherstudies of this project.

The group who tested the new webpage reported having less expe-rience with online ticket booking than the group who tested the oldwebpage. Despite this the group with the least experience judgedthe web solution they tested to be slightly more understandable,even if it isnt a statistically significant difference. If the group us-ing the old version had had as little previous experience with onlineticket booking as the group using the new version had, perhaps theywould have judged the system as less understandable than they didin this study.

The quantitative information provided the users with a seven pointscale that they were asked to use when judging the system. Thisscale has been represented numerically as a scale between 1 and 7,where each step is one unit larger than the last. This representationof possible levels of understandability is not complete, and as suchthere might be users who find that they cannot express their experi-ence with the system. It is, for example, impossible to rate part ofthe system 1,5 with the scale used in this study. However, we hopethat the seven point scale has been enough for our users to providea somewhat accurate estimation of their experience.

6 Conclusion

Norwegians new web interface used to search for flights, as wellas the old one it is replacing, are both very understandable by theKTH students who have partaken in this study. While we cannotsee any statistical difference in the quantitative data produced inthis study, observations of use sessions has provided some valuableinformation. We recommend Norwegian to provide ways for usersto even more easily diagnose misspelled inputs, making their airportquick suggestions more discoverable, and give further feedback tothe user when they put in the size of the company they want to booktheir flight for.

References

ALZUGHOUL, M. A. H. 2010. Usability Evaluation of Web-basedOnline Airline Booking System. PhD thesis, Universiti UtaraMalaysia.

CHEN, W., PAUL, A., KIBARU, F., MA, Y., AND SAPAROVA, D.2015. Two-phase usability evaluation of insurance website pro-totypes. International Journal of E-Business Research (IJEBR)11, 1, 1–22.

DINNEEN, L., AND BLAKESLEY, B. 1973. Algorithm as 62: Agenerator for the sampling distribution of the mann-whitney ustatistic. Applied Statistics, 269–273.

FLAVIAN, C., GUINALIU, M., AND GURREA, R. 2006. The roleplayed by perceived usability, satisfaction and consumer trust onwebsite loyalty. Information & Management 43, 1, 1–14.

MATZLER, K., WAIGUNY, M., TOSCHKOV, A., AND MOORA-DIAN, T. A. 2006. Usability, emotions and customer satisfac-tion in online travel booking. Information and CommunicationTechnologies in Tourism 2006, 135–146.

NORMAN, D. A. 2013. The design of everyday things: Revisedand expanded edition. Basic books.

A Image of new version of Norwegian Book-ing System

B Image of old version of Norwegian Book-ing System

C Questionnaire

Questions about user experience (First page)

1. How easy, or difficult, do you think it was to understandwhat the system requires you to do to find a flight fromone airport to another? Scale from 1: Very easy, to 10: Verydifficult

2. How easy, or difficult, do you think it was to understandwhat the system requires you to do to specify the date of aflight? Scale from 1: Very easy, to 10: Very difficult

3. How easy, or difficult, did you think it was to understandwhat the system requires you to do to specify the numberof tickets you were tasked to book? Scale from 1: Very easy,to 10: Very difficult

Interview qualitative questions (Second page)

1. Could you describe what would you think were easy to un-derstand? Why?

2. Could you describe what parts were hard to understand?Why?

3. Could you describe your experience; which parts wereconfusing? Why? Why would you think so?

Questions about you (Third page):

1. Age Number input.

2. Gender Man/Woman/Other/Do not want to disclose

3. How experienced would you describe yourself using onlinebooking systems like the one you used in this test? Scalefrom 1: Not experienced at all, to: Very experienced

D Information during the evaluation session

• We will ask you to use an airline online ticket booking system.Using this system, we want you to book a flight from Birm-ingham (BHX) to Stockholm-Arlanda (ARN) for a group ofpeople. Once the task is complete, you will be asked to fillout a short questionnaire, and asked some questions in a mini-interview.

• The session will last about 10 minutes.

• We will be recording audio when asking the questions, whichwill be used as material for writing the report. We will notpublish this audio, and it will be erased once we have finishedthe report.

• Were testing the ticket booking system, and not you. Youcan not do anything wrong, and you do not have to feel anypressure.

• While you are doing your tasks, you are allowed to think outaloud. Do not worry about telling us (or not telling us) whatyou are thinking.

• Do not worry about giving us critique. We want to hear whatyou think, so we can improve our ability to conduct researchlike this.

• If you have any questions during the test, ask them. We mightnot be able to answer them, as that could affect the test, butwe will answer all questions you have after the test.

• If you need a break, just ask.

• Do you have any questions this far?

• We would love to have your consent to publish the resultsfrom this test in our report. You will, of course, remain anony-mous in our report.