Teknik och samhälleDatavetenskap

Examensarbete15 högskolepoäng, grundnivå

Gamification of a Physics Simulation Tool

Gamification av ett verktyg för fysiksimulering

Alexander D. BaldwinSimon J.O. Dahlberg

Examen: Kandidatexamen 180 hpHuvudområde: DatavetenskapProgram: Spelutveckling

Handledare: Olle LindebergAndrabedömare: Bengt J. Nilsson

Abstract

Gamification – the use of game elements in non-game contexts – has been shown to be aneffective way of creating more enjoyable and engaging user experiences. Many applicationsof gamification rely on a limited subset of game elements such as points, badges andleaderboards – techniques which have been criticised for not being representative of whatmakes games enjoyable. Research in psychology suggests that people are most effectivelymotivated by factors that appeal to their personal satisfaction, such as fun, rather thanexternal factors such as rewards and prizes. This thesis explores the use of game designelements and related 3D technologies in the creation of a tool for the construction andsimulation of physical models. The implementation of such a tool is described, focussingon the integration of game design elements that produce a fun and engaging experiencefor the user. A study is performed to determine the effects of the use of game elements onusers’ enjoyment when interacting with the simulation tool. It was found that by appealingto themes users can relate to and which are contextually appropriate for the application,game elements can help promote an enjoyable user experience.

Sammanfattning

Gamification – bruket av spelelement i icke-spelsammanhang – har visat sig vara ett ef-fektivt sätt att skapa mer underhållande och engagerande användarupplevelser. Mångatillämpningar av gamification förlitar sig på en begränsad delmängd av spelelement sompoäng, emblem och resultatlistor – tekniker som kritiserats för att inte vara representativaför vad som gör spel underhållande. Forskning inom psykologin indikerar att personerblir mest effektivt motiverade av faktorer som vädjar till deras personliga tillfredsställelse,såsom nöje, snarare än externa faktorer som belöning och priser. Detta arbete utforskarbruket av speldesignelement och relaterade 3D-teknologier i utvecklingen av ett verktygavsett för att skapa och simulera fysikaliska modeller. Implementationen av ett sådantverktyg beskrivs med fokus på integrationen av speldesignelement vilka bidrar till en un-derhållande och engagerande upplevelse för användaren. En studie utförs för att valideraeffekten av bruket av spelelement på användarnas nöje när dessa interagerar med sim-ulationsverktyget. Det observerades att genom att använda teman som användaren kanrelatera till och som är kontextuellt passande för applikationen i fråga kan spelelementförstärka en njutbar användarupplevelse.

Acknowledgements

We would like to thank Hilding Elmqvist for his support and guidance, as well as everyonefrom Dassault Systèmes who worked with us to make this thesis possible.

Contents

1 Introduction 11.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Related Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.2.1 Game Elements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.2.2 Intrinsic and Extrinsic Motivation . . . . . . . . . . . . . . . . . . . 41.2.3 Gamification Frameworks . . . . . . . . . . . . . . . . . . . . . . . . 41.2.4 Gamification in Practice . . . . . . . . . . . . . . . . . . . . . . . . . 61.2.5 Applications of Game Technology . . . . . . . . . . . . . . . . . . . . 7

1.3 Research Question . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81.3.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81.3.2 Aims . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81.3.3 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2 Method 82.1 Prototype Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.2 User Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.2.1 Questionnaire . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92.2.2 Observation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3 Implementation 113.1 Functionality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113.2 Use of Game Elements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

3.2.1 Fantasy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143.2.2 Curiosity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163.2.3 Challenge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

3.3 Use of 3D Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

4 User Study 18

5 Results of the User Study 195.1 Ranked Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195.2 Other Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225.3 Observation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

6 Analysis 246.1 Fantasy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

6.1.1 Theme/Story . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256.1.2 Metaphors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256.1.3 Audio/Visual Effects . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

6.2 Curiosity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

6.2.1 Decoration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266.2.2 Representation System/Feedback . . . . . . . . . . . . . . . . . . . . 27

6.3 Challenge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276.4 Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276.5 User Experience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

7 Discussion 287.1 Suggestions for Further Research . . . . . . . . . . . . . . . . . . . . . . . . 30

References 31

1 Introduction

1.1 Background

Increases in the computational power of modern computers have led to advanced scientificsimulation/visualisation tools being accessible to the general public, but science educationhas been slow to adopt the use of interactive simulations. Wieman and Perkins [36] arevocal proponents of using interactive simulations as an educational tool in science andargue that traditional forms of education fail to provide students with an understanding ofscience by suppressing their interest in the subject and failing to engage them. Wieman andPerkins argue that interactive simulations are an effective complement to traditional mediain science education – an idea that is supported by research into the use of simulation toolsto engage college students [24]. According to Wieman and Perkins, the most importantfeatures of an engaging interactive simulation tool are:

• Highly interactive animation

• An appealing environment and sophisticated graphics

• Simple and intuitive controls

• Connections to real-life objects [36, p. 291]

Dymola is an environment for modelling and simulation of complex physical systemsfrom diverse engineering fields [20]. Through a JavaScript interface Dymola allows the userto specify and simulate models in a browser-based application, allowing new functionalityto be provided or existing functionality to be hidden, thereby facilitating the creation ofnew tools. While Dymola is a complex application targeted at professional users withexperience of programming in the Modelica language [19], on which its models are based,the JavaScript interface allows tools targeted at less experienced users, such as students,to take advantage of the simulation capabilities of Dymola in a simpler environment.

Research shows that an effective way of promoting user-engagement and interest inlearning environments is the incorporation of elements from digital games [30]. The use ofgame-elements in other contexts, usually referred to as gamification, has been an increas-ingly popular research topic in recent years, with many studies applying its principles inan educational context [12, 31]. The stated motivation for the use of (digital) gamificationis often to increase user engagement and, consequently, user retention in software systems[9].

Few attempts have been made to produce a formal definition of gamification and mostof the available research on gamification appears to be based on Deterding et al.’s [9]definition:

‘Gamification’ is the use of game design elements in non-game contexts.

Huotari and Hamari [14] attempt to define gamification from a marketing perspective– a tool to facilitate value creation. Their view of gamification deals with applicationsinvoking the same kind of psychological and behavioural responses that games do, whereasDeterding et al.’s focus is on the use of elements from games in an application rather

1

than the outcome [12]. Deterding et al. criticise Huotari and Hamari’s definition for be-ing too narrow and underplaying social and experiential aspects of games. Nonetheless,the use of gamification as a marketing tool has become popular, with proponents suchas Gabe Zichermann enthusiastically promoting the idea of using gamification for mar-keting [40]. Zichermann and Linder’s book “Game-based Marketing. Inspire CustomerLoyalty Through Rewards, Challenges and Contests” [42] describes means by which gamemechanics can be used for marketing, with a focus on points, leaderboards and rewards.

The limited array of game elements used in applied gamification research has drawncriticism, particularly from the games industry and the field of game studies [31] and,consequently, gamification as a whole is sometimes viewed in a negative light. Gamedesigner and critic Ian Bogost calls Zichermann “the gamification movement’s Dark Lord”,criticising what Zichermann calls “key game mechanics” as not representative of whatmakes games fun and describing gamification as “exploitationware” – taking advantage ofcustomers by replacing real incentives for loyalty with fictional ones [6]. Game designerMargaret Robertson takes a similar perspective to Bogost in stating about the prevalenceof points and badges in gamification: “What we’re currently terming gamification is infact the process of taking the thing that is least essential to games and representing itas the core of the experience.” [27]. Robertson feels that the term “pointsification”would be more appropriate when describing typical applications of gamification. Seabornand Fels [31] contend that Bogost’s view of gamification is too focussed on marketingapplications despite the fact that gamification has been used in numerous other areas andpoint out that gamification is generally viewed more positively outside of the fields of gamestudies and game design.

In addition to design-elements from games, game-related technologies such as gameengines and 3D-engines can be useful in non-game contexts and have successfully beenused in the development of simulation tools on numerous occasions [4, 13, 25]. Gameengines often boast advanced graphical features with realistic lighting and shadows andsupport for importing complex animated and textured 3D models, which can be usedto provide the appealing environment and sophisticated graphics Wieman and Perkinsconsider important. Depending on the type of simulation tool to be created, other commonfeatures of game engines such as built-in physics engines and artificial intelligence systemscan also be useful. These types of features are often provided in a development environmentwhich supports rapid production, commonly with little need for programming.

1.2 Related Research

This section examines previous research on gamification and the application of game-related technology to simulation tools.

1.2.1 Game Elements

The majority of available research on gamification uses either Deterding et al.’s definition,no explicit definition or a hybrid of multiple definitions [31, pp. 23-24]. When usingDeterding et al.’s definition as a starting point, it is important to consider what a “gamedesign element” is. Huotari and Hamari’s definition sidesteps this issue by putting the

2

focus on the added value arising from the use of “feedback and interaction mechanisms”[14] rather than the specifics of what those mechanisms actually are. Regardless of whichdefinition is used, the designer of a gamified system/service must have some idea of whata game element is and how it can be used.

In an attempt to define game elements, Deterding et al. note that it is not useful todefine elements which must be present for a system to be considered gamified, since thereexists no such set of elements that accurately represents all styles of games. They preferinstead to talk about elements that are characteristic of games:

. . . elements that are found in most (but not necessarily all) games, readilyassociated with games, and found to play a significant role in gameplay. [9]

The authors divide game design elements into five levels of abstraction, shown withexamples in Table 1.

Level Description ExampleGame interface design pat-terns

Common, successful inter-action design componentsand design solutions for aknown problem in a con-text, including prototypi-cal implementations

Badge, leaderboard, level

Game design patterns andmechanics

Commonly reoccurringparts of the design ofa game that concerngameplay

Time constraint, limitedresources, turns

Game design principles andheuristics

Evaluative guidelines toapproach a design problemor analyze a given designsolution

Enduring play, clear goals,variety of game styles

Game models Conceptual models of thecomponents of games orgame experience

MDA (Mechanics Dynam-ics Aesthetics); challenge,fantasy, curiosity; game de-sign atoms; CEGE (CoreElements of the GamingExperience)

Game design methods Game design-specific prac-tices and processes

Playtesting, playcentric de-sign, value conscious gamedesign

Table 1: “Levels of Game Design Elements” from Deterding et al. [9, p. 12]

In addition, Deterding et al. draw a distinction between game design elements andgame-related technologies, such as the use of graphics/game engines or input devices likegame controllers – classing the former as a part of gamification but not the latter, as theyare widely used outside of game-related contexts [9].

3

In two literature reviews investigating the results of empirical studies on the effectsof gamification, both Seaborn and Fels [31] and Hamari et al. [12] report that points,badges and leaderboards were the most common forms of game-element (or motivationalaffordance, in Hamari et al.’s terminology) studied. Seaborn and Fels recommend the studyof more diverse game-elements as an avenue for future research, citing the early state ofgamification research as a potential reason for this repeated focus on a limited number ofelements.

1.2.2 Intrinsic and Extrinsic Motivation

A number of attempts have been made to construct models or frameworks for the applica-tion of gamification, mostly taking inspiration from Ryan and Deci’s research on intrinsicand extrinsic motivation [7] or the related concept of Self Determination Theory, whichdeals with the behaviour of people who are motivated by purely internal factors [29]. In-trinsic motivation refers to the motivation to perform an activity because it is inherentlysatisfying, rather than because of some separate consequence or reward [28]. Ryan andDeci describe instrinsic motivation as follows:

Intrinsic motivation energizes and sustains activities through the spontaneoussatisfactions inherent in effective volitional action. It is manifest in behaviorssuch as play, exploration, and challenge seeking that people often do for noexternal rewards. [7, p. 658]

Extrinsic motivation refers instead to activities performed with the specific intention ofachieving an external outcome, such as a reward or praise. According to Self DeterminationTheory, extrinsic motivation can vary in the degree to which it is autonomous (coming froma person, rather than external sources), since a person may perform actions with perceivedpersonal benefits (rather than purely for the sake of personal satisfaction) without externalinfluences [28].

Ryan and Deci’s review of papers studying the effect of extrinsic rewards on intrinsicmotivation concludes that, generally, “tangible rewards had a significant negative effecton intrinsic motivation for interesting tasks” [7] and a later article on Self DeterminationTheory states that people who are self-motivated rather than externally controlled have“more interest, excitement, and confidence, which in turn is manifest both as enhancedperformance, persistence, and creativity” [29, pp. 658-659]. This negative impact ofextrinsic motivators suggests that it might be more successful to employ game elementsthat appeal to a user’s intrinsic motivation rather than the commonly used points andbadges, which imitate tangible rewards.

1.2.3 Gamification Frameworks

In his design philosophy for gamification [39], Gabe Zichermann recognises the potentiallynegative effects of extrinsic motivation, but instead of focussing on intrinsic motivationdirectly, he uses the Self-Determination Theory idea that extrinsic motivators can be con-verted to intrinsic motivators if they are “meaningful, pleasurable and consistent with aperson’s worldview”, stating:

4

The introduction of carefully selected extrinsic rewards, built around a designthat speaks to intrinsic motivational states (sometimes not the ones most closelyaligned with the behavior we seek to change), is the most powerful design modelwe have today.

Consequently, the two books Zichermann has co-authored on the subject of designstrategies for gamification [42, 41] have a focus on extrinsically-motivating game elementssuch as points, leaderboards and badges, with an emphasis on the social aspects of gamifiedsystems where users compete for status (exactly the kinds of gamification that people likeBogost [6] and Robertson [27] criticise).

Nicholson [21] attempts to solve the issue of the negative effects of extrinsic motivatorson intrinsic motivation by conceptualising a framework for “meaningful gamification” witha focus on the end-user rather than the organization providing the service. He also provideshis own definition of meaningful gamification: “the integration of user-centered game designelements into non-game contexts”. Besides recommending a focus on intrinsic motivators,Nicholson’s framework stresses the importance of context in the use of game elements,referring to the concept of situated motivational affordances [8], which describes how themotivational effect of a system element depends upon the background of the user and thecontext in which it is used in the system. Nicholson concludes that meaningful gamificationrelies on using elements that users with a wide variety of backgrounds can relate to in theright context within a system. In his examples of applications of meaningful gamification,Nicholson suggests removing scoring/rule-based elements and focussing on play, referringto this as “playification”. Deterding et al. refer to this concept as “ludification” and considergamification a subset of ludification [9, p. 13].

Aparicio et al.’s [1] methodology for gamification is based more closely on Self Determi-nation Theory, basing the choice of game elements on those matching the central sourcesof intrinsic motivation: autonomy (the sense of personal choice), competence (the needfor challenge) and relatedness (the need for social interaction). Game elements listed asexamples in each of these categories are:

• Autonomy: profiles, avatars, macros, configurable interface, alternative activities,privacy control, notification control.

• Competence: positive feedback, optimal challenge, progressive information, intuitivecontrols, points, levels, leaderboards.

• Relation: groups, messages, blogs, connection to social networks, chat.

Numerous studies on gamification refer to Malone’s paper [18] on using ideas fromgames in effective user interface design. While written long before the coining of theterm gamification, Malone’s ideas still have a lot in common with the aforementioneddefinitions, particularly Nicholson’s framework for user-centered meaningful gamification.Malone focuses on the user’s enjoyment, defining three principle heuristics for designingenjoyable user interfaces: challenge – the activity should have a clear goal and an uncertainoutcome, fantasy – the interface should be “emotionally appealing” and use metaphorsthat the user can relate to, and curiosity – the interface should provide the right level of

5

informational complexity in order to be novel or surprising while still understandable; itmight also use “sensory curiosity”, which refers to the use of audio and visual effects asdecorations, to enhance fantasy or as a means of representing aspects of a system.

While some of the papers reviewed by Seaborn and Fels cite Cunningham and Zicher-mann [41] or Zichermann and Linder [42] as bases for their application of gamification [31,pp. 23-24] – likely due to their focus on the concrete marketing benefits of the style ofgamification they describe – none cited the approaches of Nicholson, Aparicio et al. orother theoretical frameworks, suggesting that more research is needed in order to evaluatewhether or not these models are effective.

1.2.4 Gamification in Practice

As discovered by Hamari et al. [12] and Seaborn and Fels’ [31] literature reviews, theprevalent form of gamification in academic research today uses points, badges and leader-boards as incentives to increase enagagement and create competition in social contexts,but a number of studies also use elements of fantasy (story/theme), challenge (goals) andcuriosity (audio/visual feedback) either in conjunction with the other elements or insteadof them.

Liu et al. [17] integrate thematic elements into their study of the effects of gamificationon an application designed to decrease energy usage in the home. They argue that theuser is no longer just a “cog in an organizational machine”, but rather a partner and thatthe focus today is in user engagement in order to produce a lasting relationship between asystem and its users. As such, their design focus is on motivating users and their study isunderpinned by Ryan and Deci’s work on intrinsic motivation. The authors conclude thata topic or theme must be of interest to the user for gamification to be successful, whichis in line with Deterding [8] and Nicholson’s [21] ideas about the importance of contextwhen choosing how to implement game elements. Liu et al. also believe that (whileimportant) elements like badges and leaderboards are insufficient to create a successfulgamified experience – what is important is to make the application feel like a game, withan emphasis on fun and aesthetic qualities.

Another article which supports the importance of context in the application of gami-fication is Downes-Le Guin et al.’s study of the effects of gamifying a questionnaire [10].Elements of fantasy were introduced in the form of a storyline, attractive graphics and astylised avatar representing the respondent. While the gamified questionnaire was generallyconsidered more fun than non-gamified versions, it was not shown to increase engagementand had a significantly lower response rate than non-gamified versions. A questionnaire wasprobably not the right context for this type of gamification (or possibly any gamificationat all).

Li et al.’s [16] implementation and evaluation of a gamified tutorial system for a CADpackage is interesting in its use of a consistent theme and background story throughout,as well as its use of audio and visual feedback. The user is given the role of constructingparts for the Apollo space program – a theme which is supported by the use of attractivegraphics and progressive-disclosure to reveal the story gradually. Their description of thesystem highlights the desire to make it feel like an actual game, with elements described

6

as “arcade style” and compared to puzzle games. Li et al’s article is also one of the fewstudied for this thesis that references an actual game as a source of inspiration: Tetris [23].The authors’ choice of game elements is heavily informed by Malone’s [18] strategy foreffective user interface design.

Finally, Berengueres et al.’s study of a gamified recyling bin [2] is interesting because,unlike the majority of gamification studies, it did not involve the use of points, badges,leaderboards or any similar elements, relying instead on audio and visual feedback alone.The study reports a threefold increase in the usage of the gamified recycling system, sug-gesting that it is possible for gamification strategies not centered around points systemsto be successful.

1.2.5 Applications of Game Technology

While Deterding et al. [9] exclude applications of game-related technologies from theirdefinition of gamification, game-related technology has application in the development ofnon-game applications and has been used to enhance simulations in numerous ways. It isoften used as an easy way of enhancing the graphical fidelity of a system and, because ofthe rapidly evolving nature of the game development industry, game engines are commonlykept up to date with advances in graphics technology [4].

Bijl and Boer present a survey of the usefulness of various game engine features insimulations from the perspectives of validation, analysis and marketing [4], which suggeststhat the marketing perspective benefits most from the use of features from game engines.The survey also indicates that, regardless of the perspective, textures, lighting and shadowsare the most useful features. Features such as file converters and exporters as well as thevarious configuration possibilities of game engines are also presented as potentially usefulfeatures but were not included in the survey.

Bijl and Boer also describe a tool for simulating a shipping-container terminal (to beused for training purposes) which utilises the 3D-engine Ogre3D’s [22] graphical capabilitiesin order to present a realistic depiction of the system, increasing its usability by applyingthe engine’s support for camera and character control as well as for general performanceimprovements. The engine’s 3D animation features are also cited as beneficial in trainingscenarios using the simulation tool [4].

Several other examples of the use of game technology in a simulation context exist, butlack the detailed evaluation of specific features described above. Prasithsangaree et al. [25]use the game engine Unreal Engine [35] to develop a front end for running military simula-tions, utilising the engine’s networking capabilities to allow for distributed simulations aswell as the engine’s visualization of skeletal dynamics, Newtonian physics and explosions.Hu et al. [13] use the game engine Unity’s [34] physics engine as the basis for running anddisplaying mechanical simulations as well as saving time on implementation by reusingpredefined components of the engine.

7

1.3 Research Question

1.3.1 Background

The bulk of research on gamification suggests that gamification is an effective tool forincreasing user engagement and enjoyment of software systems [12]. However, researchhas been focussed on the use of a limited subset of game elements (sometimes referred toas pointsification [27]), which encourages users to compete for points or complete tasksfor rewards despite the psychological foundations of motivation research suggesting thatintrinsic motivators are more effective [7]. On top of this, gamification has frequently beenregarded as a layer on top of an already functional application, rather than a core part ofit (e.g. Blohm and Leimeister’s gamified service bundles [5]). Additionally, game-relatedtechnologies have been shown to be useful when constructing simulation software [4].

1.3.2 Aims

The thesis has two primary aims:

1. To investigate whether the use of game elements is effective in creating an enjoyableuser experience in a tool for simulating the behaviour of mechanical systems.

2. To investigate whether a 3D environment using game-related technologies can aid inthe understanding and ease of use of such a simulation tool.

1.3.3 Limitations

While one of the potential uses of simulation tools is in science education, this thesis doesnot consider any specific educational benefits of the application of gamification and/orgame-related technologies.

2 Method

This section describes the method used in the investigation of the aims stated in Sec-tion 1.3.2.

2.1 Prototype Development

A software prototype, Playmola, in the form of a browser-based application for simulationand visualisation of the behaviour of mechanical systems was developed. This prototypeserves two main purposes: firstly to demonstrate a design strategy for gamification ofa simulation tool based upon intrinsically motivating factors and the ways that gametechnology can support this and secondly to evaluate the benefits of using game elementsin such an environment.

8

2.2 User Study

Based upon the prototype, a small-scale user study with eight participants was carried outin order to gather user experience data. Tullis and Albert [33] describe the three maincharacteristics of user experience as follows:

• A user is involved

• That user is interacting with a product, system, or really anything with an interface

• The users’ experience is of interest, and observable or measurable [33, p. 4]

Tullis and Albert also describe a possible differentiation between usability and user expe-rience, where usability is considered to be the users’ ability to use a system to carry outa given task successfully and user experience is considered to be the broader view of theusers’ interaction with the system, including thoughts, feelings and perceptions stemmingfrom that interaction.

Since the aim of the study was to gather user experience data rather than to gaugeeducational or productivity benefits, results are largely in the form of self-reported datafrom the users relating to their experiences and perceptions when using the application.Tullis and Albert describe the importance of self-reported data in usability studies asfollows:

Self-reported data give you the most important information about users’ per-ception of the system and their interaction with it. At an emotional level, thedata may tell you something about how the users feel about the system. [33,p. 123]

As such, the format of the test (further described in Section 4) consisted of a numberof tasks to be performed while under observation, followed by a questionnaire. Numerousgeneral usability scales for post-testing questionnaires exist, as described by Tullis andAlbert, but most focus on general usability, productivity and understanding of a system,rather than the aesthetic or emotional responses of interest to this study (responses whichare, of course, also influenced by usability and understanding of the application). Thequestionnaire was inspired by the aforementioned scales, as well as tailor-made metrics usedin previous studies of gamification like Li et al.’s study of a gamified tutorial system [16],which also examines user enjoyment.

2.2.1 Questionnaire

Qualitative data were collected through open-ended questions in a questionnaire adminis-tered after the testing session was completed.

Quantitative data were collected through Likert scale-based rating questions (as de-scribed by Tullis and Albert [33, p. 123]) in the post-testing questionnaire.

9

2.2.2 Observation

In addition to self-reported data, testing sessions were observed and recorded. During eachtesting session one of the test organisers primarily focussed on writing down observationsof the users’ interaction with the system as well as the users’ verbal comments in order togather further qualitative data on user behaviour and perceptions, which were then usedto provide additional context when analysing responses to the questionnaire. Some specificfactors observed were as follows (all based on the observer’s subjective judgement):

1. Apparent enjoyment.

2. Apparent frustration.

3. Difficulty when controlling/navigating the application.

2.3 Discussion

A disadvantage of using a tailor-made format for user experience testing, rather than anestablished scale such as the System Usability Scale (described by Tullis and Albert in [33,p. 137]) is that it is not possible to quantify the usability of the system as a whole andcompare it to known benchmarks. However, since the study does not address on usabilityspecifically and focuses on parts of the system in isolation (specific game elements and 3Dtechniques), rather than the application as a whole, this approach may be more appropriatethan using an existing scale.

Another concern is the size of the study. While there is little consensus on the numberof participants required for a high level of confidence in the results of a usability study,available research suggests that a small number of users is sufficient in most cases [15].Since the aim of this study was not to exhaustively uncover problems in the prototype’susability, but rather to determine general trends in how enjoyable and engaging it is, asmall number of participants is sufficient.

The choice to use questionnaires rather than interviews in collecting qualitative datawas based on a number of factors. Preece et al. [26] recommend using a questionnairerather than an interview when studying specific design features of a system, which wasaccomplished by examining users’ responses to particular game elements, rather than thesystem as a whole. The use of well-designed open-ended questions allows a similar depth ofdata to be gathered as when using interviews, when combined with the results of observa-tions. Using a questionnaire also allows the test participants to be left alone (or supervisedfrom a distance, so assistance can be offered if necessary) while filling out the answers, inthe hope of encouraging honesty.

The idea of measuring intangible concepts such as fun is controversial since said con-cepts are highly subjective and not easily quantifiable. However, considering the benefitsof intrinsically motivating factors, like fun, shown in research on gamification, measuringfun is highly relevant. Wixon [38] argues that if subjective experiences are to be consid-ered unmeasurable then there is no point in developing software designed to be enjoyable.Within the field of interaction design, Preece et al. [26] include fun in their definition of

10

user experience and consider it worth measuring not only by itself but also in relation tousability.

In their list of current methodological limitations in academic studies of gamifica-tion [12, p. 3029], Hamari et al. criticise the lack of non-gamified control groups andthe application of multiple game elements simultaneously for making it impossible to de-termine the exact impact of gamification (or specific game elements) on a system. Seabornand Fels [31, p. 29] also recommend that future research on the subject of gamificationshould attempt to isolate its effect, preferably with comparative studies. The simulationtool developed in this work does not however lend itself easily to a comparative study, sincegamification is applied as a core aspect of its functionality, unlike many prior studies whichfocus on the application of gamification as as a layer on top of an already functional system(see, for example, Blohm and Leimeister’s concept of “gamified service bundles” [5]).

3 Implementation

In order to test whether game elements chosen to be intrinsically motivating along with adesign strategy with gamification and the use of game technology at its core rather thanas a separate layer of functionality can be effective in creating engaging simulation tools,a prototype named Playmola was produced.

Playmola is a tool for constructing physical (specifically mechanical) models, simu-lating their behaviour and playing back the animated results directly in a web browser.Simulations are performed by the modelling and simulation tool Dymola [20], through aJavaScript interface.

Playmola’s target user is a university undergraduate with an interest in physics andsome experience of university level education in physics or applied mathematics.

3.1 Functionality

Dymola is an environment for modelling and simulation of complex physical systems fromdiverse engineering fields, which are described using the Modelica language [19]. Modelsin Dymola are constructed in a traditional 2D view where components are representedby icons connected by lines, or written directly in Modelica. When run, simulations canoutput a 3D animation, showing the behaviour of the model (see Figure 1).

Playmola is based on Dymola, but limits functionality to a small subset of mechan-ical components and provides a simplified 3D interface with the combined functions ofconstructing, simulating and animating models.

A model in Playmola consists of a number of components that can be selected froma menu (see Figure 2) and dragged into the scene. Each component has a number ofconnectors – points at which it can be connected to another component.

Each component in a model has a number of associated parameters that can be setusing an input form as shown in Figure 3. Changing the component parameters is oftendirectly reflected in the 3D environment – for example, changing the length parameterfor a box will make it longer. The components used in Playmola are simplified versionsof components from the Modelica standard component library, dramatically reducing the

11

Figure 1: Two screenshots of Dymola, the left showing a model represented by iconsconnected with lines and the right showing the animated output of a simulation.

Figure 2: Screenshot of Playmola, show-ing the menu of available components.

Figure 3: Screenshot of Playmola, show-ing the input form used to change a com-ponent’s parameters.

number of parameters the user can change to a subset of key parameters that allow theuser freedom to experiment without becoming confused by an overload of information.These kinds of constraints are recommended by Podolefsky et al. in their discussion ofthe use of simulation tools in physics education [24]: “constraining what students can doreduces cognitive demands and frees up resources for sense making and development of anexpert-like mental framework.”

While constructing a model, Playmola is in one of two modes: 3D View (see Figure 4)or Exploded View (see Figure 5), each representing a different level of abstraction. 3D Viewis the highest level of abstraction, where the only components visible are those representing

12

three dimensional bodies (boxes, cylinders or complex shapes) and these components aredirectly connected to each other (without intermediate lines or other components). Allother components are hidden. Exploded View shows all components pushed apart in themanner of a technical exploded-view drawing/schematic. Connections between componentsare shown as lines in the same manner as in Dymola (see Figure 1).

Figure 4: Screenshot of Playmola, showing the 3D View for constructing models.

Figure 5: Screenshot of Playmola, showing the Exploded View for constructing models.

Models constructed in Playmola are converted to Modelica code and sent via a JavaScript

13

interface for simulation by an instance of Dymola running on a server. If Dymola reportsthat the simulation was successful, further queries are sent to the server requesting inter-polated values of variables from the simulation (for example, angles and positions) to beused in animating the model. These animations can then be played back in the same viewused for creating the model, as shown in Figure 6.

Figure 6: Screenshot of Playmola, showing a model mid-animation.

3.2 Use of Game Elements

Game elements used in the prototype are separated into categories inspired by Malone’sheuristics for designing enjoyable user interfaces [18].

3.2.1 Fantasy

Malone argues that the concept of fantasy, a system that evokes mental images of makebelieve objects and/or situations, is “probably the most important feature of computergames that can be usefully included in other user interfaces” and recommends using it toappeal to users on an emotional level. Playmola takes advantage of fantasy in a numberof ways:

Theme/StoryThe user is placed in the role of an enthusiastic amateur inventor, tinkering with machines

in a small run-down workshop or garage (see Figure 7). Deterding [8] and Nicholson [21]stress the importance of using game elements that are appropriate for the user’s backgroundand in an appropriate context. Since the main function of the system is to build andrun simulations of mechanical models, a theme based around building/experimenting is

14

contextually appropriate. The setting was chosen to be recognisable to as wide a range ofpeople as possible, but by never explicitly stating what the setting is, the user is allowedto draw his/her own conclusions about it based upon the graphical cues provided. Thegraphical cues in question include a workbench, numerous cardboard boxes, a screwdriver,a radio, dirty concrete flooring, dirty plaster walls and a corrugated metal ceiling.

Figure 7: Screenshot of Playmola, showing the application’s theme: a workshop, as well asthe interactive elements in the scene: a radio for muting/unmuting music and a “big redbutton” for starting a simulation.

In addition to appealing to user’s emotions, the use of the workshop scene has severalbenefits for navigation and construction of models. It provides a clear sense of whichdirection is down in order to prevent disorientation when navigating in the 3D environmentand also provides a reference for the measurements of the objects in the scene (for examplethe measurements of a box placed in the scene can be compared to the measurements ofthe workbench which is always present in the scene). Objects in the scene cast shadowswhich provide a visual indication of their relative locations.

MetaphorsIn addition to using fantasy to induce an emotional response, Malone points out that

using fantasies analogous to real-life objects or concepts (metaphors) can make a systemeasier to use. Wieman and Perkins [36] also highlight the importance of using “connectionsto real life objects” in the creation of engaging simulation tools. In Playmola, a number ofobjects in the scene are interactive and serve as metaphors for functionality: a 3D modelof a radio can be clicked to mute or unmute the background music, a 3D model of a “bigred button” [3] can be clicked to start a simulation (see Figure 7).

15

Audio/Visual effectsMusic, sound effects and visual effects used in Playmola are chosen to fit the theme and

thereby enhance the sense of fantasy. For example, when objects are connected a “welding”sound is played along with a visual effect representing sparks, see Figure 8. While this maynot be physically realistic for the kind of connection being made, the intention is to evokean atmosphere consistent with the workshop theme. Music played while a simulation isbeing run uses metallic “clanking” noises to contribute to the atmosphere.

Figure 8: Screenshots of Playmola showing visual feedback. On the left: an animatedhighlight/translucency effect when an object is selected. On the right: visual effect resem-bling sparks when two objects are connected. The sound of welding is played along withthe effect.

3.2.2 Curiosity

Malone defines “sensory curiosity” as an effect invoked by audio and visual effects in agame and suggests that it can be used in three main ways:

• As decoration – i.e. to enhance aesthetic qualities of the application. Wieman [36]supports this in writing about the importance of “an appealing environment and rea-sonably sophisticated graphics” when making an engaging simulation tool.

• To enhance fantasy.

• As a representation system – either through the use of metaphors or as a means ofproviding feedback in response to events.

Playmola’s aesthetics are enhanced by background music and sound effects fitting theapplication’s theme, coupled with the visual effects shown in Figure 8. These audio/visual

16

effects also provide additional feedback in response to user interaction. When connectingobjects, valid connections are highlighted in a variety of ways (depending on the methodof connection), some of which are shown in Figure 9. A clicking sound plays to indicatethat a connection has successfully been made. Figure 8 also shows interaction-driven visualfeedback in the form of a visual effect meant to represent “welding” which is played whentwo objects are connected.

The application’s background music also changes dynamically, based on the state theuser is in. While creating a model (the majority of the time spent using the application),unobtrusive piano-based background music is played. In order to signify the change instate when a simulation starts, the music changes to a similar but more energetic tune,with industrial “clanking” noises, to reinforce the workshop context. If the simulation fails,the music will return to the normal background music (along with an error tone).

Figure 9: Screenshot of Playmola, showing visual feedback when connecting objects. Avail-able connections are highlighted and the connection-line changes colour when it is nearenough to a valid target to form a connection.

3.2.3 Challenge

A common element of proposed gamification frameworks is challenge [1, 5], somethingthat Malone presents as one of the key features of games that can be used in effectiveuser interface design. Challenge is usually defined as the presence of tasks with clear goals(and often with feedback regarding how close to achieving a goal the user is). Since thefocus of this study is on intrinsically motivating factors and the purpose of the applicationis not to complete specific predetermined tasks, explicit goals are not used. Challenge ishowever still present in the system by virtue of users setting their own goals: creatingmodels which can be successfully simulated and which exhibit the expected behaviour.

17

While not specifically an application of gamification, this element of challenge may havepositive benefits on the users’ enjoyment of the system.

3.3 Use of 3D Technology

Wieman and Perkins describe sophisticated graphics as one of the critical features of aninteractive simulation [36], so tools facilitating the creation of attractive 3D environmentswere required. Numerous engines and libraries exist for creating 3D applications and, asdescribed in Section 1.2.5, game engines have previously been used for creating simulationtools.

Due to the need to run directly in a web browser without a plugin, Playmola requiredan engine either written in JavaScript or capable of exporting to JavaScript. three.js [32],a popular JavaScript 3D rendering API based on WebGL, was chosen for Playmola’s devel-opment since it provides the required graphical functionality and Dymola already includesfunctions for exporting 3D models in a format compatible with three.js. While not specif-ically a game development framework, three.js contains the graphical features expectedfrom such a framework, including: lighting, shadows, texturing and import of 3D modelsin a variety of formats. The main feature missing from three.js that would have facilitateddevelopment is a graphical editor for positioning objects in a 3D scene – a feature whichis present in most modern game engines.

Another promising choice was the game engine Unity [34], but support for exportingcode which can be run natively in the browser was not included until after developmentof Playmola began, thereby ruling it out. Larger-scale game engines such as Unreal En-gine [35] also allow development of applications that can run natively in a web browser,but also provide a large amount of extra functionality not required for an application ofPlaymola’s scale and have a steep learning curve.

4 User Study

This section describes the format of the user study introduced in Section 2.2.Eight participants took part in the user study, consisting of two mechanical engineering

students with experience of courses in physics and mathematics, four first-year game-development students who had completed a course in game physics and two third-yeargame development students who had completed an introductory physics course. Due tothe nature of the simulation tool’s intended user – a person interested in physics and withsome experience of education in physics and/or applied mathematics – and the fact thatthe prototype assumes some knowledge of linear algebra and mechanics, test subjects werechosen based on having taken at least one university-level course in physics.

Each test session lasted between 40 minutes and one hour and consisted of the followingprocess:

• Playmola was introduced and described as “a tool for constructing and running me-chanics simulations”. Care was taken not to describe the tool as a game or to usegame-related terminology.

18

• A brief (approximately 10 minute) introduction of Playmola was given, in whichone of the test organisers followed a scripted overview of controls and functionalityand demonstrated how to construct two simple models – a pendulum and a boxsliding down an inclined plane. This tutorial-style introduction took the place of atraditional built-in software tutorial, which was not present in the prototype and wasintended to speed up the initial learning-process.

• The user was then given a short period of time to get acquainted with the tool andwas encouraged to ask questions. The test organisers used their judgement to decidewhen the user was ready to progress to the next stage, generally taking between 10and 15 minutes. To aid the familiarisation process, the user was given a printed sheetof pictures and descriptions of the functionality of all of the components present inthe system (again as a replacement for a built-in tutorial or guide).

• The user was given two tasks to complete in order: a wheel rolling down an inclinedplane and a model of a swing from a fairground ride. These models were chosen firstlyfor their simplicity, to allow testers with limited knowledge of physics to completethem and for the fact that the second model would produce movement in all threedimensions, allowing the effectiveness of the 3D interface to be properly studied. Foreach task, the user was presented with an image of the final model and a schematicof the components to include and how they should be connected. This stage tookbetween 10 and 20 minutes to complete.

• Upon completion of both tasks, the user was asked to fill in a questionnaire con-sisting of a mixture of ranking-based questions rating a statement according to afive-point Likert Scale (e.g. from Strongly Disagree to Strongly Agree), yes/no ques-tions and questions requiring longer written answers. This stage took between 10and 15 minutes to complete.

During the testing sessions, the second test organiser made written observations of thebehaviour and actions of the user. An audio-recording of each test session was also made.

5 Results of the User Study

This section presents the results of the user study described in Section 4.

5.1 Ranked Questions

Nine ranked questions (see Figures 10-18) were answered using a five-point Likert Scale,from “Strongly Disagree (1)” to “Strongly Agree (5)” (with the exception of question 9,see Figure 18). All questions were answered by all participants.

Users were given the opportunity to expand upon visual feedback in relation to question1 (see Figure 10). No comments were made on the use of visual effects like the sparks effectseen in Figure 8. Several users commented on how the Exploded View construction modemade it easier to understand how their models worked. One user found it useful being ableto simulate and construct in the same view, so that changes to parameters could be made

19

directly after watching the animated simulation. Some users felt that more visual feedbackwas needed, for example to indicate the cause of the error when a simulation fails.

Users were also given the opportunity to expand upon audio feedback in relation toquestion 2 (see Figure 11). Responses were varied, but most users were indifferent to theapplication’s use of music and sound effects. All realised that there was sound, but onecommented that he hadn’t heard any sound except for the background music, which hedidn’t consider to have any effect on his ability to understand the application. Severalusers stated that the sound effect played when a simulation fails was useful.

Figure 10: Question 1 Figure 11: Question 2

Figure 12: Question 3 Figure 13: Question 4

20

Figure 14: Question 5 Figure 15: Question 6

Figure 16: Question 7 Figure 17: Question 8

Figure 18: Question 9

21

5.2 Other Questions

Qualitative data were gathered from the responses to the remaining survey questions andare presented here. All answers have been translated from Swedish.

10. How much physics/mechanics have you studied at university level?

All users had some experience of physics and/or mechanics, where at minimum onecourse had been taken and the most experienced were two mechanical engineeringstudents with regular experience of physics at university level.

11. Do you feel you understood the concepts involved in the tasks?

All users wrote that they understood the concepts involved after some time exper-imenting and with the help of the information we provided along with the tasks.Those with more experience of physics were more confident in their understanding,while several of those studying game development stated that they would not havebeen able to complete the tasks without (verbal and written) assistance.

12. How did it feel to navigate in Playmola? Was the interaction natural or forced?

The test participants were quite consistent in agreeing that navigating the interfacehad somewhat of a learning curve, but most felt that the time provided in the testingsession was sufficient to become used to it. Only one user felt that significantly moretime would be required to become familiar with navigating the interface. Consistentlycriticised was the inability to move the camera left and right (instead the camerarotates around a fixed point and can be zoomed in and out), which several testersfelt was necessary when attempting to construct a large model. One user also haddifficulty getting used to the different functions provided by the application’s twodifferent construction modes (3D View and Exploded View, see Section 3.1).

Several testers noted that more information in the form of instructions or tool-tipswould have made the interface easier to understand.

13. How would you describe the application’s setting/theme?

All eight users were able to accurately describe the setting as a “workshop” or asimilar term (words used included: “workshop”, “tool-shed” and “garage”) conveyingthe idea of an environment where things are made – particularly an informal orpersonal setting, rather than an organised large-scale factory.

In addition to identifying the setting, several of the users also described enjoying thesetting, using words like “charming”, “pleasant” and “great”. One user in particulardescribed how the environment actively enhanced the user experience: “You comeinto a calm environment with calming music which isn’t distracting. It makes iteasier to concentrate on what you actually want to do.”

One user also mentions a game, Garry’s Mod [11] (a sandbox game, where the playerhas freedom to do as he/she pleases without explicit goals, not dissimilar to the kindof free experimentation Playmola allows), as a reference point for his interpretationof the setting.

22

14. Do you think the theme was suitable for the application?

Everyone felt that the theme fit the application well. One user commented thatthe experience of using Playmola wouldn’t have been the same without the theme:“Very fitting for sitting and experimenting with the program. Without the theme itprobably wouldn’t really have had the same feeling.” Another felt the theme helpedhim focus on the tasks: “You get into your little workshop and can concentrate.There’s nothing to distract you or make you lose focus.”

15. Did you enjoy using Playmola?

All but one user described themselves as enjoying Playmola including three users whoexpressed an interest in continuing to use it, including both mechanical engineeringstudents. One user (a game-development student) felt that a tool like Playmolawould be useful to help him understand mechanics: “Very entertaining. I could seemyself using a tool like this for testing out ideas or learning about how objects behavein certain mechanical situations.”. The remaining user didn’t feel he had been givenenough time with the program to formulate a decision about whether or not it wasenjoyable – also mentioning that it didn’t feel like a program that was supposed tobe entertaining, but rather a tool for work.

16. Do you think the application’s complexity was suitable for your current skill level?

Seven out of eight users answered “yes”. No opportunity was given to expand uponthis answer.

17. How many hours per week do you spend playing games?

All the users play games regularly. The user who played games the least estimated2-3 hours per week and the user who played the most estimated 20-30 hours perweek. Most played between 5 and 15 hours per week.

18. What kinds of games do you usually play?

Out of a small selection of genres (strategy, action, puzzle, sport, role-playing andother) the most popular were action (7 users) and puzzle (5 users) and the leastpopular was sport, with no users reporting that they play sports games.

19. I would use Playmola again.

All users answered “yes”.

20. Playmola has made me more interested in physics/mechanics.

Seven out of eight users answered “yes”.

21. General Comments

Several users reiterated their enjoyment of using Playmola and a number mentionedthat it could be useful as a learning tool (it was not presented as such): “I hopethis will be used in the future for engineers and physicists to be able to perform

23

simulations in an educational context” and “it would be a really nice tool to use forsimulating problems when studying”.

One user felt there was not enough content in the application and that more helpfrom built-in instructions and tool-tips was necessary.

5.3 Observation

Every user test was accompanied by observations focussing on the interaction between theuser and the system. One of the test organisers was primarily responsible for writing downthese observations.

Two different ways of starting a simulation are available to the user, either by clickinga 3D object in the scene or by clicking a more traditional HTML button in the GUI. Amajority of the testers used the traditional GUI button most of the time, rather thanthe 3D button in the scene. In some cases it was not entirely conclusive, with some usersswitching between the one and the other. One user used the button in the scene exclusively.

The mouse can be used to change the camera’s orientation. Some of the users appearedconfident using this method to orient themselves in 3D space. This made it easier for themto orient models and connect them together. A user’s ability to orient the camera with themouse did not necessarily correspond to their impression of the ease of navigation. Someusers rarely changed the camera perspective, except when absolutely necessary (becausean object was hidden behind another object), while others rotated the camera much morenaturally. Both mechanical engineering students made little use of the camera controls,while the third-year game development students were much more likely to rotate the camerato see a model from a different angle.

When a simulation fails in the application, an “error” sound is played. All of theparticipants understood the first time they heard this sound that the simulation had failed.It is interesting to note that one user claimed not to have heard any sound except for thebackground music, despite having responded as anticipated to the error sound when usingthe application. On the other hand, several users became confused when their simulationssucceeded but did not result in any movement (because of the physical properties of themodel they had created) – assuming that it was the result of a bug in the program.

The application allows the user to switch between two modes when constructing models(see Section 3.1), each providing access to different functionality. Most users spent almostall their time in the Exploded View, which provides more functionality and some becameconfused by the functionality missing in 3D View (fewer components are available). Asmall number switched regularly between modes in order to better visualise the completemodel.

6 Analysis

This section presents an analysis of the results of the user study. Firstly, the effects ofspecific game elements are analysed according to the categories of game elements definedby Malone [18], as discussed in Section 3.2. Secondly, the effects of using 3D techniquesare analysed. Finally, the overall effects of gamification on user experience are analysed.

24

6.1 Fantasy

6.1.1 Theme/Story

The theme of a workshop was used, making use of visual and audio elements to invokea specific atmosphere with the goals of being recognisable and contextually appropriatefor the kinds of activities performed in the application (experimenting with physical mod-els). Answers to the question “How would you describe the application’s setting/theme?”revealed that the application’s implementation of a setting was effectively conveyed, withevery user independently identifying the setting as a tool-shed, garage or a workshop. Usersalso commented on how the atmosphere of the program invited them to create things andexperiment, encouraging the sort of playful/experimental behaviour that gamification canbe used to motivate. Most of the users did feel that the theme contributed to their enjoy-ment of the application (see Figure 18) and this is further supported by comments madeabout pleasant atmosphere of the setting. One user also commented that the theme hadmade it easier for him to concentrate on the tasks - stating also that the application wouldnot have been as enjoyable without the setting.

6.1.2 Metaphors

The application’s use of visual metaphors takes the form of interactive objects designed toresemble real-life objects (as recommended by Malone [18] and Wieman and Perkins [37])with easily recognisable functions - implemented here in the form of a the radio and the “bigred button” as shown in Figure 7. Users were asked not to interact with the radio (whichtoggles the application’s audio), since the effects of audio were being studied, but thefunction of the button (starting a simulation) was explained and demonstrated. While itwould have been interesting to be able to analyse whether or not users chose to disable themusic, it was decided in advance of the testing sessions that data regarding user responsesto the context-sensitive music would be more useful. The interface also included a moretraditional HTML button with the text “Simulate”. As noted in section 5.3, only one userconsistently used the interactive object rather than the traditional button and others usedit very infrequently. This is unsurprising since the traditional button is more recognisableto a regular computer user, even though none of the users felt the 3D object was out ofplace in the application’s setting. Additionally, the position of the 3D object moves asthe user adjusts the camera angle (since it is attached to the scene’s workbench), possiblymaking it harder to locate than the traditional button, which is always in the same screenlocation. It is also possible for other objects placed in the scene by the user to obscure theinteractive objects, making them harder to use. The one user who did use the 3D objectrather than the traditional button most of the time was also observed to reposition thecamera very infrequently, ameliorating the aforementioned problems in this case, but notrepresenting behaviour consistent with the way most users interacted with the application,or the anticipated behaviour.

25

6.1.3 Audio/Visual Effects

The background music received only one comment related to its relevance to the theme ofthe application, stating that it contributed to a relaxed setting.

No user commented specifically on the use of a sound effect as an indication of afailed simulation, with one even stating that he had heard no sound effects apart from thebackground music during his use of the application. This contradicts the observations madeof the users since every user was observed to react to the sound effect. This contradictionmay be related to the sound effect being short and dissonant, catching the attention of theuser as it plays but not making a lasting impression. It should also be noted that soundeffects conveying error or success messages are not unusual in applications and certain notrestricted to gamification, so the fact that the users are used to hearing this kind of soundeffect may have made them less noticeable.

6.2 Curiosity

6.2.1 Decoration

Decoration refers here to the use of audio and visual effects to enhance the application’saesthetic qualities. While it is difficult to draw any conclusions about the precise con-tributions of isolated decorative elements on the users’ perceptions of the application’sattractiveness, comments made about the theme and audio/visual feedback provide someindications. In addition to correctly identifying the application’s theme, users described itas “charming” or “pleasant” and one explicitly commented on purely decorative elements:

...it was fun that you put energy into small things like a radio or boxes in acorner. Dirty and broken windows made it a bit ‘mysterious’. It would havebeen extremely boring to have the usual standard 3D environment which neverends.

The same user also felt that the interface was difficult to navigate and was the onlyuser to explicitly state that he felt he hadn’t been given enough time with the application,indicating perhaps that an increased focus on decorative aspects actually had a negativeeffect on understanding the application. Another user said about the setting: “probablyneeds to be made a bit more cosy”, suggesting that further work might need to be done tomake a more aesthetically pleasing environment.

Comments about the application’s music and sound effects were mixed. One userdescribed the music as “nice”, but another felt that it was a little annoying after hearingit for 45 minutes. There were no comments about the aesthetic qualities of the soundeffects, most of which were not actively perceived by the users at all. A lack of commentsabout music and a general lack of realisation that there were sound effects connected tocertain actions may indicate that the application’s audio hit a good balance between beingcontextually appropriate (and providing useful feedback) and being distracting – this sayslittle about the aesthetic qualities of the audio though.

26

6.2.2 Representation System/Feedback

Seven of eight users agreed or strongly agreed with the statement of question one: “Visualfeedback made it easier to understand Playmola” (see Figure 10), suggesting that the useof visual feedback was effective. However, responses to a follow-up question giving anopportunity to expand upon this question revealed that users interpreted the question indifferent ways, commenting on Exploded View or the animated results of a simulationrather than on the use of visual effects like sparks or the selection highlight.

Responses to the statement of question two: “Audio feedback made it easier to under-stand Playmola” (see Figure 11) were varied, with the average slightly agreeing with thestatement, even though in a follow-up question none of the users had any specific commentsto make about the effects of audio on their perception of the application.

6.3 Challenge

No challenges, in the sense of tasks with clear goals, are present in the application. Thetasks presented to the users during the user study did provide some level of challenge butwill not be analysed from a gamification perspective since these are not an inherent partof the system. Some insight into the users’ subjective experience regarding how difficult,frustrating and fun they considered the experience to be can be found in Section 6.5.

6.4 Technology

The use of a single, unified 3D view for construction and animation of models - insteadof the disconnected 2D view for construction of models and 3D view for animation ofsimulation results seen in Dymola - appears useful from the perspective of helping newusers understand the application. All users agreed that the 3D environment made it easierfor them to visualise the models, one pointing out that the unified view made it easyto tweak parameters in a model and get direct visual feedback reflecting the changes.Observations also showed that the ability to view the complete model in 3D allowed usersto quickly discover the source of any issues.

In addition to the benefits of a unified view, the majority of users agreed that theuse of shadows and lighting helped them to position the models, demonstrating that moreadvanced graphical features present in modern 3D libraries or game engines can be usefulwhen constructing 3D simulation tools - agreeing with Bijl and Boer’s conclusions [4].Given the increased difficulty of navigation in a 3D environment compared to a 2D one,these kinds of visual aids may be crucial in helping users understand how to manipulateand position objects in a 3D scene.

Despite the positive aspects, it took users some time to become accustomed to navi-gating in the 3D environment, possibly corresponding to the users’ familiarity with gamesand/or game development, since the mechanical engineering students seemed to take longerto get used to using the system than the game development students. A user’s ability tonavigate the system did not, however, correspond to the user’s expressed opinion of theease of navigation at all times, with one user who struggled with the controls describing thenavigation as “very easy” and one who was used the controls more naturally describing it

27

as “complicated at first” (this user was also the only one to describe the tasks as frustrating- see Figure 16).

6.5 User Experience

When considering how enjoyable the users found Playmola, it is interesting to considertheir perceptions of what kind of application they were using. Throughout the user study,Playmola was described to users as a “tool” rather than a game, but responses to thequestionnaire and other observations made it clear that some users had the impressionthat it was in fact a game. One user in particular consistently referred to “the game” in hisresponses the the questionnaire, while a number of users verbally referred to Playmola as agame during and after the sessions. Another user wrote “It felt more like a work-tool thanan entertainment program”, implying that he was under the impression that the principalpurpose of the application was entertainment. All users were aware that the applicationwas developed by game development students, which may have led to assumptions aboutthe nature of the work, but confusion about the nature of the application may also havebeen a result of it feeling like a game, due to the use of game elements and experimentation-based functionality which resembles “play” (it should also be noted that a general personmay not appreciate the differences between “games” and “play” and use the word “game”when experiencing playful environments).

Questionnaire responses confirm that users found the tasks fun to complete (see Fig-ure 17) and, when asked how much they felt that the theme had contributed to theirenjoyment, the users responded with an average of 3.65 on a scale from 1 (not at all) to5 (significantly). This, along with observations and comments, suggests that the gamifiedsetting played a role in users’ enjoyment of the application. While, on average, the usersfound the tasks neither easy nor difficult, all but one agreed that they were not frustrating,suggesting that the tasks were of a difficulty level suitable to not negatively affect enjoy-ment. Malone’s [18] ideas on the importance of challenge in good user interface designsupport that an experience targeted correctly to the users’ abilities (that is, challenging,but not frustrating) can lead to a more enjoyable experience.

All users stated that they would use the application again and two expressed an in-terest in being able to use it again before being asked, further reinforcing results that theapplication was enjoyable to use. Additionally, seven of eight said that Playmola madethem more interested in physics/mechanics - a desirable result given that one of the aimsof gamifying an application is to increase users’ engagement and interest [9] (in this caseby making it more enjoyable for them to use).

7 Discussion

The use of gamification appears to be a successful strategy for developing enjoyable simula-tion software with users responding well to the presence of a theme and the visual stimuli.Malone’s suggestion of using the concept of fantasy to appeal to users’ emotions [18] issupported by the results and the use of a relatable environment did have a positive effecton the user experience.

28

By limiting the applications of gamification to exclude those most associated withpointsification (badges, points and leaderboards), the thesis shows that gamification canbe effective when the focus is shifted to intrinsically rather than extrinsically motivatingelements. It is interesting to consider whether the term gamification has been so colouredby negative feedback from the likes of Bogost [6] and the pervasion of pointsification-basedapplications that it should not be applied at all in this case. While the use of game me-chanics in Playmola certainly conforms to Deterding et al.’s [9] definition of gamification,it might be more useful from the perspective of terminology to use another term, suchas “playification”, which Nicholson [21] recommends for applications which focus on play-ful interaction, like the experimentation-based interaction in Playmola. It is also worthmentioning that many of the features identified as gamification, such as audio and visualfeedback would be considered by many to be simply a part of good user interface design.

In addition to its use in studying the benefits of the use of game elements, Playmolaserves as a guide for the application of gamification as a core component of an application,where the focus is on motivating the user with elements which are fun and involve the useron an emotional level. Also shown is how elements from formal models/frameworks forgamification can be applied in a way that places the user at the centre of the experience,as Nicholson [21] recommends, by the use of contextually appropriate themes and ideasthat the user can identify with.

The implementation of metaphors, the interactive radio and red button, could havebeen more thorough. The decision to leave traditional GUI buttons that had the samefunctionality as the interactive objects in the scene might provide a more user-friendly UI(the objects in the scene can be obscured by other objects in front of them) but had theynot been present, the users would have had even more incentive to interact with the scene.

Subjective evaluation by users suggests that the use of 3D techniques such as lightingand shadows made the application easier to use and understand, while features such astexturing and the loading of pre-made 3D models allowed the application’s setting to bemade more aesthetically appealing - helping to enhance the use of fantasy as a gamificationelement. Improved aesthetics could have had a larger impact if the application’s graphicalassets had been produced by professional artists.

The implementation described and studied involves the application of multiple gameelements as a whole, rather than individual elements in isolation. While an attempt wasmade to analyse each element separately, it is naturally not possible to judge accuratelythe effects of each on factors such as overall enjoyment. Hamari et al. [12] criticise thelack of studies of effectiveness of isolated elements, but as Deterding et al. [9] point out, agame is characterised by a collection of elements in combination, it is possible that part ofthe natural essence of what makes games fun is lost by studying elements in isolation (thesame criticism of gamification used by Bogost [6] and Robertson [27]).

One aspect which might have influenced the users opinions is the fact that they usedthe system during a relatively short time span. Each user expressed interest in using thesystem again, but this may have been based largely on first impressions. Several othersurveys related to the use of gamification in software include user tests taking place over amuch longer time [12]. This can be attributed to the fact that the aspects of gamificationwhich are analysed require use over time in order to be relevant, e.g. the use of points and

29

leaderboards. The elements examined in this study are not, however, of this nature - withmany of their effects being much more easily observed in a short time. First impressionsmay in fact be one of the most important aspects to consider, since they can determinewhether or not the user decides to return to an application after using it for the first time.

The lack of diversity in the participants of the study raises some questions about thegeneral applicability of the results presented. All of the participants play video games, giv-ing them additional insights and prejudices that may have altered their perceptions of theapplication. However, the application’s target user is young and therefore almost certainlyhas experience of playing games (likely on a regular basis) and one of the motivationsfor using interactive simulation tools in education is that young people are familiar withgame-like environments [36].

7.1 Suggestions for Further Research

Numerous avenues for further research present themselves, both on gamification in generaland the more specific branch explored here, with a focus on intrinsic motivation and fun.Both Seaborn and Fels [31] and Hamari et al. [12] suggest other directions for futureresearch (some of which informed this thesis), and the reader is directed to those literaturereviews for additional ideas about where the field of gamification could benefit from researchin the future.

While audio and visual stimuli are both very important parts of games, the studyshowed that users responded to and perceived visual aspects of the application more clearlythan audio aspects. Considering the high costs associated with developing sound andgraphical assets for use in applications, it would be interesting to carry out more researchinto the relative efficacy of the two, in order to better understand where to prioritise timeand spending.

A large number of previous gamification studies situate their work within education,most with positive results [31]. While the tool developed in this thesis was not exam-ined from an educational perspective, further research could examine how beneficial theuse of game elements chosen to promote a fun and playful experience is for educationalapplications (both in terms of promoting learning and user engagement).

In general, as suggested by Seaborn and Fels [31] and Hamari et al. [12], there is a needfor longer studies of gamification with larger numbers of participants, so that the long termeffects can be evaluated and errors introduced by first impressions can be eliminated.

While the study carried out in this thesis is based on models for the use of gameelements in user interface/software design, picking and choosing elements to fit the needs ofthe application, more comprehensive studies should be performed to fully evaluate specificmodels (such as Nicholson’s [21], Aparicio et al.’s [1] or Blohm and Leimeister’s [5]), inorder to determine whether they are effective foundations for the development of gamifiedsystems. Since much of the scientific background of these models of gamification lies inpsychology research, it would also be useful to make use of biometric data and behaviouralmetrics when analysing gamified applications.

30

References

[1] Aparicio, Andrés Francisco et al. “Analysis and Application of Gamification”. In:Proceedings of the 13th International Conference on Interacción Persona-Ordenador.INTERACCION ’12. New York, NY, USA: ACM, 2012, 17:1–17:2.

[2] Berengueres, Jose et al. “Gamification of a Recycle Bin with Emoticons”. In: Proceed-ings of the 8th ACM/IEEE International Conference on Human-robot Interaction.HRI ’13. Piscataway, NJ, USA: IEEE Press, 2013, pp. 83–84.

[3] Big Red Button. url: http://tvtropes.org/pmwiki/pmwiki.php/Main/BigRedButton(visited on 04/09/2015).

[4] Bijl, Jonatan L. and Boer, Csaba A. “Advanced 3D Visualization for Simulation UsingGame Technology”. In: Proceedings of the Winter Simulation Conference. WSC ’11.Phoenix, Arizona: Winter Simulation Conference, 2011, pp. 2815–2826.

[5] Blohm, Ivo and Leimeister, Jan Marco. “Gamification: Design of IT-Based EnhancingServices for Motivational Support and Behavioral Change”. English. In: Business &Information Systems Engineering 5.4 (2013), pp. 275–278.

[6] Bogost, Ian. Persuasive Games: Exploitationware. Mar. 2011. url: http://www.gamasutra.com/view/feature/6366/persuasive_games_exploitationware.php(visited on 04/07/2015).

[7] Deci, Edward L., Koestner, Richard, and Ryan, Richard M. “A meta-analytic reviewof experiments examining the effects of extrinsic rewards on intrinsic motivation”.English. In: Psychological bulletin 125.6 (1999), pp. 627–668.

[8] Deterding, Sebastian. “Situated motivational affordances of game elements: A con-ceptual model”. In: Vancouver, BC, May 2011.

[9] Deterding, Sebastian et al. “From Game Design Elements to Gamefulness: Defin-ing "Gamification"”. In: Proceedings of the 15th International Academic MindTrekConference: Envisioning Future Media Environments. MindTrek ’11. New York, NY,USA: ACM, 2011, pp. 9–15.

[10] Downes-Le Guin, Theo et al. “Myths and realities of respondent engagement in onlinesurveys”. In: International Journal of Market Research 54.5 (2012), pp. 613–633.

[11] Garry’s Mod. Digital Game. url: http://www.garrysmod.com/ (visited on 05/18/2015).

[12] Hamari, J., Koivisto, J., and Sarsa, H. “Does Gamification Work? – A LiteratureReview of Empirical Studies on Gamification”. In: System Sciences (HICSS), 201447th Hawaii International Conference on. Jan. 2014, pp. 3025–3034.

[13] Hu, Wenfeng, Qu, Zhouqing, and Zhang, Xiaoyuan. “A New Approach of MechanicsSimulation Based on Game Engine”. In: Computational Sciences and Optimization(CSO), 2012 Fifth International Joint Conference on. June 2012, pp. 619–622.

[14] Huotari, Kai and Hamari, Juho. “Defining Gamification: A Service Marketing Per-spective”. In: Proceeding of the 16th International Academic MindTrek Conference.MindTrek ’12. New York, NY, USA: ACM, 2012, pp. 17–22.

31

[15] Hwang, Wonil and Salvendy, Gavriel. “Number of People Required for UsabilityEvaluation: The 10+-2 Rule”. In: Commun. ACM 53.5 (May 2010), pp. 130–133.

[16] Li, Wei, Grossman, Tovi, and Fitzmaurice, George. “GamiCAD: A Gamified TutorialSystem for First Time Autocad Users”. In: Proceedings of the 25th Annual ACMSymposium on User Interface Software and Technology. UIST ’12. New York, NY,USA: ACM, 2012, pp. 103–112.

[17] Liu, Yefeng, Alexandrova, Todorka, and Nakajima, Tatsuo. “Gamifying IntelligentEnvironments”. In: Proceedings of the 2011 International ACM Workshop on Ubiq-uitous Meta User Interfaces. Ubi-MUI ’11. New York, NY, USA: ACM, 2011, pp. 7–12.

[18] Malone, Thomas W. “Heuristics for Designing Enjoyable User Interfaces: Lessonsfrom Computer Games”. In: Proceedings of the 1982 Conference on Human Factorsin Computing Systems. CHI ’82. New York, NY, USA: ACM, 1982, pp. 63–68.

[19] Modelica and the Modelica Association. url: https://www.modelica.org/ (visitedon 04/12/2015).

[20] Multi-Engineering Modeling and Simulation - Dymola - CATIA - Dassault Systèmes.url: http://www.3ds.com/products-services/catia/capabilities/modelica-systems-simulation-info/dymola (visited on 04/12/2015).

[21] Nicholson, Scott. “A User-Centered Theoretical Framework for Meaningful Gamifi-cation”. In: Proceedings of Games+Learning+Society 8.0. Madison, WI, USA, 2012.

[22] Ogre3D. url: http://www.ogre3d.org/ (visited on 04/19/2015).

[23] Pajitnov, Alexey. Tetris. Digital Game.

[24] Podolefsky, Noah S., Perkins, Katherine K., and Adams, Wendy K. “Factors promot-ing engaged exploration with computer simulations”. In: Phys. Rev. ST Phys. Educ.Res. 6.2 (Oct. 2010), p. 020117.

[25] Prasithsangaree, P. et al. “UTSAF: a simulation bridge between OneSAF and theUnreal game engine”. In: Systems, Man and Cybernetics, 2003. IEEE InternationalConference on. Vol. 2. Oct. 2003, 1333–1338 vol.2.

[26] Preece, Jennifer, Rogers, Yvonne, and Sharp, Helen. Interaction design: beyond human-computer interaction. English. Wiley, 2002.

[27] Robertson, Margaret. Can’t play, won’t play. June 2010. url: http://hideandseek.net/2010/10/06/cant-play-wont-play/ (visited on 04/07/2015).

[28] Ryan, Richard M. and Deci, Edward L. “Intrinsic and Extrinsic Motivations: Clas-sic Definitions and New Directions”. In: Contemporary Educational Psychology 25.1(2000), pp. 54–67.

[29] Ryan, Richard M. and Deci, Edward L. “Self-determination theory and the facilita-tion of intrinsic motivation, social development, and well-being”. English. In: Amer-ican Psychologist 55.1 (2000), pp. 68–78.

32

[30] Sabourin, J.L. and Lester, J.C. “Affect and Engagement in Game-Based Learning En-vironments”. In: Affective Computing, IEEE Transactions on 5.1 (Jan. 2014), pp. 45–56.

[31] Seaborn, Katie and Fels, Deborah I. “Gamification in theory and action: A survey”.In: International Journal of Human-Computer Studies 74 (2015), pp. 14–31.

[32] three.js. url: http://threejs.org/ (visited on 04/12/2015).

[33] Tullis, Tom and Albert, Bill. Measuring the user experience: collecting, analyzing,and presenting usability metrics. English. Amsterdam: Morgan Kaufmann, 2013.

[34] Unity. url: http://unity3d.com/ (visited on 04/15/2015).

[35] Unreal Engine. url: https://www.unrealengine.com (visited on 04/15/2015).

[36] Wieman, Carl E. and Perkins, Katherine K. “A powerful tool for teaching science”.In: Nat Phys 2.5 (May 2006), pp. 290–292.

[37] Wieman, Carl and Perkins, Katherine. “Transforming physics education”. In: PhysicsToday 58.11 (Nov. 2005), pp. 36–41.

[38] Wixon, Dennis. “Measuring Fun, Trust, Confidence, and Other Ethereal Constructs:It Isn’t That Hard”. In: interactions 18.6 (Nov. 2011), pp. 74–77.

[39] Zichermann, Gabe. Intrinsic and Extrinsic Motivation in Gamification. Oct. 2011.url: http://www.gamification.co/2011/10/27/intrinsic-and-extrinsic-motivation-in-gamification/ (visited on 04/07/2015).

[40] Zichermann, Gabe. The purpose of gamification. Apr. 2011. url: http://radar.oreilly . com / 2011 / 04 / gamification - purpose - marketing . html (visited on04/07/2015).

[41] Zichermann, Gabe and Cunningham, Christopher. Gamification by Design: Imple-menting Game Mechanics in Web and Mobile Apps. O’Reilly Media, 2011.

[42] Zichermann, Gabe and Linder, Joselin. Game-based marketing: inspire customer loy-alty through rewards, challenges, and contests. English. Chichester; Hoboken, N.J:Wiley, 2010.

33