Interactive user onboarding and its effect on activation rates1471418/FULLTEXT01.pdf · onboarding...
Transcript of Interactive user onboarding and its effect on activation rates1471418/FULLTEXT01.pdf · onboarding...
IN DEGREE PROJECT COMPUTER SCIENCE AND ENGINEERING,SECOND CYCLE, 30 CREDITS
, STOCKHOLM SWEDEN 2020
Interactive user onboarding and its effect on activation rates
A statistical study of feature introductions in applications with complex interfaces
FILIP STÅL
KTH ROYAL INSTITUTE OF TECHNOLOGYSCHOOL OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCE
Abstract
User onboarding procedures and digital feature introductions are still in their infancy. Consequently, hard data on its impli-cations and its effects on user behavior is scarce and difficult to come by. To contribute to this field of research, within the context of HCI, this study was conducted. It studies the implementation of a contextually sensitive step-by-step feature in-troduction on a digital platform, presented by a complex and composite interface, providing access to a large number of features. It then proceeds to statistically evaluate the effects of the onboarding implementation on activation rates for which the onboarding was designed to improve.
The study’s findings were that users that received the step-by-step feature introduction were associated with statistically significantly lower mean times spent completing the activation requirements, compared to users that did not. Additionally, users partaking in the feature introduction had higher overall activation rates. The study also provides insight into how user time spent completing the different steps leading up to a successful activation was divided among the steps and discusses its significance for the design of user onboarding experiences in the future.
Sammanfattning Processer fo r onboarding av anva ndare till digitala applikationer och dess funktioner a r fortfarande ett relativt nytt och ostuderat forskningsa mne. Fo ljaktligen a r kvantitativ data om dess pa verkan pa anva ndares beteende ba de knappha ndig och sva rfunnen. Studien genomfo rdes da rfo r i syfte fo r att bidra till detta forskningsomra de, inom ramen fo r HCI. Den studerar implementeringen av en kontextuellt medveten steg-fo r-steg onboarding pa en digital plattform, som representeras genom ett komplext och sammansatt gra nssnitt som portal till ett stort antal olika anva ndarfunktioner. Da refter fortsa tter den att statistiskt utva rdera effekterna som implementeringen av onboardingtja nsten fo r med sig, samt den fo ra ndrade aktiveringsgrad hos nya anva ndare som den bidrar till. Resultaten av studien var att anva ndare som fick ta del av den nya interaktiva och stegvisa onboardingprocessen fo rknippades med statistiskt signifikant la gre genomsnittliga tider fo r att fullborda aktiveringskraven, ja mfo rt med anva ndare som inte fick ta del av den nya onboardingprocessen. En fo rba ttrad aktiveringsgrad fo r anva ndare som fa tt ta del av onboardingen kan ocksa observeras. Studien ger a ven inblick i hur anva ndarnas tid spenderades mellan de olika stegen i processen som ledde till en lyckad aktivering, samt att studien diskuterar resultatens betydelse fo r utformningen av onboardingprocesser fo r digitala applikationer i framtiden.
Interactive user onboarding and its effect on activation rates
A statistical study of feature introductions in applications with complex interfaces
Filip Stål
KTH Royal Institute of Technology
Stockholm, Sweden
ABSTRACT
User onboarding procedures and digital feature intro-
ductions are still in their infancy. Consequently, hard
data on its implications and its effects on user behavior
is scarce and difficult to come by. To contribute to this
field of research, within the context of HCI, this study was
conducted. It studies the implementation of a contextu-
ally sensitive step-by-step feature introduction on a dig-
ital platform, presented by a complex and composite in-
terface, providing access to a large number of features. It
then proceeds to statistically evaluate the effects of the
onboarding implementation on activation rates for
which the onboarding was designed to improve.
The study’s findings were that users that received the
step-by-step feature introduction were associated with
statistically significantly lower mean times spent com-
pleting the activation requirements, compared to users
that did not. Additionally, users partaking in the feature
introduction had higher overall activation rates. The
study also provides insight into how user time spent
completing the different steps leading up to a successful
activation was divided among the steps and discusses its
significance for the design of user onboarding experi-
ences in the future.
KEYWORDS
User Onboarding; Software Onboarding; Feature Tours;
Feature Introduction; User Activation
1. INTRODUCTION
There are countless strategies for addressing and im-
proving user activation in digital applications, such as
newsletters, gamification, loyalty programs, and tar-
geted ads. The process from user signup to initial suc-
cessful use of a platform, henceforth referred to as the
onboarding process, is an important step in the user ex-
perience journey, and one which potentially has great
effects on customer activation rates - but where hard
data is scarce and difficult to come by. The onboarding
process is also where a significant portion of software-
as-a-service (SaaS) customers get stuck and are lost [4].
As an online platform grows and evolves, it is not uncom-
mon that its user interface becomes increasingly bloated
and complex, where Facebook and Twitter are two prime
examples where this trend can be observed. This intro-
duces usability challenges in the form of information
overload and cluttered interfaces, both of which are com-
mon and serious concerns for digital products imple-
menting new features over time. The Challengermode es-
ports platform is no exception to this phenomenon. Back
in 2016, more than four years ago, a master’s thesis study
labeled it a complex real-time web user interface [20].
Since then, a considerable number of new features, types
of users, and technologies have been added, making the
interface increasingly difficult for new users to navigate
and grasp.
When someone starts playing a new computer- or con-
sole game, they frequently get to play some introductory
and interactive tutorial of sorts, which along with in-
structions teaches the new player the movement- and in-
teraction controls of the game. The impact of such tuto-
rials and step-by-step feature introductions on user acti-
vation and retention, appears to vary with game com-
plexity [1]. Similar to this is the concept of product- or
feature tours, which is a user onboarding concept for se-
quentially teaching Software-as-a-Service- (SaaS) and
Platforms-as-a-Service (PaaS) users how to use, or do
something.
The concept of user onboarding within HCI is argued by
some, to still be in its infancy [12], and the best-practices
within the industry are still being worked out. This study
aims to examine the implementation process and the ef-
fects on user activation that the implementation of a
step-by-step user onboarding procedure has for a web
application, presented by a complex interface. The study
attempts to do so through answering the following re-
search question.
[RQ] How does the implementation of a step-by-step
feature introduction affect new user activation rates,
for a specific task, on services with complex web-inter-
faces?
2. BACKGROUND
In this section, previous related works that are relevant
to the study are presented. The context of the study as
well as definitions and descriptions of the concept of user
onboarding is researched.
2.1 Case description
This study was conducted in collaboration with Chal-
lengermode AB. Challengermode is an online platform
with a mission to make competitive esports available for
amateurs and professionals alike. The platform is a tool
providing a large number of features for its users, includ-
ing social & community communications, a marketplace
for selling esports-related services, and the ability to host
and participate in tournaments. Although this thesis is
conducted with Challengermode, the methodology is a
general one, applied in a specific context. Therefore,
hereinafter, the Challengermode platform is referred to
as The Company, The Platform, or The Product. This is
done in order to not over-empathize the specific setting
and its significance for the study’s result and its findings,
which is thought to be of general nature. The context is
generalized to a website presented by a complex web
user-interface. This is elaborated on further in the dis-
cussion section.
2.3 Teaching a product’s functionality
For a long time, the common practice for teaching a digi-
tal product’s functionality was, with few exceptions,
through frequently asked questions sections (FAQs),
user manuals and documentation [5, 18]. However, these
methods of conveying instructions have been found both
unappealing [26] and insufficiently helpful [32].
1 https://www.businessofapps.com/data/app-statistics/#7
As a result of this, more state-of-the-art solutions have
been invented and developed throughout the years.
Modern onboarding experiences often instead incorpo-
rate welcome messages, product tours and progress
tracking [2]. There is a trend towards adaptive and con-
textually aware documentation [7], utilizing action-
driven tooltips and textboxes [2, 10]
2.4 Onboarding defined
Onboarding is a term most frequently used in business
contexts and within organizational theory, where it re-
fers to an organization's practice of facilitating and intro-
ducing new employees [3, 29]. The field of human-com-
puter interaction has, to some extent, adopted this termi-
nology, but instead refers to new users of a product and
their first impressions and experiences with said prod-
uct. The subject of user onboarding within HCI is argued
by some, to still be in its infancy [12], and its definition
and terminology yet to be adequately established within
the HCI community and field of study [14, 28]. Occasion-
ally one might therefore stumble upon some analogous
terminologies, where parts of the HCI community tend to
use the more general term “learnability” [14] or the ver-
bose, yet ambiguous - “new user experience”.
2.5 Why Onboarding
The percentage of returning users varies greatly be-
tween different types of applications, but in general, Hu-
lick estimates that between 40-60 percent of users will
return a second time to a digital product [17]. Eleven in-
dividual usage sessions are frequently used as the bench-
mark for having achieved application retention, and ac-
cording to statistics report provided by busi-
nessofapps.com1, using data provided by Statista [8], the
average app retention rate 2019 was a meager 32 per-
cent. The same report also states that 25 percent of ap-
plications downloaded worldwide are only used once.
Lincoln Murphy, a well-known growth architect in the
onboarding industry strongly argues that the main rea-
son behind this, is due to lacking user onboarding expe-
riences [22]. Several other publications also highlight the
importance of having users realize a product's value as
soon as possible, [17, 22, 23].
2.6 Best practices in user software onboarding
The onboarding process should strive to make the user
realize the value of the product as soon as possible, while
also easing the user into its functionalities and interface
at a comfortable pace [17, 18, 23]. It goes without saying,
that the optimal onboarding experience varies greatly on
a case-by-case basis for different features and products.
While video documentation might be the preferred solu-
tion for some cases, in-product tours might be the more
suitable option for another. A substantial number of fac-
tors need to be considered, ranging from what types of
users there are and what is known about them to what
the onboarding implementation tries to achieve. This
fact makes the actual design and planning of the
onboarding procedure a complex, but immensely im-
portant one.
2.7 Product- and Feature Tours
Within the user onboarding industry there are a few ma-
jor players that sell SaaS’s that allow companies to, alleg-
edly without much effort, implement in-product user ex-
periences. They often allow for the creation of in-product
product tours, A/B testing and different options for col-
lecting user feedback (e.g. trychameleon.com or inter-
com.com). With industry standards being ill-defined and
best-practices not yet agreed upon, the knowledge, re-
sults and findings that these companies share are among
the few sources of information that exists out there.
Fortunately, some of these companies regularly release
guidelines for implementing onboarding experiences
and maximizing user impact. They also publish collected
statistics and benchmarks on user engagement for their
different products, one example being their product
tours [6, 19]. One must take into consideration that these
companies also have a financial incentive to make these
product tours appear as impactful as possible. Even so,
these companies have access to significant amounts of
data, and thus makes for an interesting source of insight
and statistics when it comes to user onboarding.
Chameleon published a benchmark report on their prod-
uct tours in 2019, where they shared insights gathered
from 15 million user interactions [6]. They themselves
state that motivation behind the benchmark report is
that hard data on engagement, performance and conver-
sion rates of user onboarding and product tours are
scarce. Some of their key findings were that:
• The average completion rate for a product tour
is 61%.
• Keeping tours short is of utmost importance. A
violent drop-off could be observed if tour length
exceeded 4 steps. Completion rates for 4-step
tours were 46%, while a 5-step tour had a mea-
ger 23%.
• Users averaged 12 seconds to complete the
tours.
• Design matters. Modals, tooltips and progress
indicators improved completion rates of tours.
• Context mattered a lot. Users turned out to be
123% more likely to complete a tour which they
triggered themselves, compared to one that was
triggered automatically.
3. METHODOLOGY
This section describes and motivates the different meth-
ods used in the research. It communicates what research
was conducted, what data was collected and how the
data was collected. It also includes how the data was an-
alyzed as well as materials and software that was used in
the study.
The goal for this thesis was to design, implement and
evaluate the implementation of a step-by-step feature in-
troduction on a PaaS, presented by a complex user inter-
face. The study is inherently multifaceted and has a de-
velopmental and statistical approach to its methodology,
where an attempt is made to measure artifactual impacts
on a composite system. Therefore, the Design Science
paradigm [15] was highly relevant for the study and con-
sequently, a variant of the Design Science Research
Methodology (DSRM), described by Peffers et al. [25],
was adopted. The methodology focuses on six distinct
phases and brings about specific guidelines on iteration
and evaluation for research projects.
Figure 1. Different steps of A3, the implemented feature introduction. Larger resolution images can be found in figures 2-6.
3.1 The Design Science Methodology
1. Problem- and motivation identification:
Define the problem and describe the need
for a solution.
2. Defining the objective for a solution: De-
fine what the artifact should accomplish.
3. Design and development: Creation of the
artifact.
4. Demonstration: Use of the artifact and
demonstration of how it solves the problem.
5. Evaluation: Evaluation of how well the arti-
fact solves the problem. Iterate back to de-
velopment and improve on the solution if
necessary.
6. Communication: Communicate the find-
ings of the study to relevant audiences.
3.2 Identification of the problem and its motivation
The first and preparatory part of the study consisted of a
thorough literature study on the subject of user onboard-
ing combined with iterative meetings with The Company.
The goal of the meetings was primarily to define the re-
search problem and to justify the value of a solution to
the user onboarding problem. The purpose of the litera-
ture study was to immerse in the onboarding subject
within the context of HCI and to examine different possi-
ble approaches to solving the problem of easing users
into the different functionalities of the platform. A con-
textually relevant step-by-step feature introduction was
concluded as the desired solution for teaching users the
steps necessary in order to participate in esports tourna-
ments on the platform. This feature was selected due to
it being identified as one of the more important Key Per-
formance Indicators (KPIs) for The Company. It included
connecting external game accounts with your PaaS ac-
count, which is a necessity for taking advantage of many
of the platform’s features.
3.3 Defining the objectives for a solution
Following the identification and definition of the re-
search problem was a phase of defining the desired ef-
fects of the onboarding implementation. Qualitative and
quantitative objectives were considered as to compare
with the outcome of the onboarding implementation.
The implementation should strive to statistically signifi-
cantly reduce user times spent on the platform before ac-
tivation requirements (connecting a game account) are
met. The qualitative aspects first and foremost required
the artifact to, to the greatest extent achievable, be seam-
lessly embedded into the visual identity of the platform.
It should also be an adequate fit with the existing soft-
ware code-base and follow company code conventions.
3.4 Design and development
This was an iterative design and development process
with, arguably, three created artifacts.
The desired artifact was to be an interactive feature tour,
utilizing lightboxes (modals), tooltips and progress indi-
cators, to visually guide the users, instructing them how
to set up their account in order to participate in tourna-
ments in a game of their choice. Contextually sensitive
feature introductions like these are frequently argued by
researchers to have greater potential for successful user
onboarding than static or external instructions [2, 3]. The
artifact should also be reusable, in the sense that it
should be designed and implemented in such a way that
it can be utilized for teaching and introducing different
features in future iterations.
Three different implementations were explored. The
first two artifacts (A1 & A2) were concluded as unsatis-
factory solutions, while the third and last (A3) was con-
sidered satisfactory.
3.4.1 Artifact 1 (A1)
A1 was implemented using a third-party SaaS, namely In-
tercom’s product tours. This almost entirely pre-built so-
lution left much to be desired when it came to flexibility
and options for customization and was later disregarded,
as a heuristic evaluation concluded it was unable to be
seamlessly embedded into the platform's established
visual identity.
3.4.2 Artifact 2 (A2)
A2 was developed using the third-party and open-
sourced JavaScript library, React-Joyride. While it proved
more customizable than the first iteration of the solution,
it was mutually agreed that the library’s fit with the plat-
form codebase rendered it suboptimal.
3.4.3 Artifact 3 (A3)
The final iteration of the feature tour, A3, was developed
from scratch using JavaScript (ECMAScript 2019) &
TypeScript (Version 3.8), React.js (Versions 16.13.0 &
16.13.1) and .NET (.NET Core 3.1) for the backend.
3.5 Demonstration
The solution was demonstrated to- and tested by the em-
ployees at The Company as part of the later stages of the
iterative development process. Feedback was collected
and the solution revisited and improved on. Some feed-
back, while valuable and true, was decided to be outside
of the scope of the study - and was thus taken note of as
potential future improvements on the work.
3.6 Evaluation
In order to evaluate the implications of a feature tour on
The Platform, a contextually aware and interactive fea-
ture introduction was developed. After deployment to
the live production environment, a subset (n=2000+) of
randomly selected users were presented with the choice
of being shown the feature introduction for participating
in tournaments or opting out of it. User data from inter-
action with the implemented artifact was collected over
a span of a couple of weeks and stored in an anonymized
database. Collected data consisted of an anonymized
user identification code, timestamps of interaction as
well as a code representing which step of the feature tour
the interaction belonged to. Meanwhile, some data were
collected from a subset of users not being offered the fea-
ture introduction as well. This data included unassisted
activation rates for the same tasks that the feature intro-
duction aspired to improve during the same period of
time, namely the rates at which new users set up their
account in order to be able to participate in tournaments.
A Welch’s t-test was performed to test if the mean acti-
vation times of the independent sample groups signifi-
cantly differed from one another. The timestamps of the
different steps were analyzed in order to better under-
stand user behavior within the feature tour and for iden-
tifying key parts where improvements could be made.
4. RESULTS
In this results section of the study, the users that received
the step-by-step feature introduction will be referred to
as Students. This is not to be confused with the Student’s
t-test [31], which is referenced in a paragraph, which is
the common name for a statistical test. The users not re-
ceiving the feature introduction will be referred to as
NonStudents.
4.1 Data collection
Data concerning the activation rates of more than 25000
newly registered users was collected. The two independ-
ent sample groups were made up of users not being of-
fered the feature introduction, NonStudents and users
given the option of receiving the interactive feature in-
troduction, referred to as Students. Activation in this in-
stance was defined as a user having successfully con-
nected their account on the platform, with a game ac-
count of their choice. The data cases in the respective
groups where users did manage to fulfill the activation
requirements (N=20551 for NonStudents, N=1926 for
Students) were used for further analysis in section 4.5
and onward.
Table 1: Data collected from newly registered users that completed the activation requirements.
Data Collection Cases total
NonStudent 20551
Student 1926
4.2 Acceptance rates for the optional feature introduc-
tion
More than three-fourths (78.33%) of users presented
with the option (See Figure 2) to get the feature introduc-
tion accepted it. 21.27% therefore opted out of the user
onboarding procedure for demonstrating how to con-
nect a user’s account on The Platform with a desired
game account.
4.3 Activation rates of students versus normal users
30.95% of newly registered users, not offered the feature
introduction ended up successfully fulfilling the activa-
tion requirements on their own during the span of the
study’s data collection phase. In comparison, 40.54% of
users that were offered the feature introduction did ful-
fill the requirements. If one looks at only the users that
accepted the feature introduction, this number is instead
54.07%.
4.4 Completion rates of the feature introduction steps
4.4.1 Welcome prompt (Step One)
After having signed up for an account on the PaaS, and
finding themselves on the platform’s landing page, users
were prompted with a modal. The modal (See Fig. 2)
asked the user whether they would like guidance on the
steps necessary in order to participate in tournaments
on the platform. 78.33% accepted and 21.27% declined.
Figure 2. The welcome modal where users could opt-in or opt-out of the feature introduction.
4.4.2 Games (Step 2)
97.02% of the users accepting the onboarding also com-
pleted the second step of the tour (See Fig. 3), in other
words they followed the tooltip and navigated to their
game of choice. The remainder navigated elsewhere on
the platform or left the site.
Figure 3. A tooltip with some instructions for the user, pointing at a list of available games in the sidebar. The tooltip is paired with a progress indicator with three
steps.
4.4.3 Connect Account (Step 3)
72.74% of the total tested users made it through the
third step as well. Meaning they followed the tooltip and
pressed the connect account button, shown in Figure 4.
This took them to a modal with instructions on how to
connect the accounts, paired with instructions for con-
necting to that specific game.
Figure 4. A tooltip pointing at a button. The button con-tent is different for different games and so is the content
of the modal it triggers.
4.4.4 Browse Tournaments (Step 4)
The final step in the feature introduction required users
to successfully login with the credentials for the external
account, and thus connecting their account on the PaaS
with their desired game account (See Fig. 5). If they suc-
ceeded in doing so, they were presented with a tooltip
highlighting a tab, which when clicked allowed the users
to browse the different listed tournaments for that game.
This last step was completed by 54.07% of users that
started the onboarding process for tournaments.
Figure 5. After successfully connecting a game account, this tooltip directs users to a tab with upcoming tourna-
ments they can sign up for.
Figure 6. Completion banner for the feature introduction.
4.5 Deep dive into comparisons of Students and NonStu-
dents with respect to activation times
In order to answer the question of how the feature intro-
duction affected user activation times for newly regis-
tered users, the activation times for the Students and
NonStudents that did fulfill the activation requirements
were examined. Time is presented either in seconds, the
MM:SS or the HH:MM:SS format, representing hours,
minutes and seconds respectively. The group of users
that did not receive the feature introduction but com-
pleted activation, the non-students (N=20551) was asso-
ciated with an activation time of M=06:35:25
(SD=30:39:17) and MED=00:03:11. The corresponding
value for the student group (N=1926) is the numerically
smaller values of M=04:42:22 (SD=26:29:16) and
MED=00:02:11.
The drastically different mean and median values are in-
dicative of highly skewed data, with the mean being
higher than the median resulting in a positively, or right-
skewed distribution. This results in the median being the
better and more robust indicator of a central tendency
and is thus better suited for characterizing the distribu-
tion of both samples. Furthermore, it is evident that ex-
treme outliers are present in the data, likely the cause of
the skew, and a resulting Skewness value of 8,398 and a
Kurtosis of 80,712 can be observed from a descriptive
analysis performed in SPSS (IBM SPSS Statistics for Win-
dows, version 26.0. Armonk, NY: IBM Corp.). Plotting the
distribution using bar graphs and visually inspecting it,
although difficult without zooming, indicates an approx-
imately exponential distribution of activation times.
4.6 Handling Outliers
Tukey introduced the interquartile range (IQR) multi-
plier approach [30] to detecting outliers in data sets,
which is the default one used in the IBM SPSS Statistics
software. This method of identifying and eliminating out-
liers suggests using a factor of 1.5 times the IQR and dis-
regarding or transforming data points which fall out of
this range. This does in fact produce a much more nor-
mal-like distribution, even if it in this case creates what
resembles a positive-side amputee normal distribution
[13]. However, there are studies indicating that this ap-
proach is inaccurate approximately 50% of the time and
instead suggests using a multiplier of 2.2 [16]. There-
fore, in order to be on the safer side, a more careful ap-
proximate multiplier of 2.64 on the IQR for the median of
the group with the largest range, which in this case is the
non-student group (IQR=00:10:10), is therefore chosen
to detect extreme outliers. This conveniently ends up
making 30 minutes the upper limit for the time taken to
connect an account after having signed up on the plat-
form. Paired with the heuristic assumption that it is
highly uncommon for a user to spend more than 30
minutes in succession trying to accomplish the steps re-
quired to meet the activation requirements. As can be ob-
served by the descriptive statistics of the dataset ridden
of the identified outliers in Table 3, the values of the
mean and standard deviation are significantly dimin-
ished by this process, while the median values were rel-
atively unaffected in comparison.
Figure 7 & 8. The figures illustrate the sample distributions for NonStudents and Students after outlier reduction. The horizontal axes hold the time delta from platform signup to account pairing with a game account of the user’s choice.
Table 2. Descriptive statistics of the independent samples after outlier removal, using the IQR multiplier approach
with a multiplier of 2.64 [30].
Statistics Std. Error
NonStudents N 16843
Mean 04:15 00:02
Median 02:24
Std. Dev. 05:04
Min 00:08
Max 29:59
IQR 03:40
Skewness 2.491 .019
Kurtosis 6.713 .038
Students N 1669
Mean 03:27 00:06
Median 01:47
Std. Dev. 04:37
Min 00:12
Max 29:54
IQR 02:44
Skewness 3.068 .060
Kurtosis 10.731 .120
4.7 Resulting differences
In order to test the hypothesis that the students and non-
students were associated with statistically significantly
different mean activation times, an independent sample
Welch’s t-test was performed. As can be observed in Ta-
ble 4 for the Levene’s test for equality of variances, the
significance value much lower than 0.05 means that we
can reject the null hypothesis that the samples are of
equal variances, and therefore opt for Welch’s t-test, due
to it not assuming homogeneity of variances. This can be
observed in Table 2 as well, considering that Levene’s
test for equality of variances becomes increasingly sen-
sitive to differences in variances with unequally propor-
tioned sample sizes.
Table 3. Levene’s test for equality of variances
F Sig.
Equal variances assumed 28.98 .000
The student and non-student groups are considered suf-
ficiently normally distributed, for the purpose of con-
ducting the Welch’s t-test (i.e., N of smallest sample =
1669, max skew ≤ |3.068| and max kurtosis ≤ |10,731|).
The independent samples Welch’s t-test was associated
with a statistically significant effect, t(2087.372)=6.72, p
= .000. Thus, with a p << .050, the non-students were as-
sociated with a statistically significantly longer mean
activation time than the students who received the step-
by-step onboarding procedure. Cohen’s d was estimated
at 0.17, which is a small effect based on Cohen’s [9]
guidelines.
Table 4. Descriptive statistics of the independent samples after outlier removal.
N Mean Std. Dev. Std. Error Mean
NonStudent 16843 04:15 05:04 00:02
Student 1669 03:27 04:37 00:06
Table 5. Welch’s t-test for Equality of Means.
T Df Sig (2-
tailed)
Mean
Diff.
Eq. Variances
not assumed
6.715 2087.372 .000 00:48
In an attempt to be especially thorough, a logarithmic
transformation of the dependent time values was carried
out, as to make sure not to have violated Welch's t-test’s
assumption of normality within the groups. The transfor-
mation successfully translated the positively skewed
data to better conform to the approximate normal distri-
bution. The results can be observed in Figure 9 and Fig-
ure 10.
Figure 9. Histogram of logarithmically transformed Non-Student’s sample distribution.
Figure 10. Histogram of logarithmically transformed Stu-dent’s sample distribution.
For the logarithmically transformed data, Levene’s Test
for equality of variances tells us that the variances are
now sufficiently statistically equal for a standard Stu-
dent’s t-test. For the sake of consistency and them pro-
ducing next to identical outcomes in this case, the argua-
bly more robust [11] Welch’s t-test is reused. The stu-
dent and non-student groups are considered sufficiently
normal, even more decidedly than in the previous in-
stance, for the purpose of conducting the Welch’s t-test
(i.e., N for smallest independent sample group = 1669,
skew << |2.0 | and kurtosis << |9.0|, [27]). This yielded
similar results to the previous test. The independent
samples Welch’s t-test was once again associated with a
statistically significant effect, t(2020.122)=9.343,
p=.000. Thus, with a p << .050, the non-students were
again associated with a statistically significantly longer
mean activation time than the students who received the
step-by-step onboarding procedure.
4.8 How was the feature introduction interacted with?
This part of the analysis was conducted on a subset of the
users that received the feature tour (n=783) after disre-
garding outliers. The reason for not including all users
having received the feature introduction, was firstly due
to the data being delivered in two batches, where an
analysis was conducted before having received the sec-
ond batch of data. Secondly, due to an illegitimate as-
sumption that all newly registered users would always
initially be routed to the landing page, which is where the
first step of the tutorial was set to be triggered. Thus, a
small portion of the newly registered users ended up not
being given the contextual feature tour in the right order.
This also meant that this fragment of users did not get
the immediate option of opting out of the feature tour,
which is why this subset, where such a pattern was ob-
served, was excluded from this part of the analysis. With
a significant difference in mean time spent before reach-
ing the activation requirements already concluded, a de-
cision was also made to place an upper limit of twelve
minutes for completion of the four steps of the feature
introduction, in order for a user to be included in this
analysis.
Figure 11. The mean and median times spent between dif-ferent steps of the feature introduction. The label 1 repre-sents time spent between steps One and Two. The label 2, between Two and Three, and the label 3, between Three
and Four.
As shown in Figure 11, the shortest mean and median
times for completing a step, was by a compelling margin,
the time from having accepted the tutorial to completing
the second step (Figure 2). A considerable gap can be ob-
served with regards to the mean time relative to the next
group, labeled 2, which represents the times spent be-
tween steps two and three (Figure 3). Lastly, the mean
and median times spent between the last two steps of the
tutorial are significantly longer than the rest, M=192.5s,
MED=60s.
5. DISCUSSION
This study set out to investigate how the implementation
of a step-by-step feature introduction affects user activa-
tion rates on a platform with a complex interface. The re-
sults of the study indicate that step-by-step onboarding
procedures do indeed reduce the mean time spent
completing the steps that users would otherwise have to
spend time exploring and finding on their own.
5.1 Users not completing a step
Users not completing a step does not necessarily mean
that the step did not accomplish what it intended to. This
is assuming the goal was to teach the user how to do
something, rather than making them do something. A
step in the feature introduction was set as complete once
the user clicked the element, or one of the elements, that
the contextual tooltip pointed at. If the user instead
clicked the close symbol on the tooltip, they were still
considered to have taken part of and completed that step,
as the tooltip had still fulfilled its purpose of informing
the user where to find something or how to get to the
next step, even if they decided not to navigate there right
away.
5.2 Entering credentials for their game account
The third to fourth step in the feature introduction was
perhaps to no surprise, the part where the most users
dropped off and the average and median times spent was
significantly longer. This could be due to user attention
dropping off after a set number of steps in a feature in-
troduction, as implied by Trychameleon [6]. However, it
could also be due to several other factors. It was an ob-
jectively more complex step, in which users had to recall
their credentials for the game account in question, enter
it correctly, consider the trustworthiness of the proce-
dure and much more. Which is substantially different
from the “click here, click there” nature of the three pre-
ceding steps. One takeaway from this is however that
feature introductions, like the one implemented in this
study, allows for a more thorough evaluation of how us-
ers interact with a feature. One could argue that this has
provided new insight on the pain-points of the activation
goal set by the company, which in turn can be used to im-
prove that user experience.
5.3 Different users, different goals
As concluded in the pre-study and briefly touched on in
the introduction, there exists a lot of different users with
different goals on the platform [24]. A portion of users
likely uses the Challengermode platform for other pur-
poses than participating in tournaments. This effectively
renders the feature introduction implemented less fruit-
ful, imaginably even useless, for those users. This
strengthens the argument for contextual sensitivity in
onboarding procedures. It also strengthens the
argument for allowing the onboarding procedures to be
optionally triggered. Thus, an opt-in centered user expe-
rience design approach, rather than opt-out, should be
adopted for the sake of the users.
A general recommendation sprung from this is thus to
implement a checklist of sorts for the user to complete
on the platforms or apps that are presented by complex
interfaces. The checklist should either be filled with tasks
to complete dependent on some preferences set by the
users during their account registration on the platform,
alternatively with a general set of tasks that should be of
interest to all users. The critical part is that the checklist
allows users to at their own pace and if they want to, trig-
ger a feature introduction for the selected feature or task.
It is however apparent that the features considered the
most central and critical for a platform should be the
most streamlined ones. Occasionally, instead of figuring
out how to best shepherd users where the application
wants them, it might be an equally valuable approach to
rethink and reevaluate the steps required to reach that
goal, in order to make them as visible, accessible and ef-
fortless as possible.
5.4 Method Discussion
The study and its method are mostly of a quantitative na-
ture, where data on observed user behaviors were col-
lected and analyzed using statistical methods. The bene-
fit of this, coupled with the large sample sizes, is the abil-
ity for actual substantiation and verification of general
hypotheses about feature introductions. Limitations of
the study were for example, the sorts of data that were
collected in the data collection phase. While more data
on the users, e.g. preferred genre of games, gender or age,
would allow for deeper analysis and hopefully other in-
teresting findings – there is also the aspect of user integ-
rity that needs to be considered.
Illegitimate assumptions of user and technology behav-
ior were touched on in paragraph 4.8, where some edge
user cases not accounted for, led to fragments of (for its
intended purpose) unusable data. Fortunately, the large
sample sizes made the user data not used negligible.
The pandemic outbreak of COVID-19 did affect and delay
the study, mostly due to office spaces being closed down,
restricting access to development environments, hard-
ware resources and colleagues. Fortunately, the user
data collection remained unaffected, although delayed.
5.4.1 Selection Bias
There are grounds to believe that a sampling bias may
exist where students partaking in the feature introduc-
tion are more inclined to ‘activate’, than the NonStu-
dents. This could explain the different mean and median
values observed and in turn lessen the external rele-
vancy of the study’s findings [21]. However, it should be
noted that comparisons in the results section 4.5 of the
study and onward, were only made with NonStudents
that, no matter opting out of the feature introduction,
eventually did meet the activation requirements. In
other words, they did still end up connecting a game ac-
count within the time frame where the A/B-testing was
conducted.
6. FUTURE WORK
An interesting extension of this study would be to inves-
tigate how a successful first visit on a platform, affects
the user's motivation to return to the platform. It is not
difficult to imagine that there could be benefits to letting
users figure out how to do tasks themselves, through ex-
ploration of the platform, something also worth consid-
ering for future studies. This would likely require the
study to run over a longer time, while perhaps also incor-
porating qualitative user data. Studying the users that
did opt out of the onboarding but completed the steps on
their own, is also a potential topic for an extension of this
study.
Differences that can be observed in user behavior, de-
pending on demographic information could also be im-
mensely interesting. One could try to figure out if it is
possible to successfully and sufficiently accurately pre-
dict which users visit the platform for what purpose.
Thus, creating a great incentive for deeper and smarter
personalization of the user onboarding experience on an
individual level. Incorporating some sort of qualitative
part is likely beneficial for most quantitative studies, as
they provide greater depth and understanding as to why
something turned out the way it did, which this study
probably also would benefit from.
If this study in particular were to be extended on, I would
highly suggest dwelling deeper into the areas of gamifi-
cation and user psychology which are areas tightly con-
nected to the one of user onboarding, even though this
study in particular focused primarily of filling the gaps of
non-commercially available hard-data on user onboard-
ing implications.
7. CONCLUSION
A good number of users chose to partake in the feature
introduction and out of those, 50 percent ended up com-
pleting it. Users that received the step-by-step feature in-
troduction were associated with statistically signifi-
cantly lower mean times spent completing the activation
requirements set up by the company, than users that
were left to figure out the necessary steps on their own.
Additionally, users partaking in the feature introduction
had higher overall activation rates. The implemented ar-
tifact also provided insight into the pain-points associ-
ated with the activation goal, as well as how the user’s
time was distributed on the different sub-steps of the
onboarding procedure.
8. REFERENCES
[1] E. Andersen et al., “The Impact of Tutorials on
Games of Varying Complexity,” in Proceedings of
the SIGCHI Conference on Human Factors in Com-
puting Systems, 2012, pp. 59–68, doi:
10.1145/2207676.2207687.
[2] K. Balboni, “We categorized over 500 user
onboarding experiences into 8 UI/UX patterns,”
2020. https://www.appcues.com/blog/user-
onboarding-ui-ux-patterns (accessed Jun. 19,
2020).
[3] T. N. Bauer and B. Erdogan, “Organizational so-
cialization: The effective onboarding of new em-
ployees.,” in APA handbook of industrial and or-
ganizational psychology, Vol 3: Maintaining, ex-
panding, and contracting the organization., Wash-
ington, DC, US: American Psychological Associa-
tion, 2011, pp. 51–64.
[4] A. Benlian, M. Koufaris, and T. Hess, The role of
SAAS service quality for continued SAAS use: Em-
pirical insights from SAAS using firms. 2010.
[5] S. Camacho Herrero, “Gamified Learning of Soft-
ware Tool Functionality : Design and implemen-
tation of an interactive in-app learning interface
for complex software,” 2019.
[6] Chameleon, “Chameleon product tour benchmark
report 2019,” 2019. [Online]. Available:
https://www.trychameleon.com/assets/chame-
leon-product-tour-benchmarks-report-2019.pdf.
[7] P. K. Chilana, A. J. Ko, and J. O. Wobbrock, “Lemon-
Aid: Selection-Based Crowdsourced Contextual
Help for Web Applications,” in Proceedings of the
SIGCHI Conference on Human Factors in Compu-
ting Systems, 2012, pp. 1549–1558, doi:
10.1145/2207676.2208620.
[8] J. Clement, “Mobile application user retention rate
worldwide from 2012 to 2019,” 2020. [Online].
Available: https://www.statista.com/statis-
tics/751532/worldwide-application-user-reten-
tion-rate/.
[9] J. Cohen, “A power primer.,” Psychological bulle-
tin, vol. 112, no. 1, pp. 155–159, Jul. 1992, doi:
10.1037//0033-2909.112.1.155.
[10] A. Cooper, About Face: The Essentials of User In-
terface Design, 1st ed. USA: John Wiley & Sons,
Inc., 1995.
[11] C. Delacre, Marie; Lakens, Daniel; Leys, “Why
Psychologists Should by Default Use Welch’s t-
test Instead of Student’s t-test,” International Re-
view of Social Psychology, 2017, doi:
10.31219/osf.io/sbp6k.
[12] A. Fedorov, “An intro to user onboarding, part 1,”
2015. https://www.invisionapp.com/inside-de-
sign/an_intro_to_user_onboarding_part_1/ (ac-
cessed Jun. 12, 2020).
[13] G. Gignac, “Outliers Do’s and Dont’s,” 2011.
http://www.how2stats.net/2011/09/outliers-
dos-and-donts.html.
[14] T. Grossman, G. Fitzmaurice, and R. Attar, “A Sur-
vey of Software Learnability: Metrics, Methodol-
ogies and Guidelines,” in Proceedings of the
SIGCHI Conference on Human Factors in Compu-
ting Systems, 2009, pp. 649–658, doi:
10.1145/1518701.1518803.
[15] A. Hevner et al., “Design Science in Information
Systems Research,” Management Information
Systems Quarterly, vol. 28, pp. 75-, 2004.
[16] B. Hoaglin D. C, Iglewicz, “Fine-tuning some re-
sistant rules for outlier labeling,” Journal of the
American Statistical Association, 1987.
[17] S. Hulick, The Elements of User Onboarding.
useronboard.com, 2014.
[18] C. Hulick, Samuel; Traynor, Des; Galavan Ruairí ;
Colman, Jonathon; Allan, Robbie; Swanser, Allan;
Irvine-Broque, Brendan; Yang, Siya; Chang, Inter-
com on Onboarding, 2nd ed. Intercom, 2019.
[19] J. Jenkins, “Best practices for using Intercom’s
Product Tours,” 2020. https://www.inter-
com.com/help/en/articles/3095688-best-prac-
tices-for-using-intercom-s-product-tours.
[20] R. Larsson, “A user centric approach for evaluat-
ing and enhancing the usability of a complex real-
time web user interface,” 2016.
[21] J. G. Lynch Jr., “On the External Validity of Exper-
iments in Consumer Research,” Journal of Con-
sumer Research, vol. 9, no. 3, pp. 225–239, Dec.
1982, doi: 10.1086/208919.
[22] M. Mehta, Nick; Steinman, Dan; Murphy, Lincoln;
Martinez, Customer Success: How Innovative Com-
panies Are Reducing Churn and Growing Recur-
ring Revenue, 1st ed. Wiley, 2016.
[23] L. Murphy, “The Secret to Successful Customer
Onboarding,” 2016. https://sixteenven-
tures.com/customer-onboarding (accessed Jun.
15, 2020).
[24] S. O degaard Jacobsson, “Deepening User Engage-
ment on an Esports Platform Using Gamification :
A Multi-Conceptual Study,” 2019.
[25] K. Peffers, T. Tuunanen, M. Rothenberger, and S.
Chatterjee, “A Design Science Research Method-
ology for Information Systems Research,” J.
Manage. Inf. Syst., vol. 24, no. 3, pp. 45–77, Dec.
2007, doi: 10.2753/MIS0742-1222240302.
[26] M. Rettig, “Nobody Reads Documentation,” Com-
mun. ACM, vol. 34, no. 7, pp. 19–24, 1991, doi:
10.1145/105783.105788.
[27] E. Schmider, M. Ziegler, E. Danay, L. Beyer, and M.
Buhner, “Is it really robust? Reinvestigating the
robustness of ANOVA against the normal distri-
bution assumption,” Meth Eur J Res Meth Behav
Soc Sci, vol. 6, pp. 15–147, 2010.
[28] J. Sjo berg and H. Winba ck, “Designing the user
onboarding process in analytics software Taking
an omnichannel perspective,” 2017.
[29] A. Snell, “Researching onboarding best practice:
Using research to connect onboarding processes
with employee satisfaction,” Strategic HR Review,
vol. 5, pp. 32–35, 2006, doi:
10.1108/14754390680000925.
[30] J. W. Tukey, “Performance of Some Resistant
Rules for Outlier Labeling,” Hoaglin 1986 Perfor-
mance OS, 1986, doi:
10.1080/01621459.1986.10478363.
[31] R. E. Walpole and H. Raymond. Myers, Probability
& statistics for engineers & scientists. New Delhi:
Pearson, 2006.
[32] C. J. Welty, “Usage of and Satisfaction with Online
Help vs. Search Engines for Aid in Software Use,”
in Proceedings of the 29th ACM International Con-
ference on Design of Communication, 2011, pp.
203–210, doi: 10.1145/2038476.2038516.
www.kth.se
TRITA -EECS-EX-2020:633