Vol.:(0123456789)

J. Comput. Educ. (2021) 8(1):1–32https://doi.org/10.1007/s40692-020-00169-2

1 3

Immersive virtual reality as a pedagogical tool in education: a systematic literature review of quantitative learning outcomes and experimental design

D. Hamilton1  · J. McKechnie1 · E. Edgerton1  · C. Wilson1

Received: 21 October 2019 / Revised: 16 June 2020 / Accepted: 2 July 2020 / Published online: 11 July 2020 © The Author(s) 2020

Abstract The adoption of immersive virtual reality (I-VR) as a pedagogical method in education has challenged the conceptual definition of what constitutes a learning environment. High fidelity graphics and immersive content using head-mounted-displays (HMD) have allowed students to explore complex subjects in a way that traditional teaching methods cannot. Despite this, research focusing on learning outcomes, intervention characteristics, and assessment measures associ-ated with I-VR use has been sparse. To explore this, the current systematic review examined experimental studies published since 2013, where quantitative learning outcomes using HMD based I-VR were compared with less immersive pedagogi-cal methods such as desktop computers and slideshows. A literature search yielded 29 publications that were deemed suitable for inclusion. Included papers were quality assessed using the Medical Education Research Study Quality Instrument (MERSQI). Most studies found a significant advantage of utilising I-VR in educa-tion, whilst a smaller number found no significant differences in attainment level regardless of whether I-VR or non-immersive methods were utilised. Only two stud-ies found clear detrimental effects of using I-VR. However, most studies used short interventions, did not examine information retention, and were focused mainly on the teaching of scientific topics such as biology or physics. In addition, the MERSQI

* D. Hamilton david.hamilton@uws.ac.uk

J. McKechnie jim.mckechnie@uws.ac.uk

E. Edgerton edward.edgerton@uws.ac.uk

C. Wilson claire.wilson@uws.ac.uk

1 Division of Psychology, School of Education & Social Sciences, University of the West of Scotland, Paisley, UK

2 J. Comput. Educ. (2021) 8(1):1–32

1 3

showed that the methods used to evaluate learning outcomes are often inadequate and this may affect the interpretation of I-VR’s utility. The review highlights that a rigorous methodological approach through the identification of appropriate assess-ment measures, intervention characteristics, and learning outcomes is essential to understanding the potential of I-VR as a pedagogical method.

Keywords Virtual reality · Head mounted displays · Education · Learning · Simulations

Introduction

The increasing financial feasibility of virtual reality (VR) has allowed for educational institutions to incorporate the technology into their teaching. According to research, 96% of universities and 79% of colleges in the UK are now utilising augmented or virtual reality in some capacity (UKAuthority 2019). In addition, the rising power of personal computers and associated hardware has led to a revolution in graphical fidelity, with ever more complex and realistic simulations and virtual worlds (Slater 2018). As Dickey (2005) alludes to, this has both challenged and expanded the very conceptual definition of what is defined as a learning environment. Where once this would have been restricted to classroom teaching or field trips, VR’s innate ability to give users a sense of presence and immersion has opened new possibilities in educa-tion if implemented appropriately (Häfner et al. 2018).

The use of technology-aided education as a pedagogical method is not a modern phenomenon, and investigations into its utility have been studied for almost half a cen-tury. As far back as the 1970s, Ellinger and Frankland (1976) found that the use of early computers to teach economic principles produced comparative learning outcomes with traditional didactic methods such as lectures. However, as Jensen and Konradsen (2018) allude to, it was with the release of the Oculus Rift in 2013 that VR became synonymous with head-mounted-display (HMD) based VR. This had several ramifica-tions. First, HMDs became economically feasible for consumers and educational insti-tutions to acquire en masse, due to a significant drop in price (Hodgson et al. 2015). As Olmos et al. (2018) remarks, the economic viability of VR has tackled one of the main entry barriers to adopting the technology. And secondly, academic research into the potential benefits of I-VR in education starts to expand, as well as its applied use in pedagogical settings (Hodgson et al. 2019). One of VR’s most important contribu-tions to education is that it has allowed students to repeatedly practice complex and demanding tasks in a safe environment. This is particularly true of procedural tasks such as surgical operations or dental procedures that cannot be carried out for real until a certain level of competency has been achieved (Alaraj et al. 2011; Larsen et al. 2012). Additionally, VR has allowed for students to gain cognitive skills by way of experi-ential learning, such as exposing them to environments that would be too logistically problematic to visit in reality (Çalişkan 2011). For instance, by using a HMD, Bailen-son et al. (2018) were able to expose students to an underwater environment to facilitate learning about climate change. VR has made an important contribution to education in

3

1 3

J. Comput. Educ. (2021) 8(1):1–32

that it has allowed for students to directly experience environments or situations that are difficult to replicate by using traditional teaching methods such as lectures, slideshows, or 2D videos.

A concise definition of VR’s key characteristics is challenging due to the ever-changing nature of the technology. However, Sherman and Craig (2003) proposed that there are a number of constituent elements that must underpin the VR experience, ultimately leading to the life-like perception of the virtual environment. These include the necessity for VR to be immersive, in that the participant’s own cognitive faculties produce a sense of being present and involved in the virtual space, often with reduced awareness of what is happening in the real-world around them. Additionally, the virtual space should offer a degree of interactivity, in that the user can manipulate the environ-ment and test variables. This can include interacting with objects, virtual avatars, or even collaborating with other real-life users within the computer-generated space.

Definition of key terms

Due to the multidisciplinary nature of VR research and its pedagogical applications, it is important to define key terms used. VR can broadly be broken down into two main categories: desktop VR (D-VR), and immersive-VR (I-VR). D-VR is typically classi-fied as non-immersive, in that a headset is not used, and the participant will be control-ling and manipulating the virtual environment on a computer screen with traditional keyboard and mouse hardware (Lee et al. 2010). On the other hand, I-VR is typically multi-modal in nature by providing a sense of immersion in the environment through 360° visuals by aid of a HMD, auditory stimulation through the use of earphones, and increasingly the proprioception of limbs by way of controllers and tracking (Freina and Ott 2015; Howard-Jones et al. 2015; Murcia-López and Steed 2016). Although there are a range of HMDs on the market, from high-end hardware like the HTC Vive, to via-ble low-cost options like the Google Cardboard, they all utilise the same core principals of operation (Brown and Green 2016). Typically, a HMD will feature a set of embed-ded liquid crystal displays (LCD) which will present each eye an image from a slightly different angle. This mimics natural optic function by allowing the wearer to view a ste-reoscopic image complete with depth perception and a wide field of view. Mobile VR headsets can achieve the same effect using a single display by dividing the screen down the middle and presenting each half to the corresponding eye. Therefore, the current review defines a HMD as a device worn over the head, which provides a stereoscopic computer-generated or 360° video image to the user. This includes tethered (connected to a computer), stand-alone (no computer needed), or mobile VR headsets (mobile/cell phone connected to a HMD).

Previous literature and reviews

There have been a number of systematic reviews that have previously explored the relationship between VR and pedagogical attainment. Lee (1999) reviewed 19 stud-ies from as far back as 1976 and found that 66% of students in simulation groups outperformed those in their respective control groups. However, this review did not

4 J. Comput. Educ. (2021) 8(1):1–32

1 3

focus exclusively on an educational level or age range, so featured both young kin-dergarten children, as well as higher education students. As a result, the generalis-ability of VR’s effectiveness as a pedagogical method is difficult to ascertain, with significant differences in age, task difficulty, and applications. Furthermore, all the studies are dated in terms of the technology utilised and feature early D-VR pro-grammes and rudimental computer simulations. This early technology may be prim-itive when compared with the high-fidelity graphics and immersive components of contemporary technology. Nevertheless, these early studies do help exemplify that the use of technology in education is not a new concept, and computer-based simu-lations have long been employed as a way of facilitating learning.

A more recent analysis was undertaken by Merchant et al. (2014), and looked at three specific sub-categories of VR: games, simulations, and virtual worlds. Games give the actor autonomy and freedom to move around the virtual world, testing hypotheses, achieving goals, and eliciting motivation and learning through immer-sion (Gee 2004). Simulations attempt to recreate a real-world environment that can help facilitate learning by allowing for the testing of variables and resulting out-comes. Finally, virtual worlds can provide an immersive or non-immersive sense of presence in a three-dimensional (3D) world, and the ability to manipulate, inter-act, or construct objects. Furthermore, virtual worlds can give the opportunity for multiple users to interact with one another within the digital environment (Dickey 2005). The meta-analysis showed that although game-based VR produced the high-est learning outcomes, simulations and virtual worlds were also effective at increas-ing educational attainment. Once again, the limitation of this review is that it did not restrict its analysis to exclusively one domain of education. Although higher educa-tion made up the greatest number of studies, research from elementary and middle school were also included in the analysis.

One of the most recent systematic reviews to look exclusively at I-VR through the utilisation of HMDs was carried out by Jensen and Konradsen (2018). In their comprehensive search of existing literature published between 2013 and 2017, the review identified 21 quantitative and qualitative papers that focused on both learn-ing outcomes in I-VR, and subjective attitudes and experiences on the part of the user. The review found limited effectiveness of HMD in the acquisition of cognitive, psychomotor, and affective skills when compared with less immersive technologies. However, Jensen and Konradsen (2018) did highlight the relatively low quality of studies included as a concern, and this may impede the ability to draw firm conclu-sions about the educational utility of I-VR.

Rationale for review

There are several fundamental reasons that necessitate an updated assessment of the topic area, such as the increase in relevant published literature, as well as the nar-row scope of previous reviews. The last major review looking at I-VR and HMDs as an educational tool was carried out by Jensen and Konradsen (2018), with the most recent studies featured in that paper being published in 2016. Since then, there has been a significant increase in relevant published literature, with > 70% of the

5

1 3

J. Comput. Educ. (2021) 8(1):1–32

papers included in the current review being published since 2017, and therefore not included in the previous systematic review. Additionally, unlike previous reviews, the current examination of I-VR’s pedagogical utility focuses exclusively on studies where I-VR is directly compared to a less immersive method of learning. As a result, the current paper is able to highlight not only whether I-VR is an effective medium, but also whether it is more effective when compared to alternative methods. Addi-tionally, no other systematic review looking at I-VR and HMDs has had a particular focus on the experimental design, assessment measures, and intervention character-istics of the included studies. The review also addresses the underlying methodology of the included studies, to offer an understanding of how I-VR is being employed in experimental literature. Based upon the findings of previous studies as well as areas yet to be sufficiently explored, this paper has a number of core research questions:

• To assess the subject area, discipline, and learning domain that I-VR has been employed in.

• Understand where I-VR confers an educational benefit in terms of quantitative learning outcomes over non-immersive and traditional teaching methods.

• To examine the experimental design of studies, focusing on how learning out-comes are assessed, and how the I-VR intervention is delivered.

• To inform future experimental and applied practice in the field of pedagogical I-VR application.

Methodology

Search strategy

The current systematic review included peer-reviewed journal articles and confer-ence proceedings that passed all the inclusion criteria detailed. An initial scoping review identified seven databases that could be utilised in a comprehensive litera-ture review, as well as associated keywords and search terms. These included Web of Science (Core Collection), Science Direct, Sage, IEEE Xplore, EBSCO, Taylor & Francis, and Google Scholar. These databases encompass a mixture of general, social science, and technological literature.

Each of the seven databases was searched using a series of keywords based on the following Boolean logic string:

("Virtual Reality" OR "Virtual-Reality" OR “Immersive Virtual Reality” OR “Head Mounted Display” OR “Immersive Simulation”) AND (Education OR Training OR Learning OR Teaching)

Due to the scope and parameters of the research objectives, only peer-reviewed lit-erature published between January 2013 and December 2018 was included in the final review. Early access articles due to be published in 2019 were also included if these were found using the database searches. Date criteria was based upon an initial scoping review that found a substantial growth in relevant I-VR literature from 2013 onwards. A major contributing factor was the release of the Oculus Rift

6 J. Comput. Educ. (2021) 8(1):1–32

1 3

Development Kit 1 (DK-1) in early 2013, which is regarded as one of the first eco-nomically viable and high quality HMDs that could be used both within educational institutions, and at home (Lyne 2013).

The literature search across the databases yielded more than 12,000 references from a variety of sources. After the removal of duplicate records, 9,359 unique refer-ences were included for the title and abstract screening stage of the review.

Selection and screening

The open and general nature of the search string used led to a large number of refer-ences being returned for screening. As Jensen and Konradsen (2018) already alluded to in the last major review, VR research transcends various academic disciplines. The result is a lack of a clear taxonomy of definitions and terms. This means a wide net must be cast to ensure comprehensive capture of relevant material. This review defined I-VR as either a completely computer-generated environment, or the view-ing of captured 360° video through the use of a HMD. Studies that utilised sur-gical or dental simulators and trainers such as the da Vinci Surgical System, were excluded as these represent a separate domain of both technological and pedagogi-cal application. For example, surgical simulation based VR typically combines com-puter-generated visuals with simulated surgical tools, haptic feedback, and robotic components (Li et al. 2017). This type of technology would therefore not be appli-cable for general pedagogical application. Additionally, references were excluded if they: (1) focused on using I-VR as a rehabilitation or therapeutic tool; (2) were not in English; or (3) where the full-text was not available.

After title and abstract screening was performed, 197 references remained to be included in the full-text review. Each reference had to pass an inclusion flowchart based on each of the following criteria:

1. The population being sampled was from a high school, further or higher education establishment, or was an adult education student.

2. Population sampled did not have a developmental or neurological condition, nor could VR be used as a rehabilitation tool.

3. Paper described an experimental or quasi-experimental trial with at least one control group.

4. At least one group had to have undergone an educational HMD I-VR experience, and was compared with another group who underwent a non-immersive or tradi-tional pedagogical method of education (e.g. Desktop VR, PowerPoint, traditional lecture).

5. A quantitative and objective learning outcome such as tests scores, completion time, or knowledge retention was used to assess effectiveness.

After full-text screening, 29 references passed all stages and were included in the systematic review. See Fig. 1 for a summary of the selection process by stage.

Inter-rater reliability checks were conducted at the title and abstract screen-ing stage to assess the agreement of included studies. There were four individual

7

1 3

J. Comput. Educ. (2021) 8(1):1–32

evaluators that assessed the suitability of each reference based upon the inclusion criteria, which yielded an average agreement of 96%. Where any disagreement existed, the paper was discussed among all assessors until a unanimous decision was reached as to its suitability.

Quality assessment tool

To assess the quality of the studies, the Medical Education Research Study Qual-ity Instrument (MERSQI) was used (Reed et  al. 2007). Although this tool was primarily designed to examine the quality of studies in the field of medical edu-cation, it is in practice subject neutral. As the MERSQI assesses not only the quality of experimental design and outcomes measures, but also the assessment instrumentation used, it was viewed as a suitable and comprehensive tool for quality appraisal. In addition, the same instrument was used in a previously peer-reviewed systematic review examining VR, by Jensen and Konradsen (2018).

The MERSQI tool covers six quality assessment domains. These include: study design, sampling, type of data, validity of evaluation instrument, data analysis, and outcomes. Each domain is scored out of three, with a maximum overall score of 18. Unlike Jensen and Konradsen (2018), the current review gave full points in the study design category for experimental trials with participant randomisation, as well as appropriate pre-intervention measures. This decision was made as true randomised control trials featuring random sampling is unrealistic in I-VR peda-gogical research, as the participant sample can only be drawn from an educational establishment.

Papers identi�ied through initial

database search: N = 12,055

Papers remaining after duplicate records

were removed: N = 9,362

Papers remaing after title and abstract

screening: N = 197

Total number of papers remaining

after full-text screening: N = 29

Fig. 1 Stage-by-stage selection process

8 J. Comput. Educ. (2021) 8(1):1–32

1 3

Results

Quality of studies

The first domain examined for quality was the study design of the papers. There were 20 studies (69%) that featured an experimental design with stated random allocation of participants between control and experimental group. The review featured nine studies (31%) that were quasi-experimental in nature, meaning there was non-random allocation of participants into experimental groups.

Only one of the studies featured participants being studied at more than one institution, with most of the studies included (N = 28) only sampling from a sin-gle establishment. All studies produced response rates of over 75%, which means they were given the highest score in that domain.

In terms of the type of data presented, all included studies featured an objective measure of learning outcomes such as test scores or completion times. No studies used self-assessment on the part of the participant to gauge learning outcomes.

The most pronounced weakness of the studies included in the review was the validity of the evaluation instrument used to assess learning outcomes. This domain pertained to the physical assessment instrumentation such as the quiz, test, or questionnaire that was given to the participant. Only six of the included studies (21%) reported the internal structure sufficiently through dimensional-ity, measurement invariance, or reliability using the criteria set down by Rios and Wells (2014). In addition, only 10 studies (34%) stated how the content was validated, with the majority (N = 19) not reporting this information. Only three studies (Kozhevnikov et al. 2013; Makransky et al. 2017; Molina-Carmona et al. 2018) appropriately outlined both the internal structure and validity of evaluation content. The majority of studies (N = 16) did not report either item.

Of the 29 studies in the current review, 26 scored full marks on the data analy-sis domain with both an appropriate and sufficiently complex analysis and report-ing of the findings. Three studies scored lower than this due to reporting descrip-tive statistics only (Angulo and de Velasco 2013; Babu et al. 2018; Ray and Deb 2016).

Overall, the average quality score of a study in this systematic review was 12.7 with a range of 10.5–14.5 (SD = 1.0). This was 1.8 points higher than the review carried out by Jensen and Konradsen (2018), which could in part be due to differ-ences in study design criteria which was previously outlined. A full summary of the MERSQI scores for each study can be found in Table 2 in the Appendix.

Subject areas and learning domains

Table 1 provides a summary of all 29 articles that were included in the review. Studies were first categorised by the population that was sampled. Most I-VR studies took place in a higher education establishment (college or university) using undergraduate or postgraduate students (N = 25). A smaller number of

9

1 3

J. Comput. Educ. (2021) 8(1):1–32

Tabl

e 1

Det

ails

of i

nclu

ded

studi

es in

clud

ing

dom

ain,

var

iabl

es, a

nd su

mm

ary

of m

ain

findi

ngs

Stud

y (y

ear)

Lear

ning

dom

ain

HM

D(s

) use

dC

ompa

rison

var

iabl

e(s)

Find

ings

Har

ringt

on e

t al.

(201

8)C

ogni

tive

Sam

sung

Gea

r VR

2D v

ideo

No

diffe

renc

e in

info

rmat

ion

rete

ntio

n be

twee

n bo

th

med

ium

sYo

gana

than

et a

l. (2

018)

Proc

edur

alU

nspe

cifie

d2D

vid

eoFa

ce-to

-face

trai

ning

I-VR

pro

duce

d hi

gher

scor

es in

kno

t tyi

ng sk

ills.

No

diffe

renc

e in

com

plet

ion

time

Mak

rans

ky e

t al.

(201

7)C

ogni

tive

Sam

sung

Gea

r VR

Des

ktop

VR

Kno

wle

dge

test

scor

e w

orse

whe

n us

ing

I-VR

. No

diffe

renc

e in

tran

sfer

scor

eG

reen

wal

d et

 al.

(201

8)C

ogni

tive

HTC

Viv

e2D

syste

mSt

udy

foun

d no

diff

eren

ce in

test

scor

es o

r num

ber

of m

oves

bet

wee

n bo

th m

ediu

ms.

One

sess

ion

was

sl

ower

in I-

VR

com

pare

d to

2D

Lam

b et

 al.

(201

8)C

ogni

tive

HTC

Viv

eLe

ctur

eSe

rious

gam

eH

ands

-on

activ

ity

I-VR

pro

duce

d a

bette

r tes

t sco

re th

an a

trad

ition

al

lect

ure.

No

diffe

renc

e be

twee

n I-V

R a

nd e

duca

-tio

nal g

ame

or h

ands

-on

activ

ityW

ebste

r (20

16)

Cog

nitiv

eSo

ny H

MZ

Lect

ure

I-VR

pro

duce

d si

gnifi

cant

ly h

ighe

r tes

t sco

res t

han

a le

ctur

eSm

ith e

t al.

(201

8)C

ogni

tive

and

proc

edur

alO

culu

s Rift

Des

ktop

VR

Writ

ten

instr

uctio

nsN

o di

ffere

nce

in te

st sc

ore

or c

ompl

etio

n tim

e be

twee

n I-V

R a

nd o

ther

met

hods

Ostr

ande

r et a

l. (2

018)

Cog

nitiv

eH

TC V

ive

Instr

ucto

r les

son

In 6

out

of 7

less

ons,

I-VR

was

no

mor

e eff

ectiv

e th

an a

less

on d

eliv

ered

by

an in

struc

tor

Ang

ulo

and

de V

elas

co (2

013)

Cog

nitiv

eO

ther

3D p

hysi

cal m

odel

Hig

her a

ppra

isal

of a

rchi

tect

ural

spac

es d

esig

ned

with

I-V

R th

an p

hysi

cal m

odel

Step

an e

t al.

(201

7)C

ogni

tive

Ocu

lus R

iftO

nlin

e te

xtbo

okN

o di

ffere

nce

in te

st sc

ore

at p

re-te

st, p

ost-t

est,

or

8-w

eek

follo

w-u

pSa

nkar

anar

ayan

an e

t al.

(201

8)C

ogni

tive

and

proc

edur

alO

culu

s Rift

No

I-VR

(writ

ten

mat

eria

l onl

y)U

sing

I-V

R p

rodu

ced

high

er le

arni

ng g

ains

in

proc

edur

al a

ctiv

ity a

nd tr

ansf

er te

st. T

here

was

no

diffe

renc

e in

MC

Q te

st sc

ores

Foga

rty e

t al.

(201

7)C

ogni

tive

Ocu

lus R

iftN

o I-V

R (l

ectu

res o

nly)

At p

re-in

terv

entio

n th

e pa

rtici

pant

s per

form

ed si

g-ni

fican

tly p

oore

r tha

n th

e co

ntro

l gro

up. A

t pos

t-in

terv

entio

n, th

is d

iffer

ence

dis

appe

ared

show

ing

impr

ovem

ent i

n co

gniti

ve k

now

ledg

e

10 J. Comput. Educ. (2021) 8(1):1–32

1 3

Tabl

e 1

(con

tinue

d)

Stud

y (y

ear)

Lear

ning

dom

ain

HM

D(s

) use

dC

ompa

rison

var

iabl

e(s)

Find

ings

Rupp

et a

l. (2

019)

Cog

nitiv

eG

oogl

e C

ardb

oard

Ocu

lus C

V1

Ocu

lus D

K2

2D v

ideo

(on

phon

e sc

reen

)Si

gnifi

cant

ly b

ette

r lea

rnin

g ou

tcom

es w

ere

dem

onstr

ated

on

test

scor

es w

hen

I-VR

was

use

d co

mpa

red

to n

on-im

mer

sive

con

ditio

nB

hara

thi a

nd T

ucke

r (20

15)

Proc

edur

alO

culu

s Rift

Des

ktop

VR

Func

tiona

l ana

lysi

s tas

k w

as c

arrie

d ou

t sig

nifi-

cant

ly fa

ster i

n I-V

R c

ondi

tion

com

pare

d to

D-V

RJo

hnsto

n et

 al.

(201

8)C

ogni

tive

Ocu

lus R

iftN

o I-V

R (l

ectu

res o

nly)

Thos

e pa

rtici

pant

s who

und

erw

ent t

he I-

VR

exp

eri-

ence

per

form

ed b

ette

r on

final

exa

m q

uesti

on th

an

thos

e w

ho d

id n

ot u

se I-

VR

Allc

oat a

nd v

on M

ühle

nen

(201

8)C

ogni

tive

HTC

Viv

eTe

xtbo

ok2D

Vid

eoTh

ose

who

use

d I-V

R p

erfo

rmed

bet

ter o

n a

test

than

2D

con

trol g

roup

. The

I-V

R g

roup

per

form

ed

bette

r on

rem

embe

ring

que

stion

s tha

n th

ose

in th

e te

xtbo

ok g

roup

, but

no

diffe

renc

e in

und

erst

and-

ing

ques

tions

Koz

hevn

ikov

et a

l. (2

013)

Cog

nitiv

enV

isor

SX

60D

eskt

op V

RPe

rform

ance

in u

nder

stan

ding

rela

tive

mot

ion

con-

cept

s wer

e si

gnifi

cant

ly h

ighe

r whe

n pa

rtici

pant

s us

ed I-

VR

com

pare

d to

D-V

RPa

rong

and

May

er (2

018)

Cog

nitiv

eH

TC V

ive

Slid

esho

wI-V

R w

as d

emon

strat

ed to

cau

se si

gnifi

cant

ly p

oore

r re

sults

on

a ce

ll bi

olog

y te

st th

an th

ose

who

le

arne

d us

ing

a tra

ditio

nal P

ower

Poin

tO

lmos

-Ray

a et

 al.

(201

8)C

ogni

tive

Sam

sung

Gea

r VR

Des

ktop

VR

Thos

e pa

rtici

pant

s who

lear

ned

abou

t geo

grap

hy

usin

g I-V

R d

emon

strat

ed si

gnifi

cant

ly h

ighe

r le

arni

ng g

ains

and

med

ium

term

rete

ntio

n th

an

thos

e w

ho u

sed

D-V

RA

kbul

ut e

t al.

(201

8)C

ogni

tive

Mob

ile V

RLe

ctur

ers a

nd la

b se

ssio

nsTe

st sc

ores

wer

e si

gnifi

cant

ly h

ighe

r in

thos

e w

ho

used

the

I-VR

pla

tform

com

pare

d to

thos

e w

ho

lear

ned

thro

ugh

lect

ures

and

lab

sess

ions

alo

neR

ay a

nd D

eb (2

016)

Cog

nitiv

eG

oogl

e C

ardb

oard

Slid

esho

wLe

ctur

esO

ver t

he c

ours

e of

16

sess

ions

, tho

se w

ho le

arne

d co

mpu

ter s

cien

ce th

roug

h I-V

R te

nded

to sc

ore

high

er o

n te

st sc

ores

than

thos

e w

ho d

id n

ot

11

1 3

J. Comput. Educ. (2021) 8(1):1–32

Tabl

e 1

(con

tinue

d)

Stud

y (y

ear)

Lear

ning

dom

ain

HM

D(s

) use

dC

ompa

rison

var

iabl

e(s)

Find

ings

Mor

o et

 al.

(201

7)C

ogni

tive

Ocu

lus R

iftD

eskt

op V

RI-V

R a

nd D

-VR

wer

e sh

own

to b

e eq

ually

effe

ctiv

e m

odes

of l

earn

ing

on a

n an

atom

y sk

ill te

stM

ares

ky e

t al.

(201

9)C

ogni

tive

Ocu

lus R

iftIn

depe

nden

t stu

dyTh

ose

who

use

d I-V

R to

lear

n ab

out c

ardi

ac

anat

omy

scor

ed si

gnifi

cant

ly h

ighe

r on

post-

test

than

a c

ontro

l gro

up w

ho u

sed

inde

pend

ent s

tudy

Mad

den

et a

l. (2

018)

Cog

nitiv

eO

culu

s Rift

Des

ktop

VR

Bal

l-and

-stic

k m

etho

dI-V

R w

as n

o m

ore

effec

tive

than

D-V

R o

r a tr

adi-

tiona

l mod

el d

emon

strat

ion

whe

n le

arni

ng a

bout

as

trono

my

Liou

and

Cha

ng (2

018)

Cog

nitiv

eH

TC V

ive

Trad

ition

al d

idac

tic te

achi

ngI-V

R w

as sh

own

to p

rodu

ce si

gnifi

cant

ly h

ighe

r le

vels

of l

earn

ing

in a

nato

my

and

chem

istry

tests

th

an th

ose

who

did

not

use

the

tech

nolo

gyM

olin

a-C

arm

ona

et a

l. (2

018)

Cog

nitiv

eM

obile

VR

Des

ktop

VR

I-VR

was

show

n to

impr

ove

the

spat

ial a

bilit

y of

th

ose

parti

cipa

nts w

ho u

sed

it, c

ompa

red

to th

ose

that

use

d D

-VR

Bab

u et

 al.

(201

8)C

ogni

tive

HTC

Viv

eD

eskt

op V

R (t

able

t bas

ed)

Alth

ough

I-V

R a

nd D

-VR

pro

duce

d si

mila

r pos

t-tes

t sc

ores

, I-V

R w

as sh

own

to im

prov

e re

tent

ion

of

know

ledg

e le

arne

dA

lhal

abi (

2016

)C

ogni

tive

Ocu

lus R

iftTr

aditi

onal

met

hods

I-VR

was

show

n to

pro

duce

sign

ifica

ntly

hig

her

atta

inm

ent i

n en

gine

erin

g co

nten

t tha

n tra

ditio

nal

educ

atio

nal m

etho

ds. F

urth

erm

ore,

HM

D b

ased

I-V

R p

rodu

ced

bette

r res

ults

than

a C

AVE

base

d sy

stem

Gut

iérr

ez-M

aldo

nado

et a

l. (2

015)

Affe

ctiv

e an

d co

gniti

veO

culu

s Rift

DK

1St

ereo

scop

ic d

eskt

op V

RB

oth

imm

ersi

ve a

nd n

on-im

mer

sive

med

ium

s pr

oduc

ed si

mila

r res

ults

in p

sych

iatri

c in

terv

iew

tra

inin

g

12 J. Comput. Educ. (2021) 8(1):1–32

1 3

studies used high school pupils (N = 2), or adult education students (N = 2) such as those in vocational or work-based programmes.

Each of the included studies were then examined for the topic and subject area they pertained to. This was based upon the nature of the VR experience, partici-pant pool, and intervention. In total, six main subject areas were identified. This included: medicine (N = 4), science (biology, chemistry, and physics) (N = 13), social science (human geography) (N = 1), computer science (N = 2), engineering and architecture (N = 7), and safety education (N = 1). One of the included studies (Molina-Carmona et al. 2018) did not neatly fit into one of the pre-defined cat-egories as it utilised I-VR to teach abstract spatial concept abilities to multimedia engineering students. It was therefore categorised as ‘other’. Figure 2 shows the percentage of papers included by subject area.

In addition to the subject area, the learning outcomes were also categorised into three specific domains based upon the findings of previous systematic reviews, as well as the taxonomy of learning developed by Bloom et al. (1956). The first was cognitive which related to studies that intended to teach specific declarative infor-mation or knowledge. The second was procedural which intends to teach the user how to perform a specific task or learn psychomotor skills that pertain to a cer-tain activity. Finally, the third learning outcome was affective skills which can be defined as a growth in areas relating to emotion and attitude. Most of the included studies (N = 24) concentrated on the cognitive domain, with two studies focus-ing on purely procedural and psychomotor skills. The remaining studies were a blend of two domains with Sankaranarayanan et al. (2018) and Smith et al. (2018) examining both cognitive and procedural skills, and Gutiérrez-Maldonado et al. (2015) utilising both cognitive knowledge and affective awareness in psychiatric diagnosis training. Figure 3 shows the percentage of studies included by learning domain.

Science45%

Engineering and Architecture

24%

Medicine14%

Computer Science7%

Social Science4%

Safety Educa�on3%

Other 3%

Studies by subject area

Fig. 2 Percentage of papers per subject area

13

1 3

J. Comput. Educ. (2021) 8(1):1–32

Experimental design

Outcome measures

A thorough understanding of the role of I-VR as a pedagogical practice can only be fully appreciated when consideration is given to the assessment instrumenta-tion and outcome measures used to assess its utility. As previously mentioned, when analysing the quality of the included studies, it was the evaluation instru-mentation itself that was shown to have the most pronounced weakness.

To assess the evaluation instruments being employed, the measures were bro-ken down into two broad domains: outcome measures, and assessment instru-mentation. Outcome measures can broadly be defined as how learning outcomes were quantified (e.g. by comparing test scores). Assessment instrumentation pertains to the evaluative instrument itself that is used to measure the learning outcomes (e.g. multiple-choice questionnaire, exam style questions). Twenty-seven of the included studies (93%) used test scores to assess learning outcomes, with the majority using this as their sole method. There were four studies that used completion time as a metric of learning outcome, although only one study (Bharathi and Tucker 2015) used this method exclusively. There was one study (Sankaranarayanan et  al. 2018) that used the correct order of operation in a procedural task as one of its main outcome measures. There were three papers that utilised other outcome measures that could not be easily categorised. For instance Greenwald et al. (2018) used counting the number of moves needed to complete a task, Webster (2016) used the performance on a virtual jigsaw puz-zle, and Angulo and de Velasco (2013) used a mixture of scores and evaluations of an architectural space.

Cogni�ve83%

Procedural7%

Mixed (Procedural & Cogni�ve)

7%

Mixed (Affec�ve & Cogni�ve)

3%

Studies by learning domain

Fig. 3 Percentage of papers per learning domain

14 J. Comput. Educ. (2021) 8(1):1–32

1 3

Assessment instrumentation

In terms of the direct assessment instrumentation used to examine outcome meas-ures, there was a heavy reliance on the multiple-choice questionnaire (MCQ). There were eighteen (62%) studies that utilised this method of assessment, with the majority of those using it as their sole evaluation instrument. Only five studies used extended answer questions (long or short form) to probe for a deeper understanding of the educational content, which was usually done in combination MCQs. The stud-ies that included the teaching of procedural skills used marking criteria and check-lists to assess whether the correct order was being followed. For instance Yogana-than et al. (2018) had an expert assessor use marking criteria to assess the knot tying skills of students. Similarly, Smith et al. (2018) had evaluators observe students with a decontamination checklist which evaluated performance based upon certain key tasks that were performed.

There were a smaller number of studies that used more novel instrumentation and methods for evaluation, such as the utilisation of labelling and identifying parts of a 3D model (e.g. Babu et al. 2018; Moro et al. 2017; Stepan et al. 2017). Fogarty et al. (2017) probed spatial and conceptual understanding in their assessment instrument by having participants draw shapes based on their understanding of structural engi-neering principles. Additionally, Alhalabi (2016) used quizzes on both mathematical knowledge, and the appropriate understanding of graphics and charts as an assess-ment measure for engineering students.

There were three studies (Liou and Chang 2018; Madden et  al. 2018; Ray and Deb 2016) where the nature of the assessment instrumentation could not be defini-tively ascertained from the description.

The majority of studies (62%) utilised the pretest–posttest design by comparing the test scores pre-intervention with those after the I-VR experience. The remainder of the studies tended to assess post-intervention scores only, usually by comparing the difference in learning outcome between I-VR and one or more control group. Less conventional means of post-intervention comparison was sometimes utilised, such as Johnston et al. (2018) comparing the average score on a specific exam ques-tion that pertained to an I-VR experience that some student did or did not undertake.

There were four studies that examined the short to medium term retention rate of information and learning through follow-up assessment. This ranged from as soon as 1 day after the initial I-VR experience (Babu et al. 2018), through to 6 months post-intervention (Smith et al. 2018). Olmos-Raya et al. (2018) and Stepan et al. (2017) had follow-up assessments at 1-week and 8-weeks, respectively.

Intervention characteristics

In addition to having appropriate assessment measures, it is also important to exam-ine the nature of the I-VR intervention itself. The most popular HMDs used were the Oculus (N = 13) and HTC Vive (N = 7). There were seven studies that used a form of mobile VR headset such as the Google Cardboard or Samsung Gear VR. In one study (Yoganathan et  al. 2018) the exact HMD system used could not be

15

1 3

J. Comput. Educ. (2021) 8(1):1–32

definitively ascertained. Figure 4 provides a breakdown of the HMDs used in the included studies.

Most studies (72%) featured only a single intervention with the I-VR experience, meaning that the student was exposed to the technology just once. There were a few exceptions to this, with Ostrander et al. (2018) having seven individual I-VR experi-ences in their manufacturing lesson, as well as Ray and Deb (2016) utilising smart-phone based I-VR over the course of 16 sessions. Other studies allowed a greater degree of freedom in the number of interventions or times that a student could use I-VR. This was usually a result of time being dedicated to the technology through scheduled classes or lab times (e.g. Akbulut et  al. 2018; Alhalabi 2016; Molina-Carmona et al. 2018). Despite this, the I-VR intervention was usually a single and isolated one.

As well as most of the studies featuring a single intervention, the exposure dura-tion was also typically short, ranging from 6 to 30 mins. Generally, the exception to this was when the I-VR exposure lasted as long as it took the participant to complete a specific task, assessment, or procedure within the immersive environment (e.g. Babu et al. 2018; Bharathi and Tucker 2015; Greenwald et al. 2018; Sankaranaray-anan et al. 2018). Molina-Carmona et al. (2018) supplemented the limited interven-tion duration by allowing participants to take the HMD away with them, so they could access the educational content for 2 weeks outside the classroom. However, just as with the number of interventions, exposure duration tended to be short, last-ing on average 13 mins for those I-VR experiences that had a set time limit.

Most of the studies (62%) utilised I-VR as the sole method of learning, and did not combine the technology with additional pedagogical practices or materials to encourage learning. Only a limited number of studies (38%) supplemented the I-VR lesson by providing additional aids that were designed to complement the educa-tional experience. For example, Smith et al. (2018) and Stepan et al. (2017) both had participants use web-based modules and textbooks in addition to the I-VR experi-ence before testing them on learning outcomes. A number of the included studies

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

Oculus HMD HTC Vive GoogleCardboard

SamsungGear-VR

OtherMobile-VR

Sony HMZ-T1

NVIS nVisorSX60

Other NotSpecified

Perc

enta

ge o

f stu

dies

HMD type

HMDs used

Fig. 4 HMDs used in studies

16 J. Comput. Educ. (2021) 8(1):1–32

1 3

also utilised lecture based instruction or scheduled class time to operate in tandem with the I-VR environments (e.g. Akbulut et al. 2018; Fogarty et al. 2017; Johnston et al. 2018; Ray and Deb 2016; Sankaranarayanan et al. 2018).

Theoretical frameworks

A fundamental component of any educational tool or activity is to ground its use in learning theory or educational paradigms. Learning theories can broadly be bro-ken down and defined by proposals regarding how student imbibe, process, and retain the information that they have learned (Pritchard 2017; Schunk 2011). When applied to educational I-VR, these theories should provide a pedagogical frame-work and foundation as how best to design interventions. Papers were examined for explicit statements regarding the theoretical basis for the study. Those papers that only mentioned theoretical approaches as part of the introduction or literature review were not deemed to have explicitly stated them. The majority of studies (N = 24) made no mention of a theoretical approach underpinning the intervention. There were two studies that applied a generative learning framework (Makransky et  al. 2017; Parong and Mayer 2018). This can be defined as an approach where the learner will actively integrate new knowledge with information that is already stored in the brain (Osborne and Wittrock 1985). Webster (2016) employed Mayer’s (2009, 2014) Cognitive Theory of Multimedia Learning (CTML). CTML proposes a dual channel approach where visual and auditory information is actively processed, organised, and then stored in the brain. This is contingent on neither channel (visual or auditory) becoming overloaded with information. Smith et  al. (2018) used the NLN Jeffries Simulation Theory as their theoretical basis. This theory, most com-monly employed in nursing education, is where students learn information as part of a simulated experience (Jeffries et al. 2015). For the teaching of vocational skills, Babu et al. (2018) stated that their approach aligned with situated learning. Situated learning employs a constructivist approach in that students learns professional skills by actively participating in the experience (Huang et al. 2010).

Learning outcomes

For I-VR to gain wide-spread acceptance as a reliable pedagogical method, it must be shown to confer a tangible benefit in terms of learning outcomes over less immer-sive or traditional teaching methods.

Cognitive studies

There were twenty-four included studies that fell into the cognitive domain and aimed to teach specific declarative information or knowledge through the I-VR environment. The current review found that most studies demonstrated benefits in terms of learning outcomes when using I-VR compared to less immersive methods of learning. A smaller number of studies found no significant advantage regardless

17

1 3

J. Comput. Educ. (2021) 8(1):1–32

of the pedagogical method being utilised. The results of these cognitive studies have been broken down by subject area.

Science based cognitive studies

The review found that cognitive learning activities requiring a high degree of visual-isation and experiential understanding may be best facilitated using immersive tech-nologies. For instance, both Liou and Chang (2018) and Maresky et al. (2019) found that anatomical learning facilitated by complex 3D visualisations of the human body were more conducive to learning in I-VR compared to traditional learning or inde-pendent study. Similarly Lamb et al. (2018) used a virtual environment that allowed for the manipulation and movement of strands of DNA, which produced better learn-ing outcomes in content tests than a lecture or a serious educational game. Greater attention and engagement with the I-VR environment as measured with infrared spectroscopy was one of the possible explanations given for the effectiveness of the technology. In a study by Johnston et al. (2018), participants volunteered to take part in a cell biology experience either because they were engaged with the subject mat-ter itself, or wanted supplementary instruction. Johnston et al. (2018) compared the exam scores of those students who volunteered to take part with those who did not. The study found that participants who underwent the I-VR experience scored 5% higher on the related exam question compared to the rest of the assessment. Those who did not undergo the cell biology I-VR experienced scored on average 35% worse on the same question.

The increase in graphical fidelity afforded by I-VR has allowed not only for the creation of complex computer-generated environments, but also the viewing of high resolution 360° video. In one such study, Rupp et al. (2019) had participants watch a six minute 360° video about the International Space Station with either a HMD which created a sense of immersion and presence, or on a mobile screen. The research found that those participants in the HMD condition scored significantly higher in a learning outcome test (MCQ) than those who watched the video in the non-immersive condition.

Although I-VR has been shown to confer a benefit in science education, there is evidence to suggest that not all learning objectives can be learned equally well. For instance, in task devised by Allcoat and von Mühlenen (2018), the researchers found that I-VR conferred a benefit over video or textbook learning when questions required remembering, but not ones pertaining to understanding of the material. The authors suggest that unfamiliarity and the novelty of the I-VR environments could have contributed to the lack of an obvious benefit in the latter domain. Another study that examined specific question types to understand I-VR’s effectiveness was under-taken by Kozhevnikov et  al. (2013). In this study, participants learned more con-ceptual and abstract relative motion concepts using either I-VR or D-VR. The study demonstrated that those in the I-VR condition performed significantly better in the two-dimensional problems than their D-VR counterparts, although there was no sig-nificant difference between groups in problems featuring only one spatial dimension.

There were several studies in the domain of science that showed no obvious ben-efits to using I-VR over traditional pedagogical methods. Two studies (Greenwald

18 J. Comput. Educ. (2021) 8(1):1–32

1 3

et al. 2018; Moro et al. 2017) compared science learning in I-VR with desktop based VR and 2D videos. Results showed no clear benefit of I-VR based instruction when comparing the difference and significance of learning outcomes between mediums. Similarly, Stepan et  al. (2017) found that I-VR was no more effective than online textbooks for the teaching of neuroanatomy. Interestingly, the same study found no difference in information retention rates when the participants were reassessed 8-weeks later. Madden et al. (2018) used I-VR, D-VR, and the traditional ball and stick method to teach astronomy principles pertaining to phases of the moon. The study found that I-VR and D-VR produced comparable test score results, with no significant differences in attainment. However, the authors commented on the encouraging finding that despite being a novel technology to most participants, I-VR still facilitated comparable learning outcomes to more traditional methods.

Despite the majority of studies demonstrating that I-VR learning is more effective or at least on par with traditional pedagogical methods, some studies have shown a detrimental effect of I-VR. Makransky et al. (2017) used a combination of assess-ment and EEG to find that an I-VR lab simulation produced significantly poorer test scores than a non-immersive alternative. Similarly, during another science experi-ment, Parong and Mayer (2018) found that students who used I-VR during a biology lesson scored significantly poorer than those who learned using a PowerPoint. Both of these studies cited Mayer’s (2009, 2014) Cognitive Theory of Multimedia Learn-ing as a possible explanation for the poorer performance for I-VR. The research-ers postulate that the high-fidelity graphics and animations could have significantly increased cognitive load, which would have detracted from the learning task at hand. It was therefore proposed that a less immersive, yet well designed PowerPoint presentation would facilitate better learning outcomes than a graphically rich I-VR experience.

Engineering and architectural based cognitive studies

I-VR was effective in engineering and architectural education as a tool to visualise key concepts within the discipline. For example, Fogarty et al. (2017) allowed stu-dents to volunteer for an I-VR experience who struggled with the comprehension of spatial arrangements in structural engineering. Before the intervention, those stu-dents who volunteered to take part scored significantly poorer than their non-inter-vention counterparts. At post-test, not only did those who underwent the I-VR expe-rience score significantly higher than they did at pre-test, but they also eliminated the significant difference with the non-intervention group. This would suggest that I-VR could serve an important function in supplementing or assisting learning in those students who are struggling to grasp complex problems relating to their disci-pline. Interestingly, Angulo and de Velasco (2013) used many of these same spatial and visualisation principles in a more applied setting. Their study split students into groups who were tasked with designing an architectural space (a health clinic wait-ing room), either with the assistance of an I-VR design tool (experimental group) or a physical model (control group). The study found the space that gained the most positive affect was designed by the I-VR group.

19

1 3

J. Comput. Educ. (2021) 8(1):1–32

Webster (2016) created a graphically rich immersive environment which com-bined active and passive media with elements of gamification and interactivity to teach corrosion concepts to US army personnel. The study found that although both the I-VR environment and a traditional lecture were effective pedagogical methods for teaching these principles, it was the I-VR condition that produced the highest gain in knowledge acquisition.

There was also some evidence to suggest that I-VR interventions could assist in short-term retention of information in engineering related activities. Babu et al. (2018) found that although participants performed similarly in a mechanical label-ling task using either I-VR or a tablet computer immediately post-intervention, the I-VR group had better retention of knowledge when the test was re-administered 1 day later. Furthermore, those participants in the I-VR group were also less likely to wrongly recall information compared to the non-immersive group on the retention test.

Interestingly, Ostrander et al. (2018) examined cognitive learning outcomes over seven separate manufacturing tasks utilising I-VR in one group, and a traditional class-based environment in the other. The study found that in six out of seven tasks, I-VR was no more effective than a traditional class where students could interact with the instructor or the physical models that they were accustomed to.

Medical based cognitive studies

Although papers featuring surgical simulators did not form part of this review, there were several applications of I-VR in the field of general medical education. Har-rington et al. (2018) had medical students watch a ten-minute 360° video with slides containing surgical information superimposed over it. This was viewed either on a large television screen, or through a Gear VR headset. The study found no signifi-cant differences in knowledge retention scores between those who viewed the infor-mation through a HMD, or a traditional television screen. Despite not showing a distinct advantage in cognitive learning outcomes, the authors did suggest that the 360° surgical experience may facilitate a better understanding of how teamwork and interaction takes place within an operating theatre. This type of learning may be more difficult to measure using assessment instrumentation such as the MCQ, but nevertheless it could be that the experiential nature of I-VR may facilitate an under-standing of interactions and communications. Smith et al. (2018) used either I-VR or D-VR on a computer to teach students about decontamination protocols. The research found that I-VR was no more effective than D-VR in a MCQ immediately post-intervention, or at 6-weeks follow-up.

Computer science based cognitive studies

Two studies demonstrated a significant advantage in using I-VR to teach computer science information. For instance, Akbulut et  al. (2018) found that students who underwent an I-VR experience that focused on software engineering principles scored 12% higher than students who did not undergo I-VR learning. Interestingly, in a study by Ray and Deb (2016) that ran over 16 sessions on microcontrollers in

20 J. Comput. Educ. (2021) 8(1):1–32

1 3

computing, the I-VR group performance lagged behind that of the control group who used slideshows for the first four sessions. It was only on session number five that the I-VR group outperformed the control group, and this performance enhancement remained relatively stable in the majority of the remaining 11 sessions. In effect, it took the I-VR group some time to catch up with the control group, but once they did, they tended to outperform them in the remaining lessons. The authors propose that this may have been due to the novelty of the I-VR equipment which participants took time to become comfortable and competent with.

Other cognitive studies

I-VR was also used by Molina-Carmona et al. (2018) as a means of spatial ability acquisition and visualisation. The study showed that learning outcomes as assessed by a spatial visualisation test were higher among those who undertook the task in an immersive, compared to a non-immersive environment. There was only one study in the field of social science that used I-VR to teach cognitive information. Olmos-Raya et al. (2018) used either I-VR or a tablet-based system to teach high school stu-dents about human geography. The research found that I-VR produced higher learn-ing gains on a MCQ than the tablet-based system. Further, those who used I-VR performed better than the non-immersive group on a knowledge retention quiz when administered 1-week later.

Procedural studies

Three of the four studies that attempted to utilise I-VR as a means of teaching proce-dural skills showed a distinct advantage over less immersive methods. Bharathi and Tucker (2015) found that engineering students were faster in assembling a house-hold appliance in a virtual functional analysis activity in I-VR compared to D-VR. Yoganathan et  al. (2018) also found that medical students were more accurate in knot tying practice when using I-VR as a training tool as opposed to a control group who used a standard video. Medical and surgical residents were also studied by Sankaranarayanan et al. (2018) who used I-VR as a teaching tool for emergency fire response in an operating theatre environment. This study found that 70% of those who utilised the I-VR training were able to perform the correct procedure in the correct order. This was 50% higher than the control group who were exposed to a presentation and reading material only and did not experience I-VR.

One of the studies found no significant advantage to using I-VR as a learning tool. Smith et al. (2018) split nursing students into an I-VR group, a D-VR group (desktop PC based), or a written instruction group to learn about appropriate proto-cols for decontamination. The study found that there was no significant difference in performance between the groups as measured by a decontamination checklist, or the time taken to complete the task. Furthermore, reassessment 6 months later showed that I-VR conferred no advantage in procedural knowledge retention (accuracy and speed) compared to less immersive methods.

21

1 3

J. Comput. Educ. (2021) 8(1):1–32

Affective studies

Only one of the studies attempted to use I-VR as a pedagogical tool to teach applied behavioural and affective skills. Gutiérrez-Maldonado et al. (2015) used I-VR in the field of diagnostic psychiatry in an attempt to improve interview skills when assess-ing patients for an eating disorder. Participants were exposed to a series of virtual patient avatars in either the I-VR condition, or a D-VR condition using stereoscopic glasses. Analysis showed that both conditions were equally as effective, and no sig-nificant differences were shown in the acquisition of skills between the two groups. Nevertheless, this was a novel study as it traversed the boundaries between tradi-tional cognitive skill acquisition and applied behavioural and affective change.

Discussion and implications

The purpose of this review was to investigate I-VR’s effectiveness as a pedagogical method in education, as well as examining the experimental design and character-istics of the included studies. In particular, the review found that the utilisation of I-VR is typically restricted to a small number of subject areas such as science and engineering. Furthermore, a heavy reliance has been placed on the MCQ and test score measures to assess learning outcomes. In addition, I-VR interventions were typically short and isolated, and were not complemented with additional or supple-mentary learning material. Despite this, most studies did find a significant advantage of using I-VR over less immersive methods of learning. This was the case particu-larly when the subject area was highly abstract or conceptual, or focused on proce-dural skills or tasks.

Is the utilisation of I‑VR within education restrictive?

The findings of the review suggest a relatively homogenous application of I-VR in terms of both the subject areas represented, as well as the learning domain being taught. Almost 70% of the studies were from the field of science or engineering, with other subjects being marginally represented. It is worth noting, however, that although medical disciplines made up a small proportion of the studies included (14%), this was because most medical applications of I-VR feature surgical simula-tors and therefore were not part of the current review’s inclusion criteria. Most stud-ies utilised I-VR as a way of teaching cognitive skills, with only a handful examin-ing the procedural or affective applications.

The findings of the review raise several issues when trying to assess the general effectiveness of I-VR in education. Similar to the findings of others (e.g. Jensen and Konradsen 2018; Radianti et  al. 2020), the arts, humanities, and social sciences were underrepresented in in the current review. This makes generalisable conclu-sions as to the cognitive benefit of the technology in these subjects challenging. One major reason for this under representation may be the lack of I-VR learning con-tent, experiences, and teaching tools. Jensen and Konradsen (2018) highlighted that

22 J. Comput. Educ. (2021) 8(1):1–32

1 3

instructors are restricted to the material published and produced by VR designers, and this may not necessarily meet the individual needs of the teacher, or the learning outcome trying to be achieved. The skillset needed to produce and create wholly vir-tual environments that can be rendered and displayed in a HMD is still demanding, despite the release of affordable VR creation suites. Therefore, the bespoke I-VR experiences required to teach social science lessons (or indeed any subject) is com-pletely dependent on an appropriate I-VR tool already existing or having the techni-cal proficiency to create one. A potential solution to the lack of bespoke material could be the examination of the pedagogical effectiveness of HMD 360° video in the classroom, as opposed to computer-generated environments. This content is com-paratively easier to create using appropriate video equipment and can be tailored to the individual needs of the instructor or student group. Widespread research that examines the potential of I-VR in a multitude of diverse disciplines and learning domains will continue to be constrained by the availability of the requisite material. That is until such a time where bespoke and individually tailored I-VR experiences become more accessible.

Implications of outcome measures and assessment instrumentation

One of the most striking characteristics of the assessment instrumentation used in the studies was the reliance on the MCQ to assess learning outcomes. Although there have been many debates on the respective advantages and disadvantages of utilising the MCQ, it has generally been considered that it is most appropriate for testing large amounts of surface knowledge over the course of an entire module or syllabus (Excell 2000). As O’Dwyer (2012) points out, the assessment instrumenta-tion encourages comprehensive learning of the entirety of the taught material, as opposed to just specific components. However, since most of the studies featured single interventions of between 6 and 30 mins, doubts are cast on whether MCQs are the most appropriate way to assess learning. Since the MCQ was most com-monly administered immediately after the I-VR experience, much of the information may still be stored in short-term memory, and this may not give an accurate reflec-tion of more comprehensive learning or long-term retention.

A second disadvantage associated with the heavy reliance on the MCQ is the limited breadth of knowledge that can be assessed. In Jensen and Konradsen’s (2018) systematic review, the researchers found that none of the cognitive studies went beyond teaching lower level cognitive skills as defined by Bloom’s taxonomy (Bloom et  al. 1956). Similar results were found in the current review, with most studies requiring only a knowledge of previously learned material to successfully achieve the desired learning goal. Previous research on pedagogical assessment material (e.g. Ozuru et al. 2013) has suggested that the MCQ cannot assess higher levels of cognitive understanding or conceptual knowledge. Therefore, it may not only be the nature of the I-VR experience itself that restricts the learning of higher level cognitive skills, but also the restrictive nature of the assessment instrumenta-tion that may impede an appropriate demonstration of learning outcomes. The uti-lisation of short or long form answers could be able to provide a more appropriate

23

1 3

J. Comput. Educ. (2021) 8(1):1–32

measure of the depth of learning achieved, giving the student an opportunity to dem-onstrate their conceptual knowledge of a given subject. Furthermore, I-VR research could benefit by expanding the very definition of what constitutes a learning out-come. This could be achieved by not relying exclusively on test score comparisons, but rather examine how I-VR could be used to foster deeper conceptual understand-ing through experiential learning and subsequent classroom discussions with peers or instructors.

Implication of intervention characteristics for learning outcomes

The current review examined how I-VR is being utilised in experimental and applied settings, and the implications this has for assessing its pedagogical suitability. In most studies, the participant took part in a single I-VR experience that was also short in duration. This presents several key challenges. Most importantly, the nov-elty of the I-VR technology itself may have impeded the learning experience of the user, especially if they had never used the technology before or were unfamiliar with it. This seemed to be demonstrated by Ray and Deb (2016) who found that in the initial sessions of I-VR learning, performance was on average poorer than those who underwent traditional teaching methods. It was only after the participants began to become familiar with the technology (on session number five) that learning sur-passed the control group. Similarly, studies that allowed for extended exposure to I-VR (e.g. Akbulut et al. 2018; Alhalabi 2016; Molina-Carmona et al. 2018), either through free navigation, repeated sessions, or scheduled class time, tended to show an advantage of using I-VR over non-immersive or traditional methods. It is there-fore important to address the potentially negative influence that I-VR’s novelty as a learning tool may have, especially when outcomes are directly compared to another medium or method. Scepticism for media comparison studies was highlighted in the 1980s by Clark (1983), and then later re-addressed by Parong and Mayer (2018). As Parong and Mayer (2018) put it, the side-by-side comparison of two learning meth-ods is an “apples-to-oranges type of comparison” (p. 788). This “apples-to-oranges” comparison is made starker when considering that I-VR is an unfamiliar technology to most in an educational capacity, and its pedagogical outcomes are being directly compared with familiar methods such as textbooks or lectures. It is important to consider that the novelty of HMDs and I-VR may hinder learning outcomes and classroom application, and it is therefore prudent to ensure that the degree of famili-arity with I-VR technology is factored into any direct comparison with other meth-ods. In practice, this could mean that participants require extended familiarisation trials or free navigation before the start of experimental studies as a means of miti-gating against potential problems caused by technological novelty.

In addition to the short intervention and exposure time, most studies did not com-plement I-VR with an additional method of teaching or self-learning. The limited number of studies that did tended to utilise web-based textbooks or modules, as well as lectures and scheduled class time. Encouragingly, those studies that combined or supplemented traditional class-based learning with I-VR (e.g. Akbulut et al. 2018; Fogarty et al. 2017; Johnston et al. 2018; Sankaranarayanan et al. 2018; Yoganathan

24 J. Comput. Educ. (2021) 8(1):1–32

1 3

et al. 2018) tended to show a learning advantage. This suggests that I-VR may be best employed as form of blended or multi-modal learning to supplement and com-plement class-based instruction (Garrison and Kanuka 2004). An area for investiga-tion would be to examine I-VR’s application longitudinally in a natural classroom environment. The current review contained only a limited number of studies that employed this approach, however, by implementing and studying how I-VR can be adopted and integrated into a module or syllabus, a clearer picture of its capabilities can emerge.

Learning theories ultimately provide a theoretical framework and foundation as how best to design educational interventions (Pritchard 2017; Schunk 2011). How-ever, the review found that few papers explicitly state that any predetermined learn-ing theory was used to advise the characteristics or methods of the study. Similar findings were reported in a systematic review by Radianti et al. (2020) examining I-VR use in higher education exclusively. Radianti et al.’s (2020) review found that in around 70% of the 38 studies included, no learning theory was mentioned as forming the foundation of the VR activity. Several studies have shown that educators regard clear pre-defined intervention characteristics and objectives as essential com-ponents of I-VR teaching (Fransson et al. 2020; Lee and Shea 2020). It is therefore essential that future experimental and applied research is based on a sound theo-retical basis that can advise how the technology can be appropriately utilised and assessed.

Learning outcomes in I‑VR

The current review examined learning outcomes across three domains: cognitive, procedural, and affective. By far the most popular domain was the teaching of cogni-tive skills and knowledge which made up 83% of the studies in the current review. Around half of those demonstrated a positive effect on learning when using I-VR over less immersive pedagogical methods. Most of the remaining studies showed no significant effect either way, with only a small number of papers exhibiting det-rimental results. Researchers have suggested that the increased levels of immersive content that stimulate multisensory engagement can ultimately lead to more effective learning outcomes (Webster 2016). When this is implemented in cognitive learn-ing activities that require a high degree of spatial understanding and visualisation (e.g. Maresky et al. 2019), I-VR can allow the user to gain insights that are difficult to reproduce in reality. This review has already identified scientific subjects such as biology and physics as promising avenues for educational I-VR implementation. However, other scientific disciplines that require abstract or conceptual understand-ing (e.g. chemistry, mathematics) could also benefit from the visualisation afforded by I-VR.

Studies that utilised I-VR for the teaching of procedural skills and knowledge produced encouraging results, with three of the four studies finding a significantly positive increase in learning (Bharathi and Tucker 2015; Sankaranarayanan et  al. 2018; Yoganathan et al. 2018). Interestingly, two of the studies featured a transfer component by having the user first practice the procedure in I-VR, and then use this

25

1 3

J. Comput. Educ. (2021) 8(1):1–32

form of experiential learning to complete a task in the real world. Yoganathan et al. (2018) had students practice how to tie a surgical knot in I-VR and then complete this task for real in-front of an expert. Sankaranarayanan et al. (2018) had medical students learn how to deal with an operating theatre fire by first practicing the proce-dure in I-VR, and then applying this knowledge to a mock emergency in a real oper-ating room. Both studies found a positive effect of using I-VR as the training method by demonstrating improved results when performed in a real environment. These are encouraging findings for I-VR’s effectiveness in psychomotor and procedural educa-tion, as there has been a degree of scepticism over whether I-VR simply produces a “getting good at the game” effect. For instance, Jensen and Konradsen (2018) point out that the honing of procedural skills within I-VR may simply lead to the partici-pant becoming proficient when performing the task virtually, and this may not nec-essarily transfer to the real world. The current review has identified that the two pro-cedural studies that implemented a transfer task did indeed demonstrate a significant benefit to using I-VR as an initial education method. This demonstrates that virtual training can be a successful precursor to implementation in the real world. This sug-gests that I-VR could be useful in educating students in dangerous vocational sub-jects such as electrical engineering without risk to themselves or others. However, this view is based on a small number of studies, and it is therefore important that future procedural tasks utilise a transfer activity to understand the potential scope and parameters surrounding I-VR training and real-world application.

Only one of the studies had a firm focus on the training of affective skills, namely by using I-VR as a way of teaching diagnostic interview techniques in a psychiat-ric setting (Gutiérrez-Maldonado et  al. 2015). Although this study found no clear advantage to using I-VR, other research out with the domain of education has dem-onstrated promising results in utilising the technology for affective and behavioural change. This included applying the technology successfully in areas such as expo-sure therapy, anxiety disorder treatment, and empathy elicitation (Botella et  al. 2017; Maples-Keller et al. 2017a, b; Schutte and Stilinović 2017). As a result of the strong non-educational body of literature suggesting I-VR can facilitate affective and behavioural change, future research should examine how this can be applied in an educational context, and then transferred to real-world scenarios. For instance, in their psychiatric interview experience, Gutiérrez-Maldonado et al. (2015) had users interact solely with virtual avatars, and did not have the participants demonstrate their learning with a real actor or patient. Therefore, just like with procedural skill acquisition, affective I-VR experiences should seek to understand how virtual learn-ing can then be applied to real situations.

Implications and future practice

The current review has been able to identify a body of experimental and applied research that show the potential benefits of using I-VR in education. It has already been noted that I-VR has traditionally been used to teach low level or fundamental skills and knowledge, and has not necessarily been used to facilitate what Bloom et al. (1956) would consider higher level learning. This would include analysing and

26 J. Comput. Educ. (2021) 8(1):1–32

1 3

evaluating experience. By expanding the definition of learning outcomes to encom-pass potential benefits such as an increased depth of understanding or the ability to identify complex themes, pedagogical practice can take advantage of the inherent strength of the medium. These should be comprehensively analysed to investigate learning outcomes that go beyond simple test scores.

The review has also been able to identify areas for improvement in future stud-ies, which would address confounding variables and expand the scope of research. Firstly, as Allcoat and von Mühlenen (2018) suggest, the novelty of I-VR could hamper learning outcomes due to unfamiliarity with the technology. Therefore, it is important to factor in an extended familiarisation or free navigation period that would help alleviate this concern. Additionally, follow-up qualitative analysis such as interviews or focus groups could help explore the phenomenology or direct expe-rience of using I-VR, and highlight concerns relating to unfamiliarity or technologi-cal anxiety. The biggest concern relating to the assessment instrumentation was the over reliance on the MCQ (62% of studies used it as the sold method of assessment). Although this method is deemed appropriate for assessing large amounts of surface knowledge, it may not reveal more nuanced forms of learning that extend beyond mere recall of information. Therefore, long form essay questions, oral examinations, or group discussions could be used to facilitate students’ ability to present their in-depth understanding and applied knowledge. Future research must base the nature of these interventions on a sound theoretical framework. This would assist in identify-ing specific learning objectives and methods of assessments. An explicit theoretical approach was commonly lacking in the included studies.

I-VR has already been demonstrated to be an effective tool in non-pedagogical behaviour change, such as treating phobias, mental health conditions, or as a tool for rehabilitation (Botella et al. 2017; Maples-Keller et al. 2018; Ravi et al. 2017). Research should therefore concentrate on I-VR’s potential as an acquisition tool for affective skills. There is already a strong body of evidence suggesting I-VR experi-ences can elicit high levels of empathetic response and perspective taking, and this should be explored within an educational context (Herrera et al. 2018; Shin 2018). For example, Dyer et al. (2018) used I-VR to allow health care students to take the perspective of an older patient with age-related medical conditions, which led to increased empathy. Future studies should investigate whether this perspective tak-ing ability can lead to higher domains of learning, such as evaluating one’s actions, applying problem solving skills, or creating new solutions as a direct result of the insights they received from I-VR. This will require researchers and instructors to carefully consider their tools for evaluation and assessment, perhaps incorporating mixed-methods to give a more holistic overview of learning achieved.

Conclusions

The current review found that I-VR conferred a learning benefit in around half of cognitive studies, especially where highly complex or conceptual problems required spatial understanding and visualisation. Although many studies found no significant benefit of using I-VR over less immersive technology, only a small number resulted

27

1 3

J. Comput. Educ. (2021) 8(1):1–32

in detrimental effects on learning outcomes. However, the homogenous nature of assessment instrumentation, such as an over reliance on the MCQ may have stifled the ability for participants to demonstrate learning outcomes beyond low level cog-nitive knowledge. Short exposure times and isolated interventions could also pose a problem as the novel nature of the technology could negatively impact the amount of learning able to be imbibed. Encouragingly, most procedural tasks did show a benefit to utilising I-VR, and furthermore, there was evidence that virtual skill acquisition could be transferred successfully to real world problems and scenarios. The ability to repeatedly practice a procedure in a safe environment whilst expend-ing little resources could be one of the most advantageous and intrinsic benefits of I-VR technology. Although affective behavioural change has been widely studied in non-educational applications of I-VR, the domain was underrepresented in the cur-rent review, and is an important area for future investigation.

Over the coming years, technological advancement, an increase in creative con-tent, and the possibilities for instructors to create bespoke I-VR experiences will all contribute to I-VR’s potential as a teaching tool. It is essential therefore that the implementation of such technology is based on sound theoretical and experimental evidence in order to ensure that the I-VR is utilised correctly, and to its full potential.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Com-mons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creat iveco mmons .org/licen ses/by/4.0/.

Appendix

See Table 2.

28 J. Comput. Educ. (2021) 8(1):1–32

1 3

Table 2 MERSQI quality appraisal scores for included studies

Study (year) Design Sampling Type of data Validity of instrument

Data analysis Outcomes Total

Harrington et al. (2018)

3 2 3 1 3 1.5 13.5

Yoganathan et al. (2018)

3 2.5 3 1 3 1.5 14

Makransky et al. (2017)

3 2 3 2 3 1.5 14.5

Greenwald et al. (2018)

2 2 3 0 3 1.5 11.5

Lamb et al. (2018) 3 2 3 0 3 1.5 12.5Webster (2016) 2 2 3 1 3 1.5 12.5Smith et al. (2018) 2 2 3 1 3 1.5 12.5Ostrander et al. (2018) 3 2 3 0 3 1.5 12.5Angulo and de

Velasco (2013)2 2 3 0 2 1.5 10.5

Stepan et al. (2017) 3 2 3 1 3 1.5 13.5Sankaranarayanan

et al. (2018)2 2 3 1 3 1.5 12.5

Fogarty et al. (2017) 2 2 3 1 3 1.5 12.5Rupp et al. (2019) 3 2 3 0 3 1.5 12.5Bharathi and Tucker

(2015)3 2 3 0 3 1.5 12.5

Johnston et al. (2018) 2 2 3 0 3 1.5 11.5Allcoat and von Müh-

lenen (2018)3 2 3 1 3 1.5 13.5

Kozhevnikov et al. (2013)

3 2 3 2 3 1.5 14.5

Parong and Mayer (2018)

3 2 3 1 3 1.5 13.5

Olmos-Raya et al. (2018)

3 2 3 0 3 1.5 12.5

Akbulut et al. (2018) 2 2 3 0 3 1.5 11.5Ray and Deb (2016) 3 2 3 0 2 1.5 11.5Moro et al. (2017) 3 2 3 1 3 1.5 13.5Maresky et al. (2019) 3 2 3 0 3 1.5 12.5Madden et al. (2018) 3 2 3 0 3 1.5 12.5Liou and Chang

(2018)3 2 3 0 3 1.5 12.5

Molina-Carmona et al. (2018)

3 2 3 2 3 1.5 14.5

Babu et al. (2018) 3 2 3 0 2 1.5 11.5Alhalabi (2016) 2 2 3 0 3 1.5 11.5Gutiérrez-Maldonado

et al. (2015)3 2 3 0 3 1.5 12.5

Average 2.7 2.0 3.0 0.6 2.9 1.5 12.7

29

1 3

J. Comput. Educ. (2021) 8(1):1–32

References

Akbulut, A., Catal, C., & Yıldız, B. (2018). On the effectiveness of virtual reality in the education of soft-ware engineering. Computer Applications in Engineering Education, 26, 918–927.

Alaraj, A., Lemole, M., Finkle, J., Yudkowsky, R., Wallace, A., Luciano, C., … Charbel, F. (2011). Vir-tual reality training in neurosurgery: Review of current status and future applications. Surgical Neu-rology International, 2, 52

Alhalabi, W. S. (2016). Virtual reality systems enhance students’ achievements in engineering education. Behaviour & Information Technology, 35, 919–925.

Allcoat, D., & von Mühlenen, A. (2018). Learning in virtual reality: Effects on performance, emotion and engagement. Research in Learning Technology, 26, 2140.

Angulo, A., & de Velasco, G. V. (2013). Immersive simulation of architectural spatial experiences. Blucher Design Proceedings, 1, 495–499.

Babu, S., Krishna, S., Unnikrishnan, R., & Bhavani, R. (2018). Virtual reality learning environments for vocational education: A comparison study with conventional instructional media on knowl-edge retention. In 2018 IEEE 18th International Conference on Advanced Learning Technologies (ICALT) (pp. 385–389). IEEE.

Bailenson, J. N., Markowitz, D. M., Pea, R. D., Perone, B. P., & Laha, R. (2018). Immersive virtual real-ity field trips facilitate learning about climate change. Frontiers in Psychology, 9, 2364.

Bharathi, A. K. B. G., & Tucker, C. S. (2015). Investigating the impact of interactive immersive vir-tual reality environments in enhancing task performance in online engineering design activities. In ASME 2015 International Design Engineering Technical Conferences and Computers and Informa-tion in Engineering Conference. ASME.

Bloom, B. S., Englehard, M. D., Furst, E. J., Hill, W. H., & Krathwohl, D. (1956). Taxonomy of educa-tional objectives, handbook I: The cognitive domain. New York, NY: Longmans.

Botella, C., Fernández-Álvarez, J., Guillén, V., García-Palacios, A., & Baños, R. (2017). Recent progress in virtual reality exposure therapy for phobias: A systematic review. Current Psychiatry Reports, 19, 42.

Brown, A., & Green, T. (2016). Virtual reality: Low-cost tools and resources for the classroom. Tech-Trends, 60, 517–519.

Çalişkan, O. (2011). Virtual field trips in education of earth and environmental sciences. Procedia-Social and Behavioral Sciences, 15, 3239–3243.

Clark, R. E. (1983). Reconsidering research on learning from media. Review of Educational Research, 53, 445–459.

Dickey, M. D. (2005). Three-dimensional virtual worlds and distance learning: Two case studies of Active Worlds as a medium for distance education. British Journal of Educational Technology, 36, 439–451.

Dyer, E., Swartzlander, B. J., & Gugliucci, M. R. (2018). Using virtual reality in medical education to teach empathy. Journal of the Medical Library Association, 106, 498–500.

Ellinger, R. S., & Frankland, P. (1976). Computer-assisted and lecture instruction: A comparative experi-ment. Journal of Geography, 75, 109–120.

Excell, P. S. (2000). Experiments in the use of multiple-choice examinations for electromagnetics-related topics. IEEE Transactions on Education, 43, 250–256.

Fogarty, J., McCormick, J., & El-Tawil, S. (2017). Improving student understanding of complex spatial arrangements with virtual reality. Journal of Professional Issues in Engineering Education and Practice, 144, 04017013.

Fransson, G., Holmberg, J., & Westelius, C. (2020). The challenges of using head mounted virtual reality in K-12 schools from a teacher perspective. Education and Information Technologies., 2, 20–22.

Freina, L., & Ott, M. (2015). A literature review on immersive virtual reality in education: State of the art and perspectives. In The 11th International Scientific Conference ELearning and Software for Education (pp. 133–141).

Garrison, D. R., & Kanuka, H. (2004). Blended learning: Uncovering its transformative potential in higher education. Internet and Higher Education, 7, 95–105.

Gee, J. P. (2004). What video games have to teach us about learning and literacy. Education Training, 46, 20.

30 J. Comput. Educ. (2021) 8(1):1–32

1 3

Greenwald, S. W., Corning, W., Funk, M., & Maes, P. (2018). Comparing learning in virtual reality with learning on a 2D screen using electrostatics activities. Journal of Universal Computer Science, 24, 220–245.

Gutiérrez-Maldonado, J., Ferrer-García, M., Pla-Sanjuanelo, J., Andrés-Pueyo, A., & Talarn-Caparrós, A. (2015). Virtual reality to train diagnostic skills in eating disorders. Comparison of two low cost systems. Studies in Health Technology and Informatics, 219, 75–81.

Häfner, P., Dücker, J., Schlatt, C., & Ovtcharova, J. (2018). Decision support method for using virtual reality in education based on a cost-benefit-analysis. In 4th International Conference of the Virtual and Augmented Reality in Education, VARE 2018, 103–112.

Harrington, C. M., Kavanagh, D. O., Wright Ballester, G., Wright Ballester, A., Dicker, P., Traynor, O., … Tierney, S. (2018). 360° operative videos: A randomised cross-over study evaluating attentive-ness and information retention. Journal of Surgical Education, 75, 993–1000

Herrera, F., Bailenson, J., Weisz, E., Ogle, E., & Zak, J. (2018). Building long-term empathy: A large-scale comparison of traditional and virtual reality perspective-taking. PLoS ONE, 13, e0204494.

Hodgson, E., Bachmann, E. R., Vincent, D., Zmuda, M., Waller, D., & Calusdian, J. (2015). WeaVR: A self-contained and wearable immersive virtual environment simulation system. Behavior Research Methods., 47(1), 296–307.

Hodgson, P., Lee, V. W. Y., Chan, J. C. S., Fong, A., Tang, C. S. Y., Chan, L., et al. (2019). Immersive virtual reality (IVR) in higher education: Development and implementation. In M. Dieck & T. Jung (Eds.), Augmented reality and virtual reality (pp. 161–173). New York: Springer.

Howard-Jones, P., Ott, M., van Leeuwen, T., & De Smedt, B. (2015). The potential relevance of cognitive neuroscience for the development and use of technology-enhanced learning. Learning, Media and Technology, 40, 131–151.

Huang, H. M., Rauch, U., & Liaw, S. S. (2010). Investigating learners’ attitudes toward virtual real-ity learning environments: Based on a constructivist approach. Computers and Education, 55, 1171–1182.

Jeffries, P. R., Rodgers, B., & Adamson, K. (2015). NLN Jeffries simulation theory: Brief narrative description. Nursing Education Perspectives, 36, 292–293.

Jensen, L., & Konradsen, F. (2018). A review of the use of virtual reality head-mounted displays in edu-cation and training. Education and Information Technologies, 23, 1515–1529.

Johnston, A. P. R., Rae, J., Ariotti, N., Bailey, B., Lilja, A., Webb, R. I., … Parton, R. G. (2018). Journey to the centre of the cell: Virtual reality immersion into scientific data. Traffic, 19, 105–110

Kozhevnikov, M., Gurlitt, J., & Kozhevnikov, M. (2013). Learning relative motion concepts in immer-sive and non-immersive virtual environments. Journal of Science Education and Technology, 22, 952–962.

Lamb, R., Antonenko, P., Etopio, E., & Seccia, A. (2018). Comparison of virtual reality and hands on activities in science education via functional near infrared spectroscopy. Computers and Education, 124, 14–26.

Larsen, C. R., Oestergaard, J., Ottesen, B. S., & Soerensen, J. L. (2012). The efficacy of virtual reality simulation training in laparoscopy: A systematic review of randomized trials. Acta Obstetricia et Gynecologica Scandinavica, 91, 1015–1028.

Lee, C., & Shea, M. (2020). Exploring the use of virtual reality by pre-service elementary teachers for teaching science in the elementary classroom. Journal of Research on Technology in Education, 52, 163–177.

Lee, E. A. L., Wong, K. W., & Fung, C. C. (2010). How does desktop virtual reality enhance learning outcomes? A structural equation modeling approach. Computers & Education, 55, 1424–1442.

Lee, J. (1999). Effectiveness of computer-based instructional simulation: A meta analysis. International Journal of Instructional Media, 26, 71–86.

Li, L., Yu, F., Shi, D., Shi, J., Tian, Z., Yang, J., … Jiang, Q. (2017). Application of virtual reality tech-nology in clinical medicine. American Journal of Translational Research, 9, 3867–3880

Liou, W., & Chang, C. (2018). Virtual reality classroom applied to science education. In 2018 23rd Inter-national Scientific-Professional Conference on Information Technology (IT) (pp. 1–4).

Lyne, D. S. (2013). Development of virtual reality applications for the construction industry using the Oculus Rift head mounted display. In Proceedings of the 13th International Conference on Con-struction Applications of Virtual Reality (pp. 30–31).

Madden, J. H., Won, A. S., Schuldt, J. P., Kim, B., Pandita, S., Sun, Y., … Holmes, N. G. (2018). Vir-tual reality as a teaching tool for moon phases and beyond. In 2018 Physics Education Research Conference.

31

1 3

J. Comput. Educ. (2021) 8(1):1–32

Makransky, G., Terkildsen, T. S., & Mayer, R. E. (2017). Adding immersive virtual reality to a science lab simulation causes more presence but less learning. Learning and Instruction, 60, 225–236.

Maples-Keller, J. L., Bunnell, B. E., Kim, S.-J., & Barbara, R. (2017a). The use of virtual reality technol-ogy in the treatment of anxiety and other psychiatric disorders. Harvard Review of Psychiatry, 25, 103–113.

Maples-Keller, J. L., Yasinski, C., Manjin, N., & Rothbaum, B. O. (2017b). Virtual reality-enhanced extinction of phobias and post-traumatic stress. Neurotherapeutics, 14, 554–563.

Maresky, H. S., Oikonomou, A., Ali, I., Ditkofsky, N., Pakkal, M., & Ballyk, B. (2019). Virtual real-ity and cardiac anatomy: Exploring immersive three-dimensional cardiac imaging, a pilot study in undergraduate medical anatomy education. Clinical Anatomy, 32, 238–243.

Mayer, R. E. (2009). Multimedia learning (2nd ed.). Cambridge: Cambridge University Press.Mayer, R. E. (2014). Cognitive theory of multimedia learning. In R. E. Mayer (Ed.), The Cambridge

handbook of multimedia learning (2nd ed.). Cambridge: Cambridge University Press.Merchant, Z., Goetz, E. T., Cifuentes, L., Keeney-Kennicutt, W., & Davis, T. J. (2014). Effectiveness of

virtual reality-based instruction on students’ learning outcomes in K-12 and higher education: A meta-analysis. Computers and Education, 70, 29–40.

Molina-Carmona, R., Pertegal-Felices, M., Jimeno-Morenilla, A., & Mora-Mora, H. (2018). Virtual real-ity learning activities for multimedia students to enhance spatial ability. Sustainability, 10, 1074.

Moro, C., Štromberga, Z., Raikos, A., & Stirling, A. (2017). The effectiveness of virtual and augmented reality in health sciences and medical anatomy. Anatomical Sciences Education, 10, 549–559.

Murcia-López, M., & Steed, A. (2016). The effect of environmental features, self-avatar, and immersion on object location memory in virtual environments. Frontiers in ICT, 3, 1–10.

O’Dwyer, A. (2012). A teaching practice review of the use of multiple-choice questions for formative and summative assessment of student work on advanced undergraduate and postgraduate modules in engineering. The All Ireland Journal of Teaching and Learning in Higher Education, 4, 1–12.

Olmos, E., Cavalcanti, J. F., Soler, J.-L., Contero, M., & Alcañiz, M. (2018). Mobile virtual reality: A promising technology to change the way we learn and teach. In S. M. Bell (Ed.), Mobile and ubiqui-tous learning: An international handbook (pp. 95–106). Singapore: Springer.

Olmos-Raya, E., Ferreira-Cavalcanti, J., Contero, M., Castellanos, M. C., Giglioli, I. A. C., & Alcañiz, M. (2018). Mobile virtual reality as an educational platform: A pilot study on the impact of immersion and positive emotion induction in the learning process. Eurasia Journal of Mathemat-ics, Science and Technology Education, 14, 2045–2057.

Osborne, R., & Wittrock, M. (1985). The generative learning model and its implications for science education. Studies in Science Education., 12, 59–87.

Ostrander, J. K., Tucker, C. S., Simpson, T. W., & Meisel, N. A. (2018). Evaluating the effectiveness of virtual reality as an interactive educational resource for additive manufacturing. In Volume 3: 20th International Conference on Advanced Vehicle Technologies; 15th International Confer-ence on Design Education, V003T04A018. ASME.

Ozuru, Y., Briner, S., Kurby, C. A., & McNamara, D. S. (2013). Comparing comprehension measured by multiple-choice and open-ended questions. Canadian Journal of Experimental Psychology, 67, 215–227.

Parong, J., & Mayer, R. E. (2018). Learning science in immersive virtual reality. Journal of Educa-tional Psychology, 110, 785–797.

Pritchard, A. (2017). Ways of learning: Learning theories for the classroom (4th ed.). London: Routledge.

Radianti, J., Majchrzak, T. A., Fromm, J., & Wohlgenannt, I. (2020). A systematic review of immer-sive virtual reality applications for higher education: Design elements, lessons learned, and research agenda. Computers and Education, 147, 103778.

Ravi, D. K., Kumar, N., & Singhi, P. (2017). Effectiveness of virtual reality rehabilitation for children and adolescents with cerebral palsy: An updated evidence-based systematic review. Physiother-apy, 103, 245–258.

Ray, A. B., & Deb, S. (2016). Smartphone based virtual reality systems in classroom teaching - A study on the effects of learning outcome. In 2016 IEEE Eighth International Conference on Technology for Education (T4E) (pp. 68–71).

Reed, D. A., Cook, D. A., Beckman, T. J., Levine, R. B., Kern, D. E., & Wright, S. M. (2007). Asso-ciation between funding and quality of published medical education research. JAMA, 298, 1002.

Rios, J., & Wells, C. (2014). Validity evidence based on internal structure. Psicothema, 26, 108–116.

32 J. Comput. Educ. (2021) 8(1):1–32

1 3

Rupp, M. A., Odette, K. L., Kozachuk, J., Michaelis, J. R., Smither, J. A., & McConnell, D. S. (2019). Investigating learning outcomes and subjective experiences in 360-degree videos. Computers & Education, 128, 256–268.

Sankaranarayanan, G., Wooley, L., Hogg, D., Dorozhkin, D., Olasky, J., Chauhan, S., … Jones, D. B. (2018). Immersive virtual reality-based training improves response in a simulated operating room fire scenario. Surgical Endoscopy and Other Interventional Techniques, 32, 3439–3449

Schunk, D. H. (2011). Learning theories: An educational perspective (6th ed.). London: Pearson.Schutte, N. S., & Stilinović, E. J. (2017). Facilitating empathy through virtual reality. Motivation and

Emotion, 41, 708–712.Sherman, W. R., & Craig, A. B. (2003). Understanding virtual reality: Interface, application, and

design. In Understanding Virtual Reality: Interface, Application, and Design.Shin, D. (2018). Empathy and embodied experience in virtual environment: To what extent can virtual

reality stimulate empathy and embodied experience? Computers in Human Behavior., 78, 64–73.Slater, M. (2018). Immersion and the illusion of presence in virtual reality. British Journal of Psy-

chology, 109, 431–433.Smith, S. J., Farra, S. L., Ulrich, D. L., Hodgson, E., Nicely, S., & Mickle, A. (2018). Effectiveness

of two varying levels of virtual reality simulation. Nursing Education Perspectives, 39, 10–15.Stepan, K., Zeiger, J., Hanchuk, S., Del Signore, A., Shrivastava, R., Govindaraj, S., et  al. (2017).

Immersive virtual reality as a teaching tool for neuroanatomy. International Forum of Allergy and Rhinology, 7, 1006–1013.

UKAuthority. (2019). VR and AR attract education sector interest. Retrieved December 16, 2019, from https ://www.ukaut horit y.com/artic les/vr-and-ar-attra ct-educa tion-secto r-inter est/

Webster, R. (2016). Declarative knowledge acquisition in immersive virtual learning environments. Interactive Learning Environments, 24, 1319–1333.

Yoganathan, S., Finch, D. A., Parkin, E., & Pollard, J. (2018). 360° virtual reality video for the acqui-sition of knot tying skills: A randomised controlled trial. International Journal of Surgery, 54, 24–27.

Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

D. Hamilton is a PhD student in psychology at the University of the West of Scotland. His current research focuses on the application of immersive virtual-reality technology as a pedagogical method in higher education.

J. McKechnie is a Professor of Psychology at the University of the West of Scotland. His research focuses on national and international aspects of child workers and early experiences of the workplace. As a founder member of the Child Employment Research Group, his work has included government funded projects into the nature and extent and policy and practice relating to child employment

E. Edgerton is a Reader in Psychology at the University of the West of Scotland. He is an experienced researcher in environmental psychology and has played a leading role in a wide range of environment-behaviour projects including: environmental design within a psychiatric hospital, the role of the learning environment in the educational process and dementia friendly design in care homes. He was a Board member of the International Association of People-environment Studies (IAPS) from 2005 to 2014 and was President from 2012 until 2014

C. Wilson is a Lecturer in Psychology at the University of the West of Scotland. Her research interests lie predominantly within the field of social and educational psychology. Claire is interested in applying psychological theories most commonly applied in social psychology to educational problems. She has examined attitudinal and social cognitive predictors of behaviour, automaticity of behaviour, behaviour change interventions and has used cognitive tests to assess psychological constructs across a range of contexts including teachers’ inclusive behaviours for children with disabilities, bullying in schools and parent–child interactions