Electronic reading - My English Portfolio · type of item, eg shampoo, deodorant, drinks, frozen...
Transcript of Electronic reading - My English Portfolio · type of item, eg shampoo, deodorant, drinks, frozen...
LIBRARY www.library.unisa.edu.au
University of South Australia
Extract title The characteristics of multimodal texts: implications for the teaching of reading and writing
Extract author
Book title Evolving pedagogies : reading and writing in a multimodal world
ISBN 9781742003436
Citation details Ch. 2, pp. 21-46
Publication details Carlton South, Vic. : Education Services Australia, c2010.
Book author Bull, Geoff and Anstey, Michele.
Total pgs in book vi, 170 p. :
Edition
Electronic reading
COMMONWEALTH OF AUSTRALIA
Copyright Act 1968
Notice for paragraph 135ZXA (a) of the Copyright Act 1968
WARNING
This material has been reproduced and communicated to you by or on behalf of University of South Australia under Part VB of the Copyright Act 1968 (the Act).
The material in this communication may be subject to copyright under the Act. Any further reproduction or communication of this material by you may be the subject of copyright protection under the Act.
Do not remove this notice
Book extract
In this chapteral We define and examine the characteristics of multimodal texts and consider how
these characteristics influence the ways in which we consume, use and produce text.
Et We discuss the implications of these characteristics for teaching readingand writing and more particularly, how the Four Resource Model might bereconceptualised and used to facilitate the literacy teaching and learning process.
We define the elements of multimodal texts and how these elements cohere toproduce meaning
We examine how the reading and writing processes take place with multimodal texts.
E We consider how the Four Resource Model can be reconfigured to assist in planningfor teaching reading and writing with multimodal texts.
SECTION ONE
Defining multimodal texts
-The charOcteristics Qf .
,u.ltim
In this section, we introduce and explore six concepts about multimodal texts. As partof the exploration of each concept we have provided a Reflection Strategy to assist youin understanding and applying the concept. Each strategy will require engagement withthe two texts that you choose, therefore, before you read this section we suggest youacquire the two texts that you will use. Obtain copies of two advertisements for the sametype of item, eg shampoo, deodorant, drinks, frozen meals, junk food. One advertisementshould be on paper, eg magazine, newspaper or advertising billboard, and one should beretrieved from television or the internet. Both should be in a format you can view andexamine a number of times, eg the digital-electronic advertisement should be able to beviewed with and without sound, paused and played in slow motion; or if you are usingan advertising billboard, photograph it and reproduce it on an A4 page in full colour.
Website Reference 21Access Section 2.1 of the website for more detail.
If you are having difficulty finding items to use a useful website is www.bestadsontv.com wlshowcases TV, print, radio and interactive advertisements that can be viewed free of chargeand downloaded for a small fee.
Concept One: Multimodal texts convey meaning through a combination ofelements duddraw upon several semiotic systems.
When a text combines elements that draw upon two or more semiotic systems (t hatis, the linguistic, visual, gestural, audio and spatial) in order to convey meaning, i tis defined as multimodal. Examples of multimodal texts are a live ballet performancein which gesture, music, and space are the main elements; a picture book in wh ichthe elements of images and words are arranged in space on individual pages whichcontribute to an overall set of bound pages, or the home page screen of a website inwhich elements, such as sound effects, oral language, music and still or moving images,are combined. All these texts are multimodal, and each is delivered to the reader/viewer/consumer via a different medium or technology, that is, live (the ballet), paper (thepicture book) or digital-electronic (the website).
REFLECTION STRATEGY 2.1
The purpose of this strategy is to reflect upon the six concepts about multimodal textthat we introduce by comparing two multimodal texts on the same topic but deliveredvia a different medium or technology.
1 Analyse the two advertisements you have chosen in terms of the elements theyuse and the semiotic systems present. We have completed an example using caradvertisements in the sample analysis. You may wish to use a similar table butchoose a different product.
TABLE 2.1 Sample analysis
22 Evolving Pedagogies
Type of Text Elements Semiotic systems
Billboard
advertisement for
small hatchback
sedan
Image of car
Young people surrounding it and gazing at it admiringly
Caption identifying brand and model
Phrase describing car
Logo of brand
Visual
Visual, gestural, spatial
Linguistic
Linguistic
Visual
TV advertisement
for small
hatchback sedan
Sound effects
Pop music
Inaudible conversation and gasps from people
Voice-over
Sequence of images of car being driven byyoungattractive female through the restaurant and
entertainment area of a city
Groups of people looking at parked car and commenting
Caption identifying brand and model
Logo of brand
Audio
Audio
Audio and linguistic thovve)
linguistic is inaudible)
Linguistic
Audio and Linguistic
Visual, gestural, snotiril
Linguistic;
Visual
2 Identify the common and different elements and semiotic systems used. Why arethere differences?
3 Which elements and semiotic systems are dominant in each text and why do youthink this is so?
4 Consider how the use of a particular medium or mode of delivery (paper or digital-electronic) has afforded the advertiser different options in terms of delivering amessage about this product. Think about the implications of this when reading(consuming or using) or writing (producing) multimodal texts.
Concept Two: Multimodal texts draw upon and cross the boundaries of the arts,performance and design disciplines, their knowledge, understandings and processes.
When explaining Concept One, we drew upon three examples of multimodaltext: a ballet, a picture book and the home page screen of a website. What is alsointeresting about these three examples, and multimodal texts generally, is that theycross the boundaries between the disciplines of art, performance and design (Kressand van Leeuwen, 2001:1). In the examples given, each text draws upon the arts(consider the dance of ballet, the art of the picture book or still or moving images on thewebsite), performance (consider body position, facial expression and movement of theperformers in the ballet and people depicted in the images of the picture book or thestill and moving images on the website) and all three texts had to be carefully designedin terms of the combination of the elements within a space (stage, page and picturebook or screen). Proficient users or producers of multimodal texts need the knowledgeand ability to move across and between the boundaries of performance arts and design,accessing and using the knowledge and processes they afford to make and communicatemeaning.
Website Reference 2.2Access Section 2.2 of the website for more detail.
Links to two ballet performances (one modern, one traditional) are provided to enable you to consider
how the disciplines of arts, performance and design are displayed.
www.youtube.com/watch?v=910802X31
www.youtube.com/watch?v=n8CUDVpkGk4
REFLECTION STRATEGY 2.2
Re-examine the two texts you have chosen to analyse for this series of ReflectionStrategies.
1 Identify how each draws upon the disciplines of the arts, performance and design.2 Identify the differences between the two texts in terms of the disciplines they
draw upon and how they use them.3 Why have these differences occurred and how do (or don't) they contribute to the
success of each text?
Chapter 2 23
Concept Three: The role of language in multimodal text varies, is not aIwuv, (wi hintand is only one part of the whole.
The fact that multimodal texts combine elements and semiotic systems, .1 tit I oss
disciplines means that language (the linguistic semiotic system), vvhich has lur so longdominated meaning making, is now only one part of it (Jewitt and Kress,'2.008. it).In the three examples previously provided, one (the ballet) does not use language althe picture book may foreground the image over the words (language) and the !minepage website design may foreground images and symbols to capture the reader/v iew er'sattention initially before engaging with the less obvious linguistic elements. Consequ en I lyto engage productively with multimodal texts, proficient users or producers mus I notonly be competent in the linguistic mode but with all modes and their semiotic systems.They need to use this competence to make decisions about the appropriate selection,combination and use of modes and semiotic systems in order to communicate, use anddeliver information most effectively.
Concept Four: In a multimodal text, meaning is distributed across all elements (parts)and each element has a role in contributing to the overall (whole) meaning of the text.
The meaning of multimodal texts is distributed across all elements present in amultimodal text and each element or mode has a specific role or function in contributingto the overall meaning of the text; this is the partwhole concept. Kress and van Leeuvven(2006:20) pointed out that these roles might be complementary (eg in a picture bookwhere the picture shows a child's bedroom but the words lead you to look at the bedroomin a particular way), reinforcing (eg in a ballet when music and gesture both signify thesame emotion), or hierarchical (eg at another point in the ballet when the music fadesand [he action on the stage becomes the dominant signifier of meaning and the music issecondary). Proficient readers and viewers of multimodal text need to be able to identifyhow each mode is contributing to the overall meaning and attend to those that areproviding the most pertinent information at the time. They need to differentiate [heirattention while engaging with the multimodal text, attending to the modes that bestaddress the purpose of their reading or viewing at the time. Producers of multimodaltext need to be able to make informed decisions about how the role or function of eachelement will contribute to achieving their communicative purpose.
REFLECTION STRATEGY 2.3
Use the table you commenced for Reflection Strategy 2.1 as a basis for further reviewand analysis of the texts.
1. Identify the role of each element and semiotic system (part) in fulfilling the overallpurpose (whole) of this advertisement. In Table 2.2, we have completed theexample we started in Reflection Strategy 2.1 to demonstrate how this might bedone. In the example we started, the goal of the advertisement was clearly toshowcase the car and encourage people to buy it. However, once we analysed therole of each element and the specific contributions of the semiotic systems, webegan to see that the advertisement was directed very specifically at young, imagr.rtconscious females. In other words, our analysis of the roles of the elernent.-; dudsemiotic systems (parts) showed us how some meanings were being augmentedor reinforced by each mode and element contributing to one (whole) message, Itis important to note that we needed to be able to understand and use the specific:
grammars (codes and conventions) of each of the semiotic sysiern't iti rdei
analyse the texts and identify this overall message.
24 Evolving Pedagogics
TA
BLE
2.2
Ext
ende
d sa
mpl
e an
alys
is
Typ
e of
Tex
tE
lem
ents
Rol
e of
Ele
men
tS
emio
tic s
yste
ms
Bill
boar
d
adve
rtis
emen
t
for
smal
l
hatc
hbac
k
seda
n
Imag
e of
car
Dis
play
pro
duct
Vis
ual
Col
our
of c
ar is
pop
ular
am
ong
youn
g fe
mal
e dr
iver
s
You
ng p
eopl
e su
rrou
ndin
g it
and
gazi
ng a
t it a
dmiri
ngly
Indi
cate
that
pro
duct
is p
opul
ar w
ith
the
youn
g
Vis
ual,
gest
ural
, spa
tial
Gaz
e an
d bo
dy p
osi t
ion
of y
oung
is to
war
d ca
r, fa
cial
expr
essi
ons
indi
cate
adm
iratio
n an
d en
vy
Cap
tion
iden
tifyi
ng b
rand
and
mod
elIm
prin
t bra
nd m
emor
yLi
ngui
stic
Font
,co
lour
and
siz
e ha
ve y
oung
, fem
ale
appe
al, p
lace
men
t
acro
ss to
p fr
ames
imag
e
Phr
ase
desc
ribin
g ca
rR
einf
orce
bra
nd m
emor
yLi
ngui
stic
Pla
ced
at b
otto
m w
ith li
near
art
com
plet
ing
fram
e of
imag
e an
dle
adin
gto
logo
Logo
of b
rand
Rei
nfor
ce b
rand
mem
ory
Vis
ual
Pro
min
ent b
otto
m r
ight
for
mem
ory
TV
adv
ertis
emen
t
for
smal
l
hatc
hbac
k
seda
n
Sou
nd e
ffect
sE
nhan
ce d
ispl
ay o
f vis
ual f
eatu
res
of c
ar
Aud
ioQ
ualit
y of
sou
nd (
purr
ing
thro
aty
exha
ust)
attr
acts
atte
ntio
n,
indi
cate
s po
wer
but
is n
ot m
ascu
line
Pop
mus
icE
ngag
e po
tent
ial p
urch
aser
s of
car
Aud
ioS
tyle
of m
usic
mos
t pop
ular
with
you
ng fe
mal
es a
t pre
sent
Inau
dibl
e co
nver
satio
n an
d ga
sps
from
peo
ple
Indi
cate
pop
ular
app
eal t
hrou
gh s
ound
Aud
io a
nd li
ngui
stic
(how
ever
, lin
guis
tic
is in
audi
ble)
Alth
ough
the
wor
ds a
re in
audi
ble
the
pitc
h, p
ause
ano
'
into
natio
n in
dica
te a
dmira
tion
Voi
ce-o
ver
Aud
io id
entif
ies
feat
ures
of c
ar u
nabl
eto
be
show
n vi
sual
ly a
nd r
einf
orce
s
bran
d an
d m
odel
nam
e
Aud
io a
nd li
ngui
stic
You
ng fe
mal
e vo
ice
pitc
h, p
ace,
pau
se a
ndin
tona
tion
indi
catin
g
enth
usia
sm
Seq
uenc
e of
imag
es o
f car
bei
ng
driv
en b
y yo
ung
attr
activ
e fe
mal
e
thro
ugh
the
rest
aura
nt a
nden
tert
ainm
ent a
rea
of a
city
Indi
cate
loca
tion,
con
text
s an
d
segm
ent o
f pop
ulat
ion
to w
hich
car
is s
uite
d
Vis
ual,
gest
ural
, spa
tial
Cam
era
angl
es s
how
off
car's
vis
ual a
ppea
l, w
ide
angl
es a
nd
over
head
sho
ts s
tres
s at
trac
tive
loca
tions
and
eas
e of
ste
erin
g
thro
ugh
city
Gro
ups
of p
eopl
e lo
okin
g at
par
ked
car
and
com
men
ting
Vis
ually
rei
nfor
ce p
opul
ar a
ppea
lV
isua
l, ge
stur
al a
nd
spat
ial
The
cam
era
zoom
s in
on
youn
g fe
mal
e th
en fo
llow
s th
eir
gaze
to fe
atur
es o
f car
Cap
tion
iden
tifyi
ng b
rand
and
mod
elV
isua
lly r
einf
orce
bra
nd a
nd m
odel
mem
ory
Ling
uist
icFo
nt,
colo
uran
dsi
ze h
ave
youn
g, fe
mal
e ap
peal
, pla
cem
ent a
t
begi
nnin
g an
d en
d of
adv
ertis
emen
t seq
uenc
e fr
ames
imag
es
and
rein
forc
es b
rand
and
mod
el m
emor
y
Logo
of b
rand
Vis
ually
rei
nfor
ce b
rand
and
mod
el
mem
ory
Vis
ual
P/a
cem
ent a
t beg
inni
ng a
nd e
nd o
f adv
ertis
emen
t seq
uenc
e
fram
es im
ages
and
rei
nfor
ces
mem
ory
of b
rand
and
mod
el
(-2. Having completed your analysis of the role of the elements and semiatic -;ystemsin contributing to the overall meanina of your selected texts, analysa ihn role of
the semiotic systems in relation to one another. For example, identify tha semiotic
systems that are providing complementary, reinforcing, or hierarchical meanings.When we re-examined our sample analysis in this way we found that while allsemiotic systems fulfilled a reinforcing role, different combinations reinforceddifferent meanings or messages. For example, the visual, audio, spatial and gesturalsemiotic systems combined in elements of the advertisement that focused onappeal to young females; the audio, visual, linguistic, spatial and gestural combinedin elements that reinforced car features and only the visual and linguistic combined
in elements that reinforced branding.3. VVhen you have completed your analyses think about why particular elements and
semiotic systems have been chosen by the producers of the texts to reinforceparticular messages.
Concept Five: Different modes have different functional specialisations and are usedselectively in multimodal texts.
Time and space are also critical issues in multimodal text; they affect the roles of the
modes in a text and how we use or process them. Some multimodal texts, such as videoclips, movies, dance, or theatre; are dominated by time-based modes, that is elements ofmeaning presented over time. In these texts, speech, gesture, body movement and actionprovide meaning over time, rather than instantly, as the video clip or play unfolds. Othermultimodal texts, such as sculpture, a still image, an architectural arrangement, a page
or individual website screen are dominated by space-based modes that simultaneouslypresent all the elements for meaning making through the layout and organisation of theelements. Kress (2003:45) suggested that understanding more about how time-based andspace-based modes operate and convey meaning is important to the proficient readingand production of multimodal texts. Jewitt (2006:25) referred to this as the functionalspecialisation of modes, that is, we select, attend to and use modes because they are best
suited to the task we are trying to achieve. Therefore if you are limited to a screen or
page, you will choose space-based modes, but if you are using a technology that enablesyou to build up meaning over a series of episodes or scenes (such as video) then youwill draw on both space- and time-based modes and particularly the special featuresafforded by time-based modes (for example, building atmosphere through music).
Kress (2003:44) pointed out that the functions of some modes (eg gesture and writing)
are mixed (that is they can be effective in space and time) and a good example of this
is in the picture book. While each individual page is a space in which the layout of theelements is critical to meaning, these pages are bound into a book that has an overallmeaning and is examined over a period of time. On each individual page meaningis afforded through the gestures and language (writing) and over time we use thisinformation to build up the narrative. Thus gesture and writing are used effectivelyon each page to build up meaning over time, that is, they fulfil both space and timefunctions.
26 Evolving Pedagogies
REFLECTION STRATEGY 204
The paper conveyed text you have been examining is dominated by space-based
modes because all the information mus,t be processed simultaneously to make
meaning, however, the digital-electronically conveyed text uses both space- and time-
based modes.View the digital-electronically conveyed text and identify how different modes are
used over time to build up meaning. For example, in the sample that we analysed,
sound, which included music, sound effects, voice-over and inaudible language, vvas
carefully orchestrated to enhance particular meanings at particular times.
Concept Six: The meaning of a multimodal text is afforded through the cohesion and
coherence within and between its elements and the context in which it is used.
One of the complexities of multimodal texts is that in order to make meaning of
them we have to work out how the text operates or comes together to make meaning
at several levels. How the text operates or comes together is referred to as cohesion or
coherence. We are all familiar with the concept of cohesion in the grammar of the English
language, where we understand that cohesion may occur at the sentence level (eg using
conjunctions and pronouns), paragraph level (eg using repetition of key phrases) and
whole text level (eg through the use of familiar generic structures). However, we now
need to expand our understanding of this concept to include the cohesive devices of
other semiotic systems. In the visual semiotic system, the repetition of colour or the use
of vectors in an image are cohesive devices that lead the eye to particular parts of the
image and potentially shape the viewer's meaning-making.The cohesion that occurs within the elements and semiotic systems in a multimodal
text is one level of understanding about how the text works. The other level of
understanding that is necessary is the cohesion between the elements that make up the
multimodal text. The cohesion of these elements may be chronological as in a narrative,
or spatial when visual or diagrammatic material is presented (Bearne 2009).
In many multimodal texts, both spatial and chronological cohesion of elements
will occur. For example, in the video car advertisement we have been analysing in the
Reflection Strategies, the overall cohesion is chronological. The elements develop a
narrative around a girl driving a car and the reactions of people to the car. However, in
individual screen shots of the advertisement the cohesion of the elements is spatial as
the moving images are overlayed with the caption (name and model of the car) and the
company logo. The caption and logo elements of this overlay are organised in the most
effective spatial arrangement to ensure we attend to them and remember them. They
frame the moving image top and bottom. The logo is placed at the bottom right corner
because in Western society we read left to right and top to bottom, so that placement
ensures the logo is the last thing our eyes will alight on and remember. One of the
ways in which the overall cohesion between the elements in a video advertisement
can be ensured is through the development of a storyboard. This storyboard will show
each screen shot of the overall narrative sequence of the advertisement ensuring the
chronological cohesion of the narrative of the advertisement.
However, cohesion also occurs within the elements of a multimodal text, and this
is the other level of coherence we have to process. One or more semiotic systems may
contribute to the meaning of an element within the text. For example, in the driving
sequence of the car advertisement, the visual, audio, spatial and gestural semiotic
systems are employed and the linguistic is added with the voice-over. To fully understand
the effectiveness of the driving sequence, we need to understand how each semiotic
Chapter 2 ' 27
system contributes. This necessitates an understanding of then. eud ( I
and their cohesive devices. For example, when we watch the I ithe
advertisement, we need to understand how the codes and coliveld1. ; IIu ah
semiotic system are realised in moving images through devices, so, I ,/ .it'uu uIìul,wide angles, and point of view. We need to think about how II tese I wen
combined to showcase the car.The producer of the advertisement has achieved this by onsoriiir h,, u lieyin izird
identifies each screen shot for the driving sequence, has detailed ilo.c!: i lu 1;InieraLohesionangles to be used, points of view, etc. This results in an advertisement o w111,.Ii
is developed through camera shots that move smoothly from one 10 iIi (,t her, loading
the viewer's eye to the areas of the car to be showcased. In this exri)1,!, 1,,,,wHige and
use of the contributing semiotic systems together with the use ()II:;.(ii (insured
cohesion within and between the elements in this advertisement:.
It must be remembered, however, that this advertisement was d :led ;It young,
image-conscious urban females. Therefore, all the decision making ond design was
influenced by the advertisement's purpose, audience and the con t,eX. w vIlic11 it would
be used. The meaning making, cohesion and coherence is ultimately depend enl. upon
audience and context.
REFLECTION STRATEGY 2.5
Analyse how the cohesion between and within the elements of the tc,ubeen analysing in Reflection Strategies 2.1 to 2.4 works.
1 Is there chronological or spatial cohesion between the elements and how (loo.. I he
cohesion between the elements contribute to meaning making?
2 What semiotic systems are operating within the elements and whot v. the
relationship between and within them?3 How well does the cohesion of the elements and semiotic systems oddiel~. Hie
purpose, audience and context of the text?
TABLE 2.3 Key concepts about multimodal texts
As we discussed in Chapter 1, design is a key concept dnd
nowhere is it more evident than in the production and consul-111)i. il)ii loxls.
The summary of key concepts about multimodal texts prw-ierile(1 u .111)1i' A ill(iii1;110S
28 Evolving Pedagogies
''( iii IW.IVe
nts tInt (11.1W upon :!,c-!vori.ilConcept One Multimodal texts convey meaning through a combination of den
semiotic systems.
Concept Two Multimodal texts draw upon and cross the boundaries of the arts,
disciplines, their knowledge, understandings and processes.
Concept Three The role of language in multimodal text varies, is not always domi
the whole.
Concept Four In a multimodal text, meaning is distributed across all elements (1
role in contributing to the overall (whole) meaning of the text.
Concept Five Different modes have different functional specialisations and are 1
texts.
Concept Six The meaning of a multimodal text is afforded through the cohesi(,
between its elements and the context in which it is used.
that multimodal texts are the deliberate and strategic combination of elements to achievea particular communicative function in a particular context at a particular time: they areall about design. Consequently our effective use of these te-xts is contingent upon ourunderstanding and use of design.
As indicated in Concept Six, design when applied to multimodal texts does not simplyrefer to the arrangement or coherence of content and eleme-nts but to the influence ofthe social and cultural contexts in which the text is produced and consumed. Kress(2003:49) stated:
Design does not ask 'what has been done before, how, for whom, with what?' Designasks 'what is needed now, in this situation, with this configuration of purposes,aims, audience and with these resources and given my interest in this situation?'(his emphasis)
These questions apply equally whether producing (writing) or consuming (reading)a text because engagement in design will occur. Regardless of whether producing orconsuming text, consideration needs to be given to:
the purpose of the reading or writingwhat the audience or reader knows and brings to the textthe context or conditions in which the text is, or will be, read, used or producedthe resources available to use when reading or writing the text.
Having considered these issues, we can then design a text or design our approach toreading the text we have to be more strategic and reflective about how we engage withmultimodal texts.
Walsh suggested that the whole approach to learning was being changed bymultimodal texts, because of their complexity and the role of design in how readersand writers engage with them. She concluded that reading, writing and learning withmultimodal texts required greater metatextual awareness and the use of metacognitiveskills (Walsh 2007: 48). Our investigations in this section confirm this.
THEORY INTO PRACTICE 2.1It is possible to adapt any of the Reflection Strategies in Section One to classroominvestigations, by selecting texts appropriate to the age and interest of your studentsand limiting the investigations to their conceptual capabilities, that is, simplifying themvvhere necessary.
How multimodal texts have reshaped reading
SECTION TWO
After a detailed analysis of the process of reading a monomodal paper text (linguistic) amultimodal picture book text (linguistic and visual) and a multimodal digital-electronicwebsite text (visual, audio, linguistic, spatial), Walsh (2006:32) concluded that allrequired meaning making or interpretation of signs. The differences she identifiedin reading these texts concerned the processing of the modes and how these modesactivated the meaning making of the reader.
Chapter 2 29
Bearne (2009:163-164) suggested that in order to process multimodai lexls we need
to consider:
the elements presentthe role or contribution of each element to the overall meaning of the textthe medium by which they are conveyed through time and spacethe weight or influence of the modes which are conveyed in the mediumthe modes' patterns of cohesion and how emphasis is created within them.
Walsh and Bearne's work indicated the need for a reader of multimoda I texts lo beable to choose a reading path and identify and impose what Kress (2003160) referredto as criteria of relevance. These criteria of relevance relate to the reader's purpose inaccessing the text and the characteristics of the text itself. In other words, reading amultimodal text is about continually having a conversation with oneself about:
what to attend tohow to make sense of itwhat to ignoreidentifying the relationships and links that are relevant, not only among the coo lent
but among the elements and modes presenthow the whole text is making sense.
Reading multimodal texts requires sophisticated metacognitive work. However, allthis thinking, sense-making and metacognition relies upon very specific knowledge aboutthe affordances of the medium by which the text is conveyed (paper, live and d igital-
electronic), the modes used, the semiotic systems drawn upon and how tbee systemsuse time and space. Therefore it is essential students have a detailed 111)(101!-;1 and ing of
what semiotic resources can achieve (their affordances)
1. Affordances of the medium: what is made possible oravailable given the roles of space and timeTexts delivered by paper, live and digital-electronic technologies offer differentopportunities and experiences to the reader and are therefore processed differently.These differences are not only in terms of the modes and semiotic systems available butthe physical and sensual experience of engaging with them, which in Urn influences
reading.
Affordances of texts delivered by paperThe affordances of texts delivered by paper appear more limited thaii t hose deliveredlive or by digital-electronic means, and certainly one's reading pal 11 is limited toconsideration of words and images. Therefore the complexily ut r eInig i1 conveyedby paper would appear to be more dependent on familidrily wait) die Lon telit andits terminology and ability to process the visual and lingtilsus Si.:111ii)i 1(1 sysl ems. As
we discussed in Chapter 1, while paper appears to offer only two setiults systems
(linguistic and visual), it may be read aloud and we may sill........nvoealnw i hei/r i II our heads
what we are reading if it is not read aloud. Therefore the and in itsiid lc - y-;l(,:!1» is
available, if not physically present, through the tecIiiudt)i.,.,y ()I 1.1 1.;esturai
and spatial semiotic systems also contribute to Lexts ni vi \s t ),Iitiu. I hrot.0.1 the
images and the layout and organisation of page ;Ind I)(.)()1. t l I t i i i j., and
complexity of paper texts are not as simple as they may tie..1 ppa b-,1,. i sud bypaper tech-nology also offer sensual experiere:esds lii lid paper andink (for example, thick, heavy, textured, thin or stuiv iii i is I h.,1 I I. These
experiences together with the layout of the conteld tm I a wslhetic
30 Evolving Pedagogies
present. For example, we may unconsciously accord more authority to a full colourdocument on good quality shiny paper than a poorly photocopied black and whitedocurnent on thin paper. Our values as well as aesthetics may also influence response,for example we might feel more positive towards the authors of a document printed onboth sides of recycled paper.
The physical affordances of paper relate to how it is assembled and the context inwhich it is used at the time. Pages assembled conventionally can be turned back andforth to aid meaning making, for example to review information. Advances in printingtechnology and reductions in printing costs now mean that in many texts conveyedby paper, flaps, folding, half pages, cut outs and overlays afford other opportunitiesand reading paths. Advertising, pamphlets and postmodern picture books are goodexamples of how these advances in technology have made the reading paths of textsconveyed by paper more complex. Nevertheless paper documents generally offer morelimited reading paths than texts delivered via live or digital-electronic technologiesas the conventions organising paper documents are more rigid and the signals clearand usually conventional. For example, the linguistic and visual text on the page willbe organised in linear fashion, top to bottom and left to right, columns may be used,images or diagrams will usually have captions and reading of the linguistic text will beguided by the conventions of punctuation and formatting (font, size, colour and styleunderline, bold or italics). The visual text will be limited to still images and diagrams.Text delivered on paper is dominated by the affordances of space rather than time. Weprocess the elements of each page of text simultaneously as they appear on the page,although longer texts do need to be processed over time.
It is important to note that not all text delivered via paper will follow the moretraditional conventions of layout and organisation. There is more and more crossoverbetween the layout of text and screen (digital-electronic texts) as texts are produced forsimultaneous or shared transmission via different technologies. Recently this has becomea feature of postmodern picture books (as discussed in Chapter 3) making postmodernpicture books an excellent vehicle for investigating some of the characteristics ofmultimodal texts and how they influence reading paths and meaning making. Thisintertextuality, or borrowing of layouts and organisation from one medium to another,means that readers have to be prepared to read unconventionally at times. It would befoolish to think that texts delivered via paper have not evolved and require less activeprocessing than other multimodal texts. Clearly our ability to use the linguistic andvisual semiotic systems, particularly still images, will influence our reading processand understanding, but we must be competent with the other three semiotic systems inorder to use all the affordances of texts conveyed by paper.
Affordances of texts delivered liveTexts delivered live afford similar and different opportunities and experiences to thereader/viewer to paper texts. Clearly they afford more sense rich and physical experiencesbecause they involve live face-to-face interaction and have the potential to draw uponall five semiotic systems. Sight, touch, hearing and smell may all be part of makingmeaning with a text delivered live or face-to-face. Physical proximity, movement andattitude will also influence meaning making in these situations.
Texts conveyed live may afford the opportunity or expectation of participation orinteraction by the reader/viewer. A performance, such as a play, oral presentation, balletor show, may not require or offer the opportunity of participation, except in terms ofthe social conventions of showing appreciation at particular points, for example,clapping, calling for encore or making appreciative sounds, such as whistles or cheers.
Chapter 2 31
However, sometimes an oral presentation may offer the opportunity to participate byasking questions or participating in activities as part of the presentation.
Everyday use of texts delivered live and conducted as part of our social, commercialand municipal lives necessitate participation by the reader/viewer in order to getthings done (for example, conversation, discussion, interactions at shops, banks andworkplaces). Once again, reading, viewing and participation will be shaped by purpose,but the affordances of text conveyed live will offer the following to aid meaning making.There vvill be oral linguistic and non-linguistic sound, words and grunts or other non-verbal but meaningful utterances.
Other audio components may include sound effects and music. Gesture, includingfacial expression and body position, will also influence meaning making and spacebetween participants in the text will influence meaning. Consequently there are manymore reading paths offered in text delivered live than in a text conveyed via paper andmany more elements to process. Decisions about what to attend to and the developmentof criteria for making these decisions when engaging with the text will be essential andcomplex. Furthermore a text delivered live operates over time: meaning making doesnot depend on the immediate and simultaneous processing of all elements. One of themore interesting aspects of texts conveyed live is that they are often conducted with oraround texts delivered by paper or digital-electronic means. This adds an even morecomplex dimension to meaning making, as decisions about the role of the other texts aspart of the interaction around the text conveyed live, are ascertained and played out. Thepresence of other texts immediately offers many other opportunities for differentiatingattention and meaning making that need to be considered.
REFLECTION STRATEGY 2.6
The purpose of this Reflection Strategy is to consider the decision making that takesplace when one is engaged with a text delivered live together with either paper ordigital-electronically conveyed texts and to reflect on the complexity of processingmeaning in such a situation.
Identify a recent situation in which you were required to engage with a textdelivered live (eg a conversation or transaction) together with text conveyed viapaper or digital-electronic means in order to achieve your outcome. For example, atransaction at a bank or service centre that required you to fill in a form as well asconduct a conversation with the service representative, or a situation where you gaveor received directions or instructions live but also used texts conveyed by paper ordigital-electronic means.
1 Briefly construct a timeline of the situation and what happened at each stage.2 At each stage of the timeline identify the technology by which thet texts were
transmitted (live, paper, digital-electronic).3 Identify the semiotic systems used at each stage.4 Look at the graphic representation of the situation and circle or artnotdt, to
indicate what you attended to when and the reasons for your attention rt thattime. For example when was the image on the text delivered via papei locti:-; of
your attention, when were aspects of the live interaction the focus Ow Irdcningand speaking, the gestures or facial expression?
5 Did you understand everything at all times? Use a highlighter In inch( ,rjeswhere things vvere not clear, that is, when your processing of 011 Ow .1.,,111,;ble
reading paths was not leading to effective meaning making 01110 orprevented understanding?
32 Evolving Pedagogies
In a traditional English or literacy curriculum, the role of texts conveyed live hasoften been referred to as the listening and speaking part of the curriculum, an areaoften neglected or focused on performance (eg plays, poetry reading and oral reportsand presentations) rather than everyday situations. However, texts delivered live thatinclude listening or speaking are part of almost every aspect of our lives and occur withand around other texts. Therefore it is crucial that the affordances of text deliveredlive (including listening and speaking) be actively analysed and taught as part of amultimodal approach to teaching reading and writing.
Affordances of texts delivered via digital-electronic meansTexts transmitted digital-electronically share the affordances of texts delivered liveand via paper. They may offer opportunities for interaction with others, for examplethrough gaming and social networking, or the opportunity to interact with and createnew texts. Texts conveyed digital-electronically use both space- and time-based modes.While the screen affords similar space-based characteristics to a page of paper, it mayalso provide the opportunity to access moving images and sound. The links availablemean that reader/viewers are required to construct relationships over time, not simplybetween content, but between screens and modes as they move from still images tovideo; processing visual, gestural, linguistic, audio and spatial semiotic systems. Thereare many reading paths available and decisions about which to take will be based, notsimply on the reading purpose, but by developing criteria that enable identification ofrelevance among the many possible links and elements. The opportunity for distractionand loss of focus is much greater than in texts delivered by paper.
The ability to use texts conveyed by digital-electronic means is dependent not onlyon facility with all the semiotic systems and modes but familiarity, comfort with andability to use the hardware and software of the digital-electronic technology beingused. As we discussed in Chapter 1, research such as that of Dragona and Handa (2000)indicated that gender, age and experience influence attitudes and facility with digital-electronic technology and therefore the ability to process and make meaning of the textsthey convey.
Similar to texts delivered live and via paper, the senses are used when processingtexts conveyed by digital-electronic technologies. Tactile senses are used to operateconsoles, mouse and keyboard but probably do not invoke the same reactions oraesthetic and emotional associations as the quality of paper. Hearing is used to processsound effects, music and language. Sight is used in very sophisticated ways as eyemovement, scanning and focus is used differently on screens, and with still and movingimages. Therefore processing texts conveyed digital-electronically requires decisionmaking about how to use one's senses as well as making links and sifting the relevantinformation from the irrelevant. While the stereotypical image of people who work withdigital-electronic technology such as computers is that they do not engage in physicalaction, in fact, skills of visual analysis and rapid and focused eye movement, manualdexterity and even physical movement are a part of many leisure pursuits with textsconveyed by digital-electronic technology. Consider the physicality of products suchas Playstation and Wii, in which hand-held game consoles may require both manualdexterity and efficient whole body movement. Reading texts conveyed by digital-electronic technology is therefore a semiotic and physiological act that engages allthe senses.
Jewitt (2006), Gee (2004 and 2008) and Kress (2003) have pointed out that computergames require sophisticated and quick decision making based on rapid processing ofinformation from multiple modes, semiotic systems and senses. This indicates a need
Chapter 2 33
to review the ways in which we construct and teach reading and learning to studentswho engage in these activities as part of their everyday leisure. Kress and Gee are notsuggesting that because our students engage in these activities successfu Ely they donot need to be taught about them and how they might be applied in other settings(eg in learning, school and workplaces). They are suggesting that we need to know moreabout the skills and processes students have learned to apply in settings other thanthe classroom, so that we can be more effective and relevant in our teaching of literacyand learning.
2. Affordances of the semiotic systems: their cclas andconventions, their use of space and timeIn order to read and make meaning of a multimodal text, the reader must understand thecodes and conventions of the semiotic systems within the text and how they contributeto meaning making. Kress and van Leeuwen (2006:15), drawing on the work of Halliday,suggested that it is useful to look at the function of the semiotic system in contributingto meaning making, that is, what type of information it is contributing. They reportedthat Halliday identified three functions:
The Ideational function is about representation how the grammar of the semioticsystem represents or portrays the state of affairs or events being communicated inthe text.The Interpersonal function is about relationships how the grammar of the semioticsystem represents or portrays relations between the participants within the text. It isalso about how the grammar of the semiotic system creates or shapes a relationshipbetween the viewer or reader of the text an.d what is being represented in the text.It is therefore, also about the relationship between the maker of the text and his orher audience.The Textual function is about composition how the elements of meaning portrayedthrough the semiotic systems in the text have been framed and placed in space andin relation to one another in order to shape meaning. It is about coherence among,and within, the elements of the text.
Understanding these functions is useful when considering the way in which thecodes and conventions of the semiotic systems assist meaning making, as it ensures thereader/viewer attends to all the possibilities of the text, that is, what is being represented,what relationships are being represented or shaped and how is it all coming together tomake meaning. We examine the application of the functions in more detail in Chapters4, 5 and 6.
The codes and conventions of the semiotic systems are the tools tha I enable thereader/viewer to work out the meanings of all the elements of the text and 111(111 considerhow they come together to make meaning. The codes of each semiotic system providea grammar and terminology that enable the reader/viewer to identify and dei.;cribe howattention is captured, how emphasis of particular elements is creaied iind thereforehow meaning is shaped. The auditory codes employed by a radio coinromitater includepitch, volume, pace, speed, intonation. When we listen to a radio broad ast er ha sporting event such as a football match, these codes are coin hi nctl iinninding toaccepted conventions in our social and cultural experience and Iliereforn \ a(;(:ordcertain meanings. For example, as the pitch climbs, the paceI I Io: v(.1 morerapid, the speed of the speech increases and intonation beconie-: n u I infer,
34 Evolving Pedagogies -
even i,vithout understanding the words or perhaps the game, that something excitingor significant is about to happen (for example a try is about to be scored, or a goalkicked). The radio broadcaster has effectively combined the codes and conventions ofthe audio semiotic system into an audio text that fulfils all three of Halliday's functions:the ideational function (what is happening), the interpersonal fuinction (how theparticipants in the game are relating to one another and how the commentator is shapingour relationship to the participants in the game), and the textual function (the audio textis a coherent sequence of descriptive narrative about the game).
In the television broadcast of a sporting event, the visual semiotic system ofmoving images is accompanied by the audio semiotic system of commentary. Visualsemiotic codes, such as camera angles, pace, close-up shots, long shots, top down,bottom up and eye level views, are edited by the producer to provide a visualrepresentation of the match. For example, fast-paced cutting between close-up and longshots increases excitement and the choice of camera angles shapes focus on specificaction or the setting in which it takes place. In addition, linguistic and symbolic textmay appear on the screen to augment the visual and auditory information, for example,a clock might show how many minutes have elapsed and numbers and symbols mayindicate a score.
As we read/view a television broadcast, we choose which semiotic system to attend todepending upon the information we want and which is providing the best informationat the time. For example, if watching a football match, as we view the close-up of agoal being scored we might check the time and the score provided on the screen inorder to calculate the likelihood of any further goals before the end of the game. Ifthe commentary has become unintelligible, we might concentrate more on the actionprovided through the moving images.
An important point to remember about the use of semiotic systems is that the easewith which we process the information afforded to us by the semiotic systems is relatedto our background knowledge that is our knowledge of the content and genre or type ofinformation being presented. In the situation we have just described, viewing a footballmatch for leisure, our familiarity with the genre of 'sports broadcast' and the content,(the game of football) means we can automatically process the codes and conventionsof the semiotic system.
However, when we are reading or viewing a topic and/or genre that is less familiar orwhere we are trying to learn or do something new, then our knowledge of the semioticsystems and our ability to use what they afford us becomes critical. In these situations,we will need to consciously employ knowledge about semiotic systems and how theyindicate cohesion and emphasis to help us process (read and understand) the text. Forexample, we may look for cohesive ties such as 'therefore', 'first', 'after' or 'before' in thelinguistic element of the text to aid processing the chronology of the text. When listeningto the audio element of a text, we might listen for changes in volume or pitch or pausesbecause we know that this is the way emphasis is indicated in speech and therefore thesechanges may help us identify important information in the text. Similarly, we might useknowledge about layout in the visual element of the text (for example foreground andbackground) to identify what is important in an image.
In summary, knowledge of the semiotic systems helps shape metacognitive activitywhen reading by providing information that aids choice of reading paths and informsdecisions about the parts of the text to focus upon.
Chapter 2 35
REFLECTION STRATEGY 2.7
The purpose of this Reflection Strategy is to develop more explicit understandingsabout:
the function (ideational, interpersonal and textual) of each element in a textthe role of each semiotic system and the codes and conventions of that semioticsystem in the texthow the elements and semiotic systems work together to shape meaning.
Overall the strategy is designed to reinforce the complexity of reading andprocessing a multimodal text and how understanding the affordances of each semioticsystem can assist in meaning making.
1 Find five minutes of a sport with which you are familiar in a format that you canview several times.
2 VVatch it once, as you would normally watch such an event, and list the majorevents or action.
3 View the excerpt again, each time attending to only one of the semiotic systems.Describe how each of the semiotic systems are used in the text, note any codesand conventions that you can identify that aid or influence your meaning making.Identify the function of the elements and semiotic systems in your processing ormeaning making.
4 Review your summary. Are there points at which one element is more useful ordominant or where they complement one another to clarify what is happening? Anexample is provided in Table 2.4. Note that italics highlight how space and time areutilised in the semiotic systems and modes to shape meaning.
THEORY INTO PRACTICE 2This activity could also be conducted in a more simplified fashion as a carefullysequenced whole-class discussion where students contribute to the joint constructionof the analytic table. It would be important to conduct the analyses yourself priorto engaging with the class to ensure you use the cortect terminology and cancarefully structure the discussion (guided discovery). However, beware that your prioranalysis does not result in the discussion becoming 'guess what's in the Leacher'shead'. Be open to students' additional observations, asking them tb oxplain their.reasoning.
Our discussions of the reading process in this section indicate that molt i od al textsreshape our reading in a number of ways. It is clearly a more demaoding process asthere are multiple elements, modes and semiotic systems present to 1)roces;. There aremore choices about what to attend to and therefore more decisions to be [wide aboutappropriate reading paths. These decisions have to be monitored as many ()I tliose textscomprise elements that are time based and therefore one's stratogìs und wndiuç. pathsmay need to change as one engages further with the text. The reading tn.sk itself willrarely take place in isolation in terms of people or place. In t eractioi I \A' i h lilnis (Jr withthe media itself may be demanded as part of the engageMelli ri 1;1 V Illt111 that
several texts are operating simultaneously (eg texts delivered I-.)v li :it'd 1 i,"11)! A' or live
36 Evolving Pedagogies
TA
BLE
2.4
Sam
ple
anal
ysis
of f
unct
ion
and
role
s of
sem
iotic
sys
tem
s in
a te
xt
Exc
erpt
from
a te
levi
sed
tenn
is m
atch
bet
wee
n F
eder
er a
nd N
adal
.
Eve
nt/A
ctio
nV
isua
l sem
iotic
sys
tem
(mov
ing
imag
e)F
unct
ion
Ling
uist
ic s
emio
ticsy
stem
(au
dio)
Fun
ctio
nLi
ngui
stic
sem
iotic
sys
tem
(prin
t on
scre
en)
Fun
ctio
n
Fed
erer
ser
ves
Nad
al r
etur
ns
Ret
urn
ball
goes
out
Nad
al w
alks
to
othe
r se
rvic
e
cour
t
Clo
se-u
p of
Fed
erer
's fi
rst s
erve
Long
sho
t of w
hole
cou
rtP
anni
ng to
sho
w w
here
bal
l
land
s ou
t
Clo
se-u
p of
Fed
erer
's s
econ
d
serv
e
Long
sho
t of w
hole
cou
rt
Pan
ning
to s
how
whe
re b
all l
ands
Cut
to c
lose
-up
of N
adal
's g
roun
d
stro
ke r
etur
nP
an fo
llow
ing
ball
Cut
to lo
ng s
hot s
how
ing
that
bal
l
goes
out
Clo
se-u
p of
Nad
al's
face
show
ing
his
gaze
mov
e to
war
d
Fed
erer
and
mou
th g
rimac
ing
in
frus
trat
ion
Slig
htly
wid
er a
ngle
sho
t sho
ws
who
le b
ody
post
ure,
sho
ulde
rs
slum
ped,
sla
sh o
f rac
quet
in a
ir
Idea
tiona
l: se
quen
ce o
fim
ages
ove
r tim
e sh
ow
wha
t hap
pens
Tex
tual
: The
sel
ectio
n
and
editi
ng o
f thi
s
sequ
ence
of s
hots
prod
uces
a ti
ght,
sequ
entia
l and
coh
esiv
e
pict
ure
of th
e ac
tion
that
took
pla
ce
Inte
rper
sona
l: U
se o
ftw
o se
mio
tic s
yste
ms
(vis
ual a
nd g
estu
ral)
draw
s at
tent
ion
to-
rela
tions
bet
wee
n
Nad
al a
nd F
eder
er a
nd
posi
tions
the
read
er/
view
er to
see
the
actio
n
from
Nad
al's
poi
nt o
f
view
Sof
t voi
ce q
uiet
ly
desc
ribes
ser
ve
Exc
lam
atio
n ab
out s
peed
and
accu
racy
of s
erve
(into
natio
n ris
es a
nd
volu
me
incr
ease
s)
Nor
mal
vol
ume
and
pace
resu
mes
and
des
crib
es
Nad
al's
app
aren
t
frus
trat
ion
with
Fed
erer
's
abili
ty to
ser
ve in
way
s
Nad
al c
anno
t ret
urn
Idea
tiona
l: te
lling
wha
tha
ppen
s an
d
Inte
rper
sona
l: po
sitio
ns th
ere
ader
/vie
wer
to fo
cus
on th
e
skill
of F
eder
er
Tex
tual
: A c
oher
ent a
nd c
lear
com
men
tary
. The
lang
uage
uses
app
ropr
iate
tech
nica
l
term
s, a
djec
tives
and
adv
erbs
in
desc
riptio
ns o
f act
ion,
iden
tifie
s
the
play
ers,
and
pro
vide
s a
com
plem
enta
ry a
dditi
on to
and
inte
rpre
tatio
n of
vis
ual t
ext
(mov
ing
imag
es)
at a
ppro
pria
te
times
Idea
tiona
l: te
lling
wha
tha
ppen
s an
d al
so In
terp
erso
nal:
indi
catin
g re
latio
ns b
etw
een
Fed
erer
and
Nad
al. P
ositi
onin
g
the
read
er/v
iew
er to
hav
e so
me
sym
path
y fo
r N
adal
's fr
ustr
atio
n
and
to fu
rthe
r ad
mire
Fed
erer
's
skill
Top
left
of s
cree
n:
Tab
le s
how
ing
scor
e by
set
and
gam
e
Spe
ed o
f ser
ve
flash
es o
n sc
reen
Rev
ised
gam
e sc
ore
show
n
Idea
tiona
l: P
rovi
des
spec
ific
info
rmat
ion
abou
t act
ion
taki
ng
plac
e
Tex
tual
: App
ears
at
the
appr
opria
te ti
me
toco
here
wel
l with
vis
ual
and
audi
o el
emen
ts.
Tab
le a
nd la
yout
adhe
res
to c
onve
ntio
ns
of te
nnis
sco
re n
otat
ion
and
is th
eref
ore
easi
ly r
ead
(coh
eren
t)
Spa
tial a
rran
gem
ent
(pla
cem
ent t
op le
ft
on s
cree
n) d
oes
not
inte
rfer
e w
ith m
ovin
g
imag
e
Inte
rper
sona
l:C
ontin
uous
upd
ate
of s
core
ove
r tim
e
indi
cate
s ch
angi
ng
pow
er r
elat
ions
betw
een
play
ers
(rei
nfor
ces
mes
sage
s
of v
isua
l and
aud
io te
xt)
and digital-electronic technologies). The role of live or face-to-face encounters as part ofpaper or digital-electronic reading or viewing in everyday life is increasing. This in turnreminds us that our background knowledge, ie what we bring to the text socially andculturally, together with the context in vvhich the reading takes place are also factorsthat readers need to be aware of and take into account as they read.
To be competent, readers of multimodal texts must know and understand howall semiotic systems work, both in isolation and when combined with each other. Theyneed metacognitive skills and metatextual skills they need to be aware of and ableto monitor their cognitive processes and be aware of and understand how texts work.Readers need to understand the role of background knowledge and the context inmaking meaning.
Kress (2003:50) suggested multimodal texts mean that reading is moving from readingthe world as told (reading as interpretation) to reading the world as shown, that is readingrequires the imposition of relevance and order on what is being shown. He suggestedthat when reading the world as shown, reading becomes a process of designing. This isbecause the reader has to take control of the process and impose hypotheses and rulesfor processing all that is afforded by the text. The reader has a role in designing thereading path and therefore controls the outcome of the reading.
SECTION THREE
How multimodal texts have reshaped writingOur discussion about how reading has changed provides insight into how multimodaltexts have reshaped writing. Just as the reader has more reading paths and decisionmaking, so writers or producers of text have more writing paths and decision making todo when composing or producing a text to achieve the desired communicative purpose.Writers/producers need to consider many factors in order to ensure the text fulfilsits ideational, relational and textual functions effectively. The opportunity to conveyinformation live, using paper or digital-electronic technologies (or some combinationof these), means the writer has more modes available and more semiotic systems fromwhich to select and combine. While consideration of purpose, audience, context andavailable resources will still be critical, the questions the writer or prod ucer mightconsider will be more complex. We will now discuss these areas of decision ma king.
How purpose, audience and context influence selectionof resourcesPurpose will not only have to be considered in terms of desired communicat lye outcomebut also the best way to convey it. Previously, when communication was dominatedby the word (linguistic semiotic system), selection of genre and la nguage W;IS mostimportant in terms of meeting purpose. However, now consideration has lo be givento whether some aspects of the communication are more suited to particular modes orsemiotic systems. For example, would images an.d diagrams or an oral explatialion bebetter when providing instructions, or is there opportunity to choose a technology ormedium of communication that can combine all these in an effective wavY
38 Evolving Pedagogies
When thinking about shaping the text to appeal to, and be understood by, theidentified audience, consideration must be given to the age, gender, socioculturalbackground and general literacy skills of the audience, together with their experiencewith digital-electronic hardware and software and their ability to process differentsemiotic systems. For example, as we have discussed previously, particular age groups,genders and cultures will be more experienced with and attuned to process and engagewith particular technologies (paper, live or digital-electronic) and may have a preferencefor particular semiotic systems. For some, the actual technology or medium of deliverymay impede their ability to process the information.
Context will need to be considered in terms of what is available for conveying the text.It is useless planning a PowerPoint® presentation if there is no suitable equipment inthe context where it will be used, the software is incompatible, or the electricity supplyis unreliable and subject to surges. Similarly, other forms of electronic communication,such as email and SMS, rely on the audience and context having reliable equipment andcoverage. These are real considerations in a country as large and diverse as Australia.In addition to this, consideration needs to be given to how the text will be accessedand used in particular contexts. Will it be mediated, not only by the technology, butalso by someone on site such as a teacher, presenter or employer or employee? Will oneperson use the text individually, or will it be used in a group? How many people will beengaging with the text at once? What sort of interaction might occur around the text?The answers to these questions, all of which are about context, will also inform theshaping of the text.
Selecting and weighting available resourcesOther decision making relates to selection and weighting of modes and semiotic systems,which are in turn influenced by selection of the medium or technology of delivery(paper, live or digital-electronic).
Kress (2003:51) pointed out that the selection of modes and semiotic systems caninfluence the meaning of a communication. The focus of attention may be changedand different aspects of knowledge may be emphasised because of the combinationof semiotic systems and which one is dominant. Bearne (2009:189) suggested that thewriter or producer of the text should consider what content or messages they want thereader/viewer to attend to and what would be useful to the reader/viewer in makingmeaning of the text. The writer/producer of the text should then select modes, semioticsystems and media that facilitate processing of that content and weight those modes andmedia that best facilitate the message, that is, make them dominant.
If the information or content is complex, then it may be appropriate to provide thereader/viewer with opportunity to review the information either by returning to it or byhaving it presented to them in another form. For example, if using the medium of paperor screen the same information could be provided through the linguistic and visualsemiotic systems (words plus image or diagram). If using a digital-electronic mediumto deliver film then there is also the opportunity to add voice-over, sound effects ormusic to clarify or draw attention to particular content. However, when putting thesesemiotic systems and modes together, the writer/producer has to consider where theemphasis or weight should be, and that will depend on the purpose, content, mediumand audience.
In order to demonstrate the complexity of writing a multimodal text and theconcepts we have been discussing, we composed two texts about the life cycle of a frog.
Chapter 2 39
We monitored our cognitive processes as we composed and have provided a descriptionof the decision making we engaged in and the reasons behind our choices. We havedescribed choices about semiotic systems, their relative weight or dominance in thetext, the selection and composition of images and language, the organisation and layoutof text. The projected audience for our texts is an eight-year-old school student. Thecontext or place of use we envisage is a year 3/4 primary classroom where studentsmight use the text independently, in groups or as a text mediated by the teacher whenreading it to the whole class.
Our knowledge of audience and context influenced our choice of language,organisation and layout of the texts. We chose to use conventional layout and organisationas the emphasis was on providing information and therefore this information shoul dbe easily accessible through familiar conventions and organisation.
When composing Text One, we identified two purposes: to describe the life cycleof the frog and to emphasise their importance in the environment. We chose to weightthe linguistic semiotic system more heavily as it was more suited to conveying thisinformation. Therefore in Text One, the linguistic is dominant, both in quantity (morewords than image) and in the organisation and layout of the page. The organisation andlayout is dominated by linguistic conventions rather than visual, there are headings andparagraphs, and the linguistic text comes first and takes up more space on the page. Thevisual semiotic system (in this case an image) is smaller (takes up less space overall),is placed after the linguistic text and does not provide any additional information. Itis a labelled illustration of the frog environment and includes pictures of frogs at thevarious stages of development referred to in the linguistic text. It represents visuallyonly parts of the linguistic text and does not provide information about relationshipsor sequence.
To fulfil our two purposes, the linguistic text not only provides facts about the life cyclebut also presents information about the relationship of frogs and the environment anda conclusion about their usefulness in monitoring the environment. This information ispresented in the first paragraph and re-presented in the final paragraph, reinforcing theimportance of this information. A convention of linguistic text is to repeat informationthat is important at key points in a text (in this case as an introduction and conclusion).Text One is presented in Figure 2.1.
In Text Two, our purpose was to present only information about the life cycle of thefrog, the emphasis being on the fact that it is a continuous cycle. Therefore, we decidedto make the visual semiotic system dominant in terms of space (more images than words)and the conventions of layout and organisation. The concept of a cycle was foregroundedin the text by using a circular diagrammatic layout to organise all informti on in the text.To further reinforce the cyclic nature of a frog's stages of development, we used arrowsto indicate the relationship of one stage to the next. Numbers were used lo indicate thepoint of entry into the cycle (1) and reinforce the progress of the cycle. In tl t is way, weachieved clear cohesive ties between all elements on the page. images were used toillustrate and foreground the main chauges in the frog at each stage of ils lile cycle. Weused the linguistic semiotic system for labels on the diagram, naming t lie stages andproviding additional information about time, diet and development,i1 t.haracteristicsthat could not be represented in a simple diagram. Because we clime to nlake t he visualsemiotic system dominant the language is concise and presenten in nnIn forn TI is textis presented in Figure 2.2.
40 Evolving Pedagogies
FIGURE 2.1 Text One: The life cyc/e of the frog
THE LIFE CYCLE OF THE FROG
Frogs are an important part of our habitat. Because they like to live in areas that have clean
water and do not have large amounts of pollution, their presence or absence can alert us about
the health of the environment. Their life cycle has five main sta9es.
Stage One: The Ea9
Frogs l9 their es in water in a slippery jellylike clump. There may be several thousand
es in the clump and this helps protect them as female frogs do not stay and look after their
es or young. Not all the es will survive because some will be eaten by fish, birds and other
water animals. Gradually the embryo in the egg forms organs and 9iliS and after 21 cla9s if
leaves the jelly and attaches itself to weed in the water.
Stage Two: The Tadpole
The embryo will become a tadpole within 3 cla9s to 3 weeks, depending on what tye of fro9 itwill become. Tadpoles eat ver9 small plants or algae and must hide from larger pond animals
that might eat them. They have very long tails that help them move around in the water.
Stage Three: The Tadpole Be9ins to Cliange
After about five weeks, the tadpole rows hind le9s and soon after it arows forelegs. Next,
their tails become smaller and lungs develop so they can live on land when they become adult
frogs.
Stage Four: Final clian9es
Gradually the tadpole begins to look more like a frog as its mouth widens, the tail becomes
mucb smaller, and its leas arow. It also learns how to use its lunas more effectively.
Stage Five: The Adult Frog
After eleven weeks, the frog is fully developed. Now it will live mostly on land and only swim
sometimes. It will begin to eat insects and worms. And eventually it will find a mate and the
whole cycle will loe9in a9ain. As 9ou can see, durin9 its whole life the frog depends on the pond
and land to provide shelter and food, so it is important for the frog that the area is clean and
not polluted. Otherwise they will not survive. This is why frogs are an important part of our
life; their presence tells us the environment is healthy.
Chapter 2 41
FIGURE 2.2 Text Two: Life cycle of a Frog
cmIcpeSWQdun, calle fro3 staampn&AO,
ce&F'
Mcd-i_kre
11 v\iee
levelce-s LArvs,+IAA kscleeear-s
"Embro leave_,sClUrv
4-ckclede.
\It_tr ko 3 weeks
Tqct pole. hotsloy-o +cal.Ect-t-s cal9qe,
lecci pole gro.asshori-e_r, e.ccl19(33z3(
Ectis play-tfs ctret qnwr-tql
REFLECTION STRATEGY 2.8
Strategy is to further reinforce concepts about decisiond weighting of elements, modes and semiotic systems
cing multimodal texts.
1 Review Texts One and Two and plan a text for the same audience and context onthe same topic but using a digital-electronic technology to convey the text. Forexample, you could plan a PowerPoint presentation including the actual slides andthe oral text that will accompany it, or you could plan a video clip. Before you beginidentify the communicative purpose and focus of the text.
2 Use a storyboard to help identify the sequence of content in the text, the semioticsystems used and their roles, the balance between them and at which points theywill enter and leave the text. For example, for the video you would plan what is ineach shot, the camera angles, auditory commentary, sound effects or music.
3 As you plan, try to be metacognitively aware of the decisions you make and thereasoning behind them. Make notes about the decision-making process, focusingon the reasoning behind the decision and how it is contributing to the text fulfilling
your purpose.
e processes we demonstrated in the description about how we wrote the twotexts and that you employ as you write this text are those that you should be
delling to students when teaching about writing multimodal texts. By engagingand reflecting upon the task you will be teaching to students, you become more
aware of the processes that need to be taught and practised. Thus your teaching and
modelling becomes more explicit.
The purpose of this Reflectionmaking and the balance anwhen composing or produ
Th
frogMO
in
42 Evolving Pedagogies
The analysis and discussions in this section indicate that composing or prodtici lig
text is not simply about juxtaposing modes (eg image, text and sound). It is also aboutthe design creating multiple interconnections between the modes and elements of thetext in order to create a sign or text that shares meaning with others. Some of theselinks and interconnections will be explicit, achieved by using the shared codes andconventions of the semiotic system that will be recognised by others. Others will beimplicit, potential links that may be identified because of what the reader/viewer bringsto the text, their purpose, knowledge or socioeconomic background. Ultimately, theproducer of the text cannot control what the reader/viewer will take from it, but theknowledge and conscious decision making the writer/producer brings to the task willresult in a text that influences the way the reader/viewer enters the text, the readingpaths taken and the particular parts of the text they to which they attend. This in turnwill have some influence on their meaning rnaking.
The Four Resource Model and multimodal textsPrior to considering the role of the Four Resource Model in assisting us with the readingand writing of multimodal texts, it is appropriate to reflect on what we now understandabout the design and composition of multimodal texts and how this influences ourproduction and consumption of them. This will help us understand what students needto know and be able to do in order to produce and consume multimodal texts andthus provide links to how the Four Resource Model can assist us with this. Figure 2.3provides a summary.
Our exploration and discussion in Sections Two and Three indicated that particularknowledge and understandings are necessary in order to read or write (consume orproduce) multimodal texts. Students need to know how texts are designed and producedand how the elements within them work individually and as a whole, that is, theyrequire metatextual knowledge. Reading or writing multimodal texts requires strategicthought, hypothesising, decision making and metacognitive processing in order to usethis metatextual knowledge effectively (see Figure 2.3).
The complexity of multimodal texts, their technologies, and constant change meansthe concept of reading and writing has moved from the passive interpretation and writingof signs or texts to active choice-making and ordering; taking control of, designing andtransforming meaning In passive reading and writing, the power is attributed to thetext. In active reading and writing, in imposing order and relevance, the power is in thereader or writer.
In Chapter 1, we reported the origins of the Four Resource Model; it arose from aconcern that any approach to the teaching of literacy should be flexible and able to copewith change. The technologies by which multimodal texts will be conveyed, and the waysin which multimodal texts will be designed, will continue to change; so the flexibilityof the Four Resource Model is very useful. The Four Resource Model also focuses onachieving a balance between resources (knowledge and skills) and practices (processes).This makes the processes of reading and writing more transparent and ensures studentssee reading and writing as strategic problem-solving processes, responsive to change.The focus on what you need to know and be able to do (knowledge, skills and processes)
SECTION FOUR
Chapter 2 43
FIGURE 2.3 The design and composition of multimodal texts
facilitates the development of metatextual knowledge, strategic thought, hypothesising,decision making and metacognitive processing, all necessary to reading and writing
multimodal texts.The concept of active reading and writing, of reading and writing as design, indicates
that knowledge, skills and processes beyond code breaking and meaning making are
required beyond simple interpreting and sign making or sign reading Decisions about
using or developing texts appropriately for particular audiences and purposes and, inparticular for contexts also require knowledge skills and processes, such as hypothesising,
monitoring and making judgements, that is, text user and text analyst resources.In Figure 2.4, we have illustrated how an active reader of multimodal texts might
move between the resources available for use as code breaker, meaning maker, text userand text analyst when reading a text over time. There is constant movement betweenthe four sets of resources, drawing upon those appropriate to fulfil the reading or writing
purpose at the time. The reader moves from one set of resources to another as needed,
that is, as he imposes order and relevance among the elements of the text, and fulfils hisreading purpose. It is important to note that resources are often used simultaneously.For example, when scanning the text to identify the elements present, the reader willalso be identifying known language and images therefore employing meaning-makingand code-breaking skills together. Similarly while conducting the scan, the reader isconsidering how the text might best be used (text user activity). Ultimately the reader
uses all this information to make a hypothesis or judgement about the best source of
information in the text, that is, where to focus attention (text analyst). The reader also
engages text analyst resources when monitoring his use of the text and decides that the
video is not relevant to his needs.
44 Evolving Pedagogies
Texts operate in time and space
Texts are designed to achieve particular communicative purposes.
The design of a text is influenced by each of the following on its own and in combination with one another:
purpose
audience or user
the context anci conditions in which the text is accessed or used
the technologies by which the text is conveyed
the availability, selection and use of resources available for the design of the text.
Resources can include:
elements or modes, such as speech, sound, writing, or image
semiotic systems, their codes and conventions (linguistic, audio, visual, gestural and spatial).
Coherence within and between the resources in a text is essential.
Coherence between resources (ie between the elements or modes and semiotic systems in the text) can be
achieved through:
the weighting or dominance of each one of them
the relationships among them (complementary, reinforcing, hierarchical).
their function (ideational, interpersonal or textual)
how they operate individually and together in time and space
the sensual and aesthetic effects they invoke.
FIGURE 2.4 How the Four Resource Model might be employed by a reader during the readingof a multimodal text
Context: The reader, a year 3 student, is accessing a website that provides information about frogs. His
reading purpose is to identify the life cycle of a frog for a report to be collated and presented as an oral
presentation, aided by PowerPoint. The website has linguistic information presented in conventional
paragraphs with headings, dot points and diagrams. There are also photographs, and a link to a video
about frogs in Australia.
Chapter 2 45
74a,....,
.E
1E
rcl
Ts
i =
o
lc
en
a,
a,c4
c
=ö
0-
g
Skills, processes and knowledge employed Main focus of skills, processesand knowledge being employed
1 Prior to accessing the text, reader considers purpose for.
reading and decides what criteria will be used to judge
relevant information on the site.
1. Text analyst
2. Reflects upon what he already knows about the life cycle
of frogs.
2. Meaning maker
3. Decides to look for key words, such as 'life cycle',
'breeding', 'tadpoles', 'eggs', 'adult'.
3. Text analyst
4. Reflects upon what he already knows about websites,
how to use them, their format and what might be available
to him and decides to look for both visual and print
information as they would be most useful and appropriate
for a PowerPoint presentation.
4. Text user and text analyst
5. Accesses website and decides to scan text to identify the
elements present that he will need to access (linguistic
information, paragraphs, headings, dot points and
diagrams, photographs, links to video).
5. Code breaking and meaning
making
6. While scanning, looks for keywords and images and
diagrams that illustrate them.
6. Meaning maker
7. Having identified keywords and diagrams, considers the
most relevant by evaluating against previously determined
criteria. Identifies paragraph headed 'Life cycle' and a
diagram with similar title and video on Australian frogs as
best sources.
7. Text analyst
8. Uses knowledge about diagrams and note-taking from
other experiences to read and note-take from diagram
and paragraph.
8. Meaning making and text user
9. Starts watching video and has trouble finding any
information on life cycles; it simply describes different
types of frogs. Decides this is not relevant to reading
purpose and stops video.
9 Text user and text analyst
10. Reviews notes and original purpose for reading. Scans text
once more for any other information that meets criteria.
Decides he has all that is relevant and closes down site.
10. Text user and text analyst
It is important that students learn to employ the four resources strategically andunderstand the need to monitor their effectiveness when employing them. If studentslearn to consciously employ the Four Resource Model, they will be more aware of theirresponsibilities as a reader and become more strategic, self-conscious and active readers;all characteristics necessary to becoming an effective reader and writer of multimodaltexts.
ConclusionThis chapter has focused specifically on exploring the characteristics of multimodaltexts and how these characteristics influence the ways in which we consume, use andproduce text. In Section One, we defined multimodal texts, the elements of which theyare comprised and how these elements cohere to produce meaning. Sections Two andThree examined how the reading and writing processes take place with multimodaltexts. In Section Four, we considered how the Four Resource Model can be used toassist students in becoming more strategic and active consumers and producers of text,characteristics necessary to using multimodal texts effectively.
We have very deliberatively constructed Reflection Strategies and Theory intoPractice sections that encourage you and your students to carefully examine how youread, and the decision making involved. Similarly, we have engaged you in tasks thathave promoted your understanding about the very complex and structured design ofmultimodal texts. The purpose of such detailed investigation was to reactivate yourconsciousness about hovv you read and write, because as competent adult readers muchof what we do is automatic. We are not aware of the thinking processes and decisionmaking that we now do automatically, therefore it is difficult for us to really understandthe complexity of the tasks our students engage in or to adequately model them.Modelling and conversations around the processes of reading and writing have alwaysbeen important aspects of literacy pedagogy, but are probably even more important astexts and literacy become more complex and subject to rapid change.
We hope this chapter has provided a firm foundation of knowledge about multimodaltexts and a beginning understanding of the knowledge, processes and skills necessary toengage in consuming and producing them. We hope the activities have also demonstratedthe way in which pedagogy, particularly talk about and around text, is an importantaspect of teaching and learning about multimodal texts.
In Chapter 3, we will commence more detailed practical classroom investigationsof multimodal text with postmodern picture books. We have chosen to do this for anumber of reasons. Firstly, almost all schools have access to a collection of picturebooks, therefore it facilitates immediate engagement with multimodal texts for you andyour class. Secondly, there is a danger that in addressing multimodal texts we considerthat all multimodal texts involve technology. While many do, many do not, and manyof the multimodal texts that we engage with in everyday life are conveyed by print andpaper, for example, books, newspapers, maga2dnes, information pamphlets, instructions.Therefore, we thought it useful to start with what we see as familiar (picture books) butexamined from a new perspective, as a multimodal text that enhances our understandingof multiliteracies. Finally, investigation of the modes of picture books (language andimages) and the semiotic systems employed (visual and linguistic) together withtheir design and cohesion, provides a knowledge base that can be used with all othermultimodal texts and technologies and is therefore a very good place to start.
46 Evolving Pedagogies