Department of English, German and Slavic Studies

Department of Psychology and Cultural Anthropology

International Symposium on Multimodal Approaches

to Communication

Wed, 27 May 2009 (8.45 am – 6.45 pm) Room T2, Polo Didattico ‘G. Zanotto’, University of Verona

PROGRAMME AND

ABSTRACTS

The Symposium will gather researchers in the field to discuss their works

with reference to how multimodal analysis sheds new light onto linguistic studies and studies on communication. The Symposium

presents itself as an opportunity for opening a discussion onto new possibilities and directions for interpreting human communication.

Scientific committee: Elisabetta Adami, Roberta Facchinetti, Cesare Gagliardi, Carey Jewitt, Gunther Kress

Website: http://profs.lingue.univr.it/multimodality/

University of Verona, 27 May 2009

2

Symposium on Multimodal Approaches to Communication

PROGRAMME

8.45-9.00 Welcome address Keynote speech 9.00-10.00 CAREY JEWITT: Different approaches to multimodality Session: Dynamic images 10.00-10.30 DEIRDRE KANTZ: A multimodal diachronic study of public information

films: application in the classroom 10.30-11.00 ILARIA MOSCHINI: Music & Series: The Verbalizing Role of Soundtracks in

Contemporary TV Series 11.00-11.30 coffee break Keynote speech 11.30-12.30 GUNTHER KRESS: Modes as technologies of transcription: Some

comments on contemporary forms of composition Session: Web 12.30-13.00 ELISABETTA ADAMI: Video-interaction on YouTube: Contemporary

changes in semiosis and communication 13.00-13.30 IVANA MARENZI: Web 2 interaction and educational genres: social,

interactional, technological and multimodal aspects 13.30-14.30 lunch Session: Face-to-face interaction 14.30-15.15 ANTHONY BALDRY & PAUL THIBAULT: Multimodal turn-taking 15.15-15.45 MARTIN SOLLY: ‘Giving the graduates an earful’: identity and

interaction in commencement speeches 15.45-16.15 FRANCESCA COCCETTA: Bringing a multimodal perspective to the

investigation of spoken corpora 16.15-16.45 coffee break Session: Specific discourses 16.45-17.15 GIULIANA GARZONE: Exploiting the potential of the web-based

environment for political purposes: The case of U.S. public diplomacy

17.15-17.45 SANDRA CAMPAGNA & CECILIA BOGGIO: Multimodality in Economic discourse

17.45-18.15 ELENA MONTAGNA: Politicians and their dress code: the case of Margaret Thatcher

18.15-18.45 Poster presentations ILARIA FORNASINI: Illuminated multimodality: Communicative dynamics

in later medieval manuscripts CATERINA GUARDAMAGNA: Medical Communication: A corpus of medical

simulation/animation ROSALBA RIZZO: Multimodal and intercultural analysis of today’s

online newspapers MARIA GRAZIA SINDONI: Mode switching: how oral and written modes

alternate in videochats

3

University of Verona, 27 May 2009

4

Symposium on Multimodal Approaches to Communication

ABSTRACTS KEYNOTE SPEECHES:

- CAREY JEWITT (Centre for Multimodal Research, London Knowledge Lab): ‘Different approaches to multimodality’.........................................................................p.6

- GUNTHER KRESS (Centre for Multimodal Research, Institute of Education, University of London): ‘Modes as technologies of transcription: Some comments on contemporary forms of composition’ ..............................................................................p.7

5

University of Verona, 27 May 2009

DIFFERENT APPROACHES TO MULTIMODALITY

Carey Jewitt Comments on the multimodal character of communication, texts and media are increasingly commonplace across a range of disciplines (e.g. anthropology, education, design, linguistics, media and culture studies, musicology, sociology). But what does it actually mean to call a text, practice, or indeed anything else, multimodal or to approach communication from a multimodal perspective? I briefly introduce the term multimodality and how it is used, the main assumptions that underlie this approach, and the core concepts that inform multimodal analysis. There are a number of theoretical and methodological connections across different perspectives on multimodality and the interests that bind them: the fine grained analysis, the need to understand the details of texts and interactions, the focus on meaning making as social and semiotic, the interest in the place of language within a multimodal ensemble, and the demand to broaden conceptions of communication beyond language. The shared interests, and to some extent the shared histories of different approaches to multimodality make multimodal approaches distinct from other takes on communication and meaning making. For instance, multimodality is distinct from cognitive psychological approaches that focus more explicitly on the internal, notions of mind, and cognitive processes. Nonetheless, there are differences between particular approaches to multimodality and I explore the particular accent to multimodality that different approaches give. I suggest that the differences between these approaches stem from the historical influences and directions that have shaped them as well as the degree of emphasis each gives to context, the internal relations within modes or modal systems (e.g. level and rank), and the agentive work of the sign maker. I show that how different approaches to multimodality configure these factors is both constituted by and constitutes the kind of research that is typically undertaken – in other words, how each has evolved to attend to particular aspects of multimodality and, to some extent to specific sites of research – in order to tease out what this means for the kinds of data and questions each perspective might address. I argue that multimodality can be understood as a theory, a perspective or a field of enquiry, or methodological application. I conclude with a brief introduction to some of the disciplines and theoretical approaches that are in conversation with multimodality.

6

Symposium on Multimodal Approaches to Communication

MODES AS TECHNOLOGIES OF TRANSCRIPTION: SOME COMMENTS ON CONTEMPORARY FORMS OF COMPOSITION

Gunther Kress

The contemporary landscape of communication is not what it was even thirty years ago. That much seems beyond debate really; and it is not merely a replay of usual generational or temporal/historical differences. If we accept that theories are shaped in the historical / social environments of the periods of their making, we must ask what kinds of theories around communication and representation are adequate to the descriptive and analytic tasks demanded of them by present conditions of representation and communication. In my talk I want to focus on two issues: the issue of 'transcription' and of 'technologies of description' in an era marked by the use of multiple forms of representation; and the issue of contemporary forms and principles of composition in their interrelation with social, semiotic and technological factors.

7

University of Verona, 27 May 2009

8

Symposium on Multimodal Approaches to Communication

ABSTRACTS PAPERS:

- ELISABETTA ADAMI (University of Verona): ‘Video-interaction on YouTube: Contemporary changes in semiosis and communication’ ................................................................................................................p. 10

- ANTHONY BALDRY (University of Messina) and PAUL THIBAULT (University of Agder, Norway): ‘Multimodal turn-taking’ .....................................................................................................p. 12

- SANDRA CAMPAGNA (University of Torino) and CECILIA BOGGIO (University of Torino): ‘Multimodality in Economic discourse’ ...........................................................................p. 14

- FRANCESCA COCCETTA (University of Padova): ‘Bringing a multimodal perspective to the investigation of spoken corpora’ ................................................................................................................................p. 15

- GIULIANA GARZONE (University of Milano): ‘Exploiting the potential of the web-based environment for political purposes: The case of U.S. public diplomacy’ ................................................................p. 17

- DEIRDRE KANTZ (University of Pavia): ‘A multimodal diachronic study of public information films: Application in the classroom’......................................................................................................................p. 19

- IVANA MARENZI (University of Pavia): ‘Web 2 interaction and educational genres: Social, interactional, technological and multimodal aspects’ .........................................................................p. 20

- ELENA MONTAGNA (University of Pavia): ‘Politicians and their dress code: The case of Margaret Thatcher’.............................p. 21

- ILARIA MOSCHINI (University of Firenze): ‘Music & Series: The verbalizing role of soundtracks in contemporary TV series’ .....................................................................................................................................p. 22

- MARTIN SOLLY (University of Firenze): ‘‘Giving the graduates an earful’: identity and interaction in commencement speeches’............................................................................................p. 23

9

University of Verona, 27 May 2009

VIDEO-INTERACTION ON YOUTUBE: CONTEMPORARY CHANGES IN SEMIOSIS AND COMMUNICATION

Elisabetta Adami

In spite of their different approaches and further reinterpretations (cf., for example, Davies 2007; Giora, 1997), both coding-decoding (Shannon and Weaver’s 1949) and inferential models of communication – such as Grice’s (1967) Maxims and Sperber and Wilson’s (1986) Relevance Principle – share the assumption that successful communication is based on the receiver’s understanding of the (speaker’s intended meaning of the) text. Analogously, semantic notions of cohesion and coherence (Halliday and Hasan 1976; Halliday, 1985; van Dijk, 1979; 1980; 1985; Fairclough, 1992) assign to both internal features of the text and to the interpretative work of the receiver the task of retrieving ‘what the text is (intended to be) about’. In other words, traditional theories of communication hinge on the rather commonsensical idea that successful communication is a matter of interlocutors understanding each other. However, when trying to describe new forms of interaction, such as the one occurring by means of videos replying one to another on the Website YouTube, these communicative models prove to be quite inadequate. Indeed, rather than working on the interlocutors’ mutual understanding, video-interaction is driven by the participants’ diversified interests in taking up, misinterpreting and transforming each other’s texts. Texts in the exchanges are used as prompts to which to respond, often incoherently (Adami 2009), through the interested transformation, assemblage and recontextualization of selected signifiers of the other texts. These ‘incoherent’ chains of semiosis create generally successful exchanges, totally accepted and acknowledged by video-interactants, while the interlocutors’ intended meaning remains quite irrelevant. In sum, like many other forms of contemporary communication (e.g., those produced through ‘copy-and-paste’ techniques), video-interaction frequently disregards (traditionally conceived) patterns of coherence, in favour of an interested recontextualization of other texts, often irrespectively of their authors’ intended meaning. By examining an exemplary video-thread on YouTube, the paper discusses these apparently incoherent, yet successful, instances of video-interaction. The analysis gives evidence to the non-conformity of the practices of video-interaction to traditional communicative models. Consequently, by endorsing and developing a social semiotic perspective (Hodge and Kress 1988; Kress 2008; van Leeuwen 2005), the paper hypothesizes some heuristics for an adequate description of the chains of semiosis produced in contemporary forms of communication. References: Adami, E. 2009. ‘We/YouTube’: Exploring sign-making in video-interaction. Visual

Communication, 8/4. Davies, B. L. 2007. Grice’s Cooperative Principle: Meaning and rationality. Journal of

Pragmatics, 39.2308-31. Fairclough, N. 1992. Discourse and Social Change. Cambridge: Polity Press. Giora, R. 1997. Discourse coherence and theory of relevance: Stumbling blocks in

search of a unified theory. Journal of Pragmatics, 27.17-34. Grice, H. P. 1967. Logic and conversation. William James Lectures, unpublished. Halliday, M. A. K. 1985. Dimensions of discourse analysis: Grammar. Handbook of

Discourse Analysis, ed. by Teun A. van Dijk. London: Academic Press. Vol.2. 29-56

Halliday, M. A. K. and Hasan, R. 1976. Cohesion in English. Harlow: Longman. Hodge, R. and Kress, G. 1988. Social Semiotics. Cambridge: Polity.

10

Symposium on Multimodal Approaches to Communication

Kress, G. . 2008. New Literacies, New Democracies. A Beyond Current Horizons challenge paper. http://www.beyondcurrenthorizons.org.uk/wp-content/uploads/bch_challenge_paper_democracies_gunther_kress.pdf

Shannon, C. and Weaver, W. 1949. The Mathematical Theory of Communication. Urbana: University of Illinois Press.

Sperber, D. and Wilson, D. 1986. Relevance. Communication and Cognition. Oxford: Blackwell.

van Dijk, T. A. 1979. Relevance assignment in discourse comprehension. Discourse Processes, 2.113-26.

van Dijk, T. A. 1980. The semantics and pragmatics of functional coherence in discourse. Speech act theory: Ten years later, ed. by A. Ferrara, 49-65. Milano: Special issue of Versus 26/27.

van Dijk, T. A. . 1985. Semantic discourse analysis. Handbook of Discourse Analysis, ed. by Teun A. van Dijk, 103-37. London: Academic Press.

van Leeuwen, T. 2005. Introducing Social Semiotics. London: Routledge.

11

University of Verona, 27 May 2009

MULTIMODAL TURN-TAKING

Anthony Baldry and Paul Thibault

This talk looks at multimodal turn taking and reports on joint work in relation to the analysis of the BBC’s Hardtalk interviews (http://news.bbc.co.uk/1/programmes/ hardtalk/). It is part of research into multimodal corpus linguistics (Baldry and Thibault, 2001, 2006b, 2008) which suggests that this type of approach can add new and valuable dimensions to work in corpus linguistics in general. Anthony Baldry will look at the macro aspects of multimodal turn-taking (Baldry 2000: 14-6) reviewing pair and group turn-taking (Baldry, 2005b) in relation to both distance and face-to-face interactions (Baldry, 2005a) suggesting the added value of multimodal approaches to turn-taking vis-à-vis classic descriptions (e.g. Coulthard, 1985; Goffman, 1981; Sacks et al 1974). Paul Thibault will carry out a fine-grained micro-analysis of turn-taking within a scalar system of multimodal text analysis (Baldry and Thibault, 2006a). In this framework, he will consider the exploratory and coordinative nature of “turns”. These will be discussed in relation to the norms and conventions of the (Hardtalk) television interview. Against the widespread view that meaning-making is reducible to predictable arrangements or sequences of determinate textual or discourse-level units such that interaction reduces to “text-in-context”, Thibault will focus on real-time bodily activity and dynamics as the basis around which emergent understandings and actions are coordinated (Thibault 2004: 194-201, 2008). In engaging with multimodal analysis, he will illustrate some of the ways in which these perspectives can be analysed and theorized (Thibault 2003, 2005). In doing so, the audience will be invited to think anew about the nature of “language” itself and how we can feed this thinking back into our conception of multimodal corpus linguistics. References: Baldry A. P. (2000). “Introduction”. In A. P. Baldry (ed.) Multimodality and

multimediality in the distance learning age, Campobasso: Palladino Editore, pp. 11-39.

Baldry A. P. (2005a). A Multimodal Approach to Text Studies in English, Palladino: Campobasso.

Baldry A. P. (2005b). Children’s use and awareness of genre: A case study of the evolution from multimodal transcription to multimodal concordancing based on system networks. In: I Centri Linguistici: approcci, progetti e strumenti per l’apprendimento e la valutazione. 3° Convegno Nazionale AICLU / 6° Seminario AICLU. Trieste/Letojanni (Messina). 13-14 giugno 2003/30-31 ottobre 2003. Trieste: EUT. Vol. II, pp. 423-441.

Baldry, A.P; Thibault, P. J. (2001). ‘Towards Multimodal Corpora’. In G. Aston and L. Burnard (eds.), Corpora in the Description and Teaching of English. Bologna: CLUEB, pp. 87-102.

Baldry A. P.; Thibault, P. J. (2006a). Multimodal Transcription and Text Analysis, London and New York: Equinox.

Baldry A. P.; Thibault, P. J. 2006b). ‘Multimodal Corpus Linguistics’. In G. Thompson and S. Hunston (eds), System and Corpus Exploring Connections, London and New York: Equinox, pp. 164-183.

Baldry, A.P.; Thibault, P. J. (2008). ‘Applications of multimodal concordances’. In Hermes: Journal of Language and Communication Studies: 41: 11-41

Coulthard, M. (1985). An introduction to discourse analysis. London and New York: Longman.

Goffman, E. (1981). Forms of Talk. Oxford: Blackwell.

12

Symposium on Multimodal Approaches to Communication

Sacks, H., Schegloff, E. A. & Jefferson, G. (1974) ‘A Simplest Systematics for the Organisation of Turn-Taking for Conversation’, in Language, 50:696-735.

Thibault, P. J. (2003). Body dynamics, social meaning-making, and scale heterogeneity: re-considering contextualization cues and language as mixed-mode semiosis', in Discussing John J. Gumperz, Susan Eerdmans, Carlo Prevignano, and Paul J. Thibault (eds.), `127-147. Amsterdam/Philadelphia: Benjamins.

Thibault, P. J. (2004). Agency and Consciousness in Discourse: Self-other Dynamics as a Complex System. London and New York: Continuum.

Thibault, P. J. (2005). ‘What kind of minded being has language: Anticipatory dynamics, arguability, and agency in a normatively and recursively self-transforming learning system, Part 2’. Linguistics and the Human Sciences 1, 3, 355-401.

Thibault, P. J. (2008). ‘Face-to-face communication and body language’. In Handbooks of Applied Linguistics (HAL) Linguistics for Problem-Solving: Perspectives on Communication Competence, Language and Communication Problems, and Practical Solutions, Karlfried Knapp and Gerd Antos (eds.), pp. 285-330. Volume 2: Interpersonal Communication. Gerd Antos & Eija Ventola (eds.). Berlin. Mouton.

13

University of Verona, 27 May 2009

MULTIMODALITY IN ECONOMIC DISCOURSE

Sandra Campagna and Cecilia Boggio This paper is the very first presentation of Multimodal Business and Economics (Campagna, S., /Boggio, C., 2009. Milano: LED). Drawing on seminal work in multimodal literature (Kress and van Leeuwen 1996, 2001; Jewitt/Oyama 2001; Iedema 2003; Lemke 2002; Scollon/Scollon 2003, 2004) the volume is the outcome of research on multimodality as a practical toolkit of resources used for interpreting semiotic configurations in Business and Economics. Moving from the assumption that every text is multimodal, the volume intends to guide university students towards the analysis of texts and hypertexts representative of the business/economic domain. These mapping guidelines are given through the construction of reading paths applied to a variety of different text types ranging from informative/descriptive to persuasive/argumentative. The cohesive ‘glue’ which links the selected texts and characterizes the corresponding materials is the prevailing focus on the visual dimension of the representation of business and economic discourses. Thus, each of the seven units in the book prioritizes the denotative/connotative meaning of iconic/symbolic cues, such as visual shapes in print/web adverts and data graphics in quantitative reasoning.

14

Symposium on Multimodal Approaches to Communication

BRINGING A MULTIMODAL PERSPECTIVE TO THE INVESTIGATION OF SPOKEN CORPORA

Francesca Coccetta

In an age when corpus linguistics has undoubtedly established itself in a variety of fields such as lexicography (e.g. Cobuild, 2006), grammatical studies (e.g. Biber et al. 1999) translation studies (e.g. Laviosa, 1998), and, of course, studies relating to language teaching and learning (Johns, 1991; Partington, 1998; Hunston, 2002), we should reflect on those areas where corpus linguistics lags behind. The research community should reflect on criticisms relating to the form corpora take in linguistic investigations (Leech, 2000; Mishan, 2004), which deprives texts of their multimodal nature (Kress/van Leeuwen, 2006). Some have started thinking about these issues (e.g. Adolphs, 2008) while others (Baldry/Thibault 2001; 2006a, 2006b; 2008) have provided both the theoretical and technological means to create, annotate and concordance multimodal corpora. With particular reference to spoken texts, this talk illustrates how innovations which have taken place in the new field of multimodal corpus linguistics, especially within the MCA project (Baldry, 2008; Baldry/Thibault, 2008), have been applied to some of the film texts in the Padova Multimedia English Corpus (Ackerley/Coccetta, 2007) in such a way as to promote communicative language competence by language learners at various levels of proficiency. In particular, the talk will show how the multimodal concordancer MCA (Baldry, 2005; Baldry/Beltrami, 2005) can be used to annotate texts for functions and notions (van Ek/Trim, 1998; 2001) and will illustrate the benefits of the kind of information provided to language learners by the concordance lines produced by MCA and their associated video clips-cum-multimodal cotext (Baldry, 2008). References: Ackerley, Katherine/Coccetta, Francesca 2007. “Enriching Language Learning

through a Multimedia Corpus.” In ReCALL 19/3: 351-370. Adolphs, Svenja 2008. Corpus and Context. Investigating Pragmatic Functions in

Spoken Discourse. Amsterdam: John Benjamins. Baldry, Anthony 2005. A Multimodal Approach to Text Studies in English: The Role of

MCA in Multimodal Concordancing and Multimodal Corpus Linguistics. Campobasso: Palladino.

Baldry, Anthony 2008. “Turning to Multimodal Corpus Research for Answers to a Language-course Management Crisis.” In Taylor Torsello, Carol/Ackerley, Katherine/Castello, Erik (eds.), Corpora for University Language Teachers. Bern: PeterLang: 226-237.

Baldry, Anthony /Beltrami, Michele 2005. The MCA Project: Concepts and Tools in Multimodal Corpus Linguistics. In Asplund Carlsson, Maj et al. (eds.), Multimodality: Text, Culture and Use. Kristiansand: Adger University College and Norwegian Academy Press, 79–108.

Baldry, Anthony/Thibault, Paul J. 2001. “Towards Multimodal Corpora.” In Aston, Guy/Burnard, Lou (eds.), Corpora in the Description and Teaching of English. Bologna: CLUEB, 87-102.

Baldry, Anthony/Thibault, Paul J. 2006a. Multimodal Transcription and Text Analysis. A Multimedia Toolkit and Coursebook. London and New York: Equinox.

Baldry, Anthony/Thibault, Paul J. 2006b. “Multimodal Corpus Linguistics.” In Thompson, Geoff/Hunston Susan (eds.), System and Corpus: Exploring Connections. London and Oakville: Equinox, 164-183.

Baldry, Anthony/Thibault, Paul J. 2008. “Applications of Multimodal Concordances.” In Hermes 41, 11-41.

15

University of Verona, 27 May 2009

Biber, Douglas et al. 1999. Longman Grammar of Spoken and Written English. London: Longman.

Cobuild 2006. Collins Cobuild English Dictionary for Advanced Learners. Glasgow: Harper Collins.

Hunston, Susan 2002. Corpora in Applied Linguistics. Cambridge: Cambridge University Press.

Johns, Tim 1991. “Should You Be Persuaded: Two Examples of Data-driven Learning Materials.” Classroom concordancing, ELR Journal 4: 1–16.

Kress, Gunther/van Leeuwen, Theo 2006. Reading Images. The Grammar of Visual Design. London and New York: Routledge.

Laviosa, Sara 1998. “Core Patterns of Lexical Use in a Comparable Corpus of English Narrative Prose.” In Meta 43/4: 557-570.

Leech, Geoffrey 2000. “Grammars of Spoken English: New Outcomes of Corpus-Oriented Research.” Language Learning 50/4: 675-724.

Mishan, Freda 2004. “Authenticating Corpora for Language Learning: A Problem and its Resolution.” ELT Journal 58/3, 219-27.

Partington, Alan 1998. Patterns and Meaning: Using Corpora for English Language Research and Teaching. Amsterdam: John Benjamins.

van Ek, Jan Ate/Trim, John Leslie Melville 1998. Threshold 1990. Cambridge: Cambridge University Press.

van Ek, Jan Ate/Trim, John Leslie Melville 2001. Vantage. Cambridge: Cambridge University Press.

16

Symposium on Multimodal Approaches to Communication

EXPLOITING THE POTENTIAL OF THE WEB-BASED ENVIRONMENT FOR POLITICAL PURPOSES: THE CASE OF U.S. PUBLIC DIPLOMACY

Giuliana Garzone

In recent times, Web-mediated communication has qualified as a powerful strategic resource in foreign policy, being central to any effort aimed at reaching foreign audiences pervasively, also in areas of the world which – for various reasons – are otherwise difficult to access for Western actors. In the last few years it has played a particularly important role in US public diplomacy, a crucial area of US foreign policy consisting in “government-sponsored programs intended to inform or influence public opinion in other countries" (U.S. Department of State, Dictionary of International Relations Terms, 1987, p. 85) and addressing primarily non-governmental individuals and organizations abroad rather than governments. Especially in the aftermath of 9/11, public diplomacy has been part of an increasing effort to promote the image of the United States with foreign audiences, also in those areas of the world where the war on terrorism and the US engagement in the Middle East have resulted in a surging wave of anti-Americanism. This paper will look at the use of web-mediated multimodal resources in contemporary US public diplomacy, on the basis of the analysis of a website, America.gov, produced and maintained by the Department of State, which represents the main effort so far by the US government to use web-based resources in this area of activity. After discussing the scope and relevance of public diplomacy as an instrument in foreign policy, the analysis will look at the discursive practices deployed on the America.gov website, analysing not only the linguistic strategies enacted, but also their role within the semiotic organization of the website, evaluating the effectiveness of communication effected through the WWW in terms of successful exploitation of multimedia and multimodal affordances. The main questions the analysis will seek to answer concern the strategies enacted to exploit web-mediated communication and its affordances for political purposes, and the functional effectiveness of the layout and the semiotic organization of the Website. Last but not least, the role of language in the wider picture of multimodal web-mediated communication will be explored, looking in particular at the effectiveness of discursive practices in an intercultural perspective. An element that will be given attention in connection with all these different aspects is the process of change that the website has undergone in recent times under the direction of President Obama’s new Administration. The basic methodological framework of this study is in the area of critical discourse analysis, which provides instruments to identify and highlight the discursive – and often ideological – relevance of linguistic choices at all levels, including the microstructural one. The exploration of the use of hypermedia web-mediated resources will mainly rely on principles put forth in recent studies applying discourse analysis to multimodality, usually grouped under the denomination “multimodal analysis” (Kress – van Leeuwen 2001, 1996/2006; Iedema 2003; O’Halloran 2004; LeVine – Scollon 2004; Garzone 2007). Some useful tools will also be drawn from the sociological and political literature on public diplomacy (Leonard 2004; Blankey 2008; Gilboa 2008), and on place/nation branding (Anholt / Hildreth 2004; van Ham 2008).

17

University of Verona, 27 May 2009

References: Anholt, Simon / Hildreth, Jeremy 2004. Brand America. The Mother of All Brands.

London: Cyan Books. Blankley, Tony / Dale, Helle C. / Horn, Oliver 2008. Reforming U.S. Public Diplomacy

for the 21st Century. Heritage Foundation. Garzone, Giuliana 2007. Genres, multimodality and the world wide web: Theoretical

Issues. In Garzone, Giuliana / Poncini, Gina / Catenaccio, Paola (eds) Multimodality in Corporate Communication. Web Genres and Discursive Identity. Milan: Franco Angeli.

Gilboa, Eytan 2008. Searching for a Theory of Public Diplomacy. The ANNALS of the American Academy of Political and Social Science 616, March 2008: 55-77.

Iedema, R. 2003. Multimodality, Resemiotization: Extending the Analysis of Discourse as Multi-semiotic Practice. Visual Communication (2)1, pp. 25-57.

Kress, G. & van Leeuwen, T. (2001), Multimodal Discourse, London: Arnold. Kress, G. & van Leeuwen, T. 1996/2006. Reading Images. The Grammar of Visual

Design. 2nd ed. London & New York. Routledge. Leonard, Mark 2004. Public Diplomacy. London: The Foreign Policy Centre (with C.

Stead and C. Smewing). LeVine P., Scollon R. (eds) 2004. Discourse and Technology. Multimodal Discourse

Analysis. Washington D.C.: Georgetown University Press. O’Halloran K. (ed.) 2004. Multimodal Discourse Analysis. Systemic-Functional

Perspectives. London: Continuum. Van Ham, Peter (ed.) 2008. Place Branding: The State of the Art. The ANNALS of the

American Academy of Political and Social S

18

Symposium on Multimodal Approaches to Communication

A MULTIMODAL DIACHRONIC STUDY OF PUBLIC INFORMATION FILMS: APPLICATIONS IN THE CLASSROOM

Deirdre Kantz

This paper analyses students’ reactions to the Public information films [hereafter: Pif or Pifs] presented in the UK National Archives website (http://www.nationalarchives. gov.uk/films/) and to a course that explored the construction of social identities in these films from a multimodal perspective (Baldry/Kantz 2009). As stated in the site’s introduction:

For the first time on The National Archives’ website you can view complete public information films from 1945-2006. Joining with the Central Office of Information (COI) to celebrate their 60th Anniversary, we have featured a selection of some of the most memorable and influential COI public information films that cover some fascinating events from Britain’s post-war history. Many of the films have both historic and nostalgic value. Historically, they reflect the issues of the day; nostalgically, everyone has a favourite.

Teachers and students of postwar English film texts will welcome the availability of a fascinating site covering 60 years of state TV and cinema propaganda, a de facto longitudinal corpus that is an endless source of entertainment as well as a genuine window on weirdness. The site’s Pifs may be examined from many perspectives ranging from changes in social attitudes and voice prosodics to the textual composition of the various Pif subgenres. The site’s special diachronic filter is particularly appealing as it helps shift the current focus in multimodal research from how multimodal texts are constructed to why they are so constructed and why readers/viewers react to them in the ways that they do. In its belief that the young discipline of multimodality constantly needs to reflect on its roles in linguistics and Higher Education, this paper will explore the evolution of identity construction in this site in an attempt to understand how multimodal text analysis might function most advantageously in undergraduate courses. To this end this paper reports on the reactions of 20 dentistry students in a 32-hour course of multimodal English as they came to grips with the medical, social, political, and mediational aspects of 60 years of government propaganda. As a genre, Pifs inevitably deal with unpleasant and shocking thematics but usually provide a sugar coating, often visual or musical rather than linguistic, vis-à-vis threats to a citizen’s well-being. In so doing, they construct social identities of individuals in rather grotesque, stereotypical ways that guarantee a strong response in terms of critical classroom discussions and detailed analytical comparisons in students’ exam work (Baldry, 2008).

References: Baldry Anthony, 2008, “What is Multimodality for? Syllabus Construction in English Text

Studies for Communication Sciences”, in Martin Solly, Michelangelo Conoscenti and Sandra Campagna (eds.), Verbal/Visual Narrative Texts in Higher Education, Peter Lang Bern, pp. 229-48

Baldry Anthony, Kantz, Deirdre (in press) “New Dawns and New Identities for Multimodality. Public Information Films in The National Archives”, in Nicoletta Vasta con C.R. Caldas-Coulthard (eds.) Identity Construction and Positioning in Discourse and Society. Textus, vol. XXII(1). [Numero monografico 2009/1 (Jan.-Apr.) della rivista dell'Associazione Italiana di Anglistica Textus - English Studies in Italy. GENOVA: Tilgher. ISBN: 1824-3967].

19

University of Verona, 27 May 2009

WEB 2.0 INTERACTION AND EDUCATIONAL GENRES: SOCIAL, INTERACTIONAL, TECHNOLOGICAL AND MULTIMODAL ASPECTS

Ivana Marenzi

Precisely because Web 2.0 is still evolving, a consensus on its exact meaning has not yet been reached and is unlikely to be reached in the near future. Those who have proposed or promoted the Web 2.0 concept say that it differs from early web developments, retroactively labelled Web 1.0, in that it constitutes a move away from static websites, the use of search engines and web surfing, to a more dynamic and interactive system of communication that relies on the notion of individual users participating in communities. Others argue that the original and fundamental concepts of the web are not actually being superseded, while sceptics hold that the term is little more than a buzzword, that it means whatever its proponents want it to mean in order to convince their customers, investors and the media that they are creating something fundamentally new, rather than continuing to develop and use well-established technologies. In the author’s experience, Web 2.0 may be looked upon from diverse standpoints: • socially, it refers to an approach in which the market is seen as a ‘conversation’,

a place for open communication in which the freedom to share and above all reuse and recontextualise texts arises from the apparent decentralisation of authority;

• economically, it represents a shift in value of the web, a marketing term that differentiates new web businesses from those of the rather discredited and somewhat superseded dot.com boom of the late 1990s;

• technologically, it has more organised and categorised content, made possible by deep-linking web architectures; it marks a transition from websites construed as sets of isolated texts to platforms that provide web applications such as Google, YouTube, Netvibes, Doodle, Ning, Flickr and so on; all this promotes websites as sources of content, knowledge, information and social functionality;

• semiotically, it represents a series of innovations in the genres used in the Internet which instantiate rather special combinations of action and meaning potential (Baldry, Thibault, 2006: Chap 3).

My paper will attempt to sketch out a framework in which Web 2.0 will be presented generally from a synthesising but multiple viewpoint. While it is commonplace to refer to social networking software in terms of tools, nevertheless the ultimate focus in this context will be on their implementation of new genres involving new expectations and new perspectives in interaction and learning whose characteristics require a multimodal approach (Marenzi, in press). References: Baldry, Anthony, Thibault, Paul J., (2006), Multimodal Transcription and Text Analysis: A

Multimodal Toolkit and Coursebook with Associated On-line Course. London and New York: Equinox.

Marenzi, Ivana (in press) “Designer Genres: Social, interactional, technological and multimodal aspects of Web 2 genres”. In Anthony Baldry and Elena Montagna, (eds.), Interdisciplinary Perspectives on Multimodality: Theory and practice. Campobasso: Palladino.

20

Symposium on Multimodal Approaches to Communication

POLITICIANS AND THEIR DRESS CODE: THE CASE OF MARGARET THATCHER

Elena Montagna

This paper uses a multimodal framework to explore changes in public identities over time. Specifically, the paper reconstructs the public image and dress code of Lady Thatcher, one of the very few women in British society who on the basis of the image and identity constructed for her by the press was able to show masculinity at work and femininity at home. The study relates to the textual forms that newspapers took in the years when Mrs. Thatcher was in power, namely 1979-90, an approach that differs from those that would be used to look at today’s politicians in age when online newspapers with their blogs and podcasts play a major role in the identity and image construction and/or destruction of leading politicians. The paper identifies 10 key moments in Thatcher’s career and suggests a progression through 3, and perhaps 4, clearly marked stages of identity transformation. The research is based on British and Italian press reports and is supported by interviews with leading Italian journalists who were working at the time in the United Kingdom and who knew Thatcher personally. The approach to multimodality is essentially diachronic: it uses multimodality as one of the tools to identify changing states in the relationship between Thatcher and her public and to explain how and why these changes came about. It thus contributes to a tradition in research, undertaken prevalently in Italy, regarding the multimodal construction of public identities (Baldry, 2000; Baldry and Montagna in press; Baldry and Kantz in press; Montagna, in press, Vasta and Caldas-Coulthard in press) in the field of politics (Vasta 2001) and economics (Lombardo, Vasta, 2005, forthcoming) in particular. References: Baldry A. P. (2000). English in a visual society: comparative and historical dimensions in

multimodality and multimediality’. In: Baldry A.P (ed.) Multimodality and multimediality in the distance learning age. Campobasso: Palladino, pp. 41-89.

Baldry A.P., Kantz, D. (in press). New dawns and new identities for multimodality: Public information films in The National Archives. In Vasta. N., Caldas-Coulthard, C. R. Identity Construction and Positioning in Discourse and Society. Textus, 2009 (1).

Baldry, A.P., Montagna, E. (in press). Interdisciplinary perspectives on multimodality: theory and practice. Campobasso: Palladino.

Fairclough N, (1992), Discourse and social change, Polity Press, Cambridge. Fairclough N, (2000), New Labour, New Language?, Routledge, London, New York. Hodge R., Kress G., (1993), Language as ideology, Routledge, London. Lombardo, L. (2001). Selling it and telling it: a functional approach to the discourse of

print ads and TV news. Roma: Luiss. Montagna, E. (2009), Male logic, female intuition: Margaret Thatcher, “The Iron Doll”,

in Luraghi S., Il Mondo alla rovescia – Il potere delle donne”, Franco Angeli, Milano.

Montagna, E (in press) “Male logic, female intuition: Margaret Thatcher, all in one”. In Baldry, A.P., Montagna, E. (in press). Interdisciplinary perspectives on multimodality: theory and practice. Campobasso: Palladino.

Vasta N. (1997), “New Labour – New language? The moderniser’s vision between tradition and change”, in La costruzione linguistica della comunicazone politica, a cura di Donna R. Miller e N. Vasta, Padova,CEDAM.

Vasta N. (2001). Rallying Voters: New Labour’s Verbal-Visual Strategies. Padova: CEDAM.

21

University of Verona, 27 May 2009

MUSIC&SERIES: THE VERBALIZING ROLE OF SOUNDTRACKS IN CONTEMPORARY TV SERIES

Ilaria Moschini

In 1984 the creators of Miami Vice – a famous NBC detective drama – added pop and rock hits to the original themes of the soundtrack. This new use of music, later defined as “MTV-inspired”, has progressively increased in the last 10 years and has now become one of the widely accepted standards for music and images integration in US TV series and movies. Recently, it has also been fully exploited by users of social networking platforms like YouTube in designing their own videos. Actually, this metalinguistic use of songs influences the communicative structure of the filmic text because it modifies the function associated to musical comments. As a matter of fact, the prominent role is here given to lyrics, which literally ‘verbalize’ the characters’ emotions and often vehicle the authors’ point of view. The soundtrack is thus turned into a narrative instrument, a function that integrates the traditional role of melodies to “emotionalize the abstract and aestheticize the sensory” (Van Leeuwen, 1999). The aim of my paper is to analyze this new semiotic scenario within the methodological framework of multimodality (Kress&Van Leeuwen, 1996, 1998, 2002, 2007) with a special focus on two “resemiotization processes” (Iedema, 2003). The first process takes place at a macro level and involves the narrative strategies chosen to convey meaning through this new semiotic resource. The second process takes place at the level of each ‘musical moment’ and involves the concept of “transduction” - the shift of semiotic material across modes (Kress 2003). To be examined here is the reification of meaning stemming from the recontextualization of songs. Finally, it will be studied how these processes have to be considered in translating any filmic text in order to avoid the loss of a relevant portion of the actual meaning.

22

Symposium on Multimodal Approaches to Communication

‘GIVING THE GRADUATES AN EARFUL’: IDENTITY AND INTERACTION IN COMMENCEMENT SPEECHES

Martin Solly

Commencement speeches (CS) are traditionally delivered at graduation ceremonies in North American universities by prominent figures from various fields of endeavour. Originally addressed to a more restricted audience, these days CS constitute an increasingly public moment in the life of an academic institution: they are often widely reported in the national press as well as being available on the Internet. Part of a larger study, this paper looks at a small sample of CS in order to examine their communicative purposes and discourse features and presents some preliminary findings. In particular it focuses on two aspects of CS that are key factors in their communicative success: their portrayal of identity and their interactivity. Presumably intended to pass on words of advice and wisdom to the graduating students, CS speakers nonetheless appear to have almost complete freedom of license to talk about whatever they choose in their addresses, which can be highly personal and often have a significant autobiographical element. CS are of course examples of spoken discourse; at the same time they are usually the end product of careful planning and revision. They might even be the result of several hands working on the text (in the case of CS presented by senior politicians). The paper investigates the use of rhetorical strategies and lexico-grammatical devices in the texts, in order to identify underlying patterns in their discourse structure and organization. It also highlights some of the intrinsic dilemmas and ambiguities of CS, as the speakers ‘give the graduates an earful’.

23

University of Verona, 27 May 2009

24

Symposium on Multimodal Approaches to Communication

ABSTRACTS POSTERS:

- ILARIA FORNASINI (University of Verona): ‘Illuminated multimodality: Communicative dynamics in later medieval manuscripts’ .........................................................................................................................p. 26

- CATERINA GUARDAMAGNA (University of Pavia): ‘Medical Communication: A corpus of medical simulation/animation’ ...................p. 27

- ROSALBA RIZZO (University of Messina): ‘Multimodal and intercultural analysis of today’s online newspapers’ ......................p. 28

- MARIA GRAZIA SINDONI (University of Messina): ‘Mode switching: how oral and written modes alternate in videochats’ ..................p. 29

25

University of Verona, 27 May 2009

ILLUMINATED MULTIMODALITY: COMMUNICATIVE DYNAMICS IN LATE MEDIEVAL MANUSCRIPTS

Ilaria Fornasini

Taking the cue from a question posed by Lawrence G. Duggan (“Was art really the ‘Book of the Illiterate’?”), the contribution of the visual mode inside the manuscript media will be analyzed from a particular point of view. A significant number of British manuscripts which survived from the late Middle Ages presents illuminations in conjunction with their texts: the iconographic device was meant to ornate and embellish the page, as well as to illustrate what was explained in the text. Moreover, these illuminations often helped the reader, by clearly dividing the different chapters of the book. When set alongside the narration, the images constituted a sort of gloss, providing a visual comment to the text and even adding further details. The integration of visual and verbal resources and their interaction inside the manuscript as a medium is currently studied by the so-called New or Material philology. This kind of integrational philology owes much to the new linguistic theories: it is defined as interdisciplinary, re-traditionalizing and relativistic, and pays particular attention to the material artefacts, focusing on the historical situations of texts and the function of texts in time and place. This poster will analyze some examples taken from late medieval British manuscripts in order to illustrate, by means of a synchronic multimodal analysis, their peculiar communicative dynamics. References: Duggan, L. G., “Was art really the ‘Book of the Illiterate’?”, in Mostert, M. & Hageman,

M. (eds.), Reading images and texts. Medieval images and texts as forms of communication, Papers from the Third Utrecht Symposium on Medieval Literacy (Utrecht, 7-9 December 2000), Turnhout, Brepols Publishers, 2005.

Kress, G. & van Leeuwen, T., Reading images, London, New York, Routledge, 1996. Nichols, S. G., “Introduction: Philology in a Manuscript Culture”, in Speculum 65 (1990),

pp. 1-10. van Leeuwen, T. & Jewitt, C. (eds.), Handbook of visual analysis, London, Thousand

Oaks (CA), Sage Publications, 2001.

26

Symposium on Multimodal Approaches to Communication

MEDICAL COMMUNICATION: A CORPUS OF MEDICAL SIMULATIONS/ANIMATIONS

Caterina Guardamagna

As part of developments into multimodal corpus linguistics (Baldry and Thibault 2001, 2006, 2008), this poster illustrates work carried out in relation to a corpus of Medical Animation Demos. An MA Demo is an advertisement of a specialised kind, a free web sample that advertises a Medical Animation, a longer, purchasable film. Both text types typically illustrate body processes. Thus like an MA, an MA demo will zoom in on an artery, swivel it through 90° degrees, and further zoom in on a shaded and/or lesioned area. At the same time, an off-screen narrator will describe stages and cause-and-effect patterns in the onset of pathological conditions, together with associated surgical or pharmacological procedures. The narration is typically synchronized with visual selections and labellings of body parts and processes. MA demos thus use the resources of the cartoon genre to simulate body conditions and processes at the same time as they metatextually map scientific information onto the cartoon simulation, in visual and verbal ways that are strikingly different. They demonstrate that, like verbal resources, visual resources also distinguish between an event level and a comment level, thereby contributing significantly to the overall meaning-making processes being enacted. In so doing, MAs create a significant problem for multimodal description and analysis: how can visual processes be best described given that, while verbal processes have a long tradition of description, visual processes do not? The poster session will suggest that the solution to this complex problem lies in the brevity of MAs and the power of multimodal corpus linguistics to provide fine-grained analyses and comparisons of linguistic and visual patterns that provide clues to the nature of the visual processes at work in many film genres and ultimately as to how they integrate with verbal processes. References: Baldry, A.P., Guardamagna C. (in press). “Medical animations: their place in

multimodal corpora and concordancing”. In Baldry, A.P., Montagna, E. (in press). Interdisciplinary perspectives on multimodality: theory and practice. Campobasso: Palladino.

Baldry, A.P., Montagna, E. (in press). Interdisciplinary perspectives on multimodality: theory and practice. Campobasso: Palladino.

Baldry, A.P; Thibault, P. J. (2001). ‘Towards Multimodal Corpora’. In G. Aston and L. Burnard (eds.), Corpora in the Description and Teaching of English. Bologna: CLUEB, pp. 87-102.

Baldry A. P.; Thibault, P. J. (2006). ‘Multimodal Corpus Linguistics’. In G. Thompson and S. Hunston (eds), System and Corpus Exploring Connections, London and New York: Equinox, pp. 164-183.

Baldry, A.P.; Thibault, P. J. (2008). ‘Applications of multimodal concordances’. In Hermes: Journal of Language and Communication Studies: 41: 11-41.

27

University of Verona, 27 May 2009

MULTIMODAL AND INTERCULTURAL ANALYSIS OF TODAY’S ONLINE NEWSPAPERS

Rosalba Rizzo

This poster analyses the thematics of the current economic and financial crisis as represented in three online daily newspapers in different parts of the English-speaking world: South Africa, Australia and Ireland. These newspapers are less well known than major online newspapers such as The Guardian, The Times, The Washington Post, The New York Times and so on. However, we will be concerned with the different editorial policies adopted by these newspapers to inform their readership about the state of the market both domestically and worldwide. We will indicate the main differences in the overall textual and cultural approaches adopted and how intertextual relationships are created to more well-known newspapers. All this raises questions about how online newspapers construe their relationships with their readership in a global market. Are there in fact South African, Australian and Irish viewpoints? In attempting to answer this and other questions we will use a combined multimodal and intercultural approach. References: Baldry, Anthony, Thibault, Paul J., (2006), Multimodal Transcription and Text Analysis: A

Multimodal Toolkit and Coursebook with Associated On-line Course. London and New York: Equinox.

Morley, John (2006), “Sticky Business: A Case Study of Cohesion in the Language of Politics in The Economist”. In Linda Lombardo, Louann Haarman, John Morley and Christopher Taylor (eds.) Massed Medias: Linguistic Tools for Interpreting Media Discourse, Milano: LED, pp.19-83

Knox, John (in press), “Designing the news in an online newspaper: a systemic description”. In Anthony Baldry and Elena Montagna (eds.) Interdisciplinary perspectives on multimodality: theory and practice, Campobasso: Palladino.

28

Symposium on Multimodal Approaches to Communication

MODE SWITCHING: HOW ORAL AND WRITTEN MODES ALTERNATE IN VIDEOCHATS

Maria Grazia Sindoni

This poster sets out to illustrate forms of communication in which oral and written modes are integrated in ways that allow participants to “mode switch”, i.e. alternate between the two modes for specific communicative purposes. Videochats are the main focus of investigation as they exhibit a set of different traits, which, as well as mode switching, include new arrangements of interpersonal resources (e.g. non reciprocal gaze) and, more generally, a different systematization of verbal strategies and proxemic patterns that, to some extent, attempt to simulate real-life conversations. Three main steps are envisaged in this research:

1) Data is gathered through screencasting: i.e. digital recordings of computer screen output (also known as video screen capture) are made, which include audio narration and which capture the complexity of online interaction; screencasting of videochats is thus holistic making it possible to carry out fine-grained multimodal analysis of all the events and resources codeployed by participants;

2) The data is annotated and analysed systematically: the study of the integration of oral and written modes requires an equally fine-grained and integrated framework of analysis; the research draws on the theoretical support provided by current multimodal research, in particular such categories of multimodal film genre analysis as the investigation of visual frames, visual images, kinesic action and metafunctional interpretation (Baldry, Thibault, 2006); more generally it relies on systemic-functional grammar (Halliday and Mathiessen, 2004);

3) Reflections and conclusions are drawn vis-à-vis the potential contribution to the multimodal analysis of web genres: Web 2.0, with its vast range of emerging textual genres, is constantly expanding and challenging our notions of multimodal analysis. New textual configurations need to be explored using flexible tools, that take into account unprecedented systems of communication among web-users. In this sense, mode switching underlies a wealth of new systems of human interaction that raise philosophical and ethical questions, such as the extent of the extra (if any) communicative potential made available by Web 2.0 textual genres.

References: Baldry A. P.; Thibault, P.J. (2006). Multimodal Transcription and Text Analysis, London

and New York: Equinox. Halliday M.A.K.; Mathiessen C. (2004). An Introduction to Functional Grammar, Third

Edition, London: Arnold.

29

University of Verona, 27 May 2009

NOTES:

30