This sixth volume in the Document Design Companion Series, like its predecessors, is devoted to issues of written, spoken, and visual (electronic) discourse as a contextual undertaking. While other volumes have roots in social semiotics, this one is unique for the breadth of its multimodal curiosity. Its cross-section of essays emerged from discussions that took place during the First International Symposium on Multimodal Discourse at the University of Salzburg. The symposium's organizers, who are also this book's editors, hope their work will foster discussion encompassing theory, method, and an eclectic array of applications, from the multisemiotic construction of mathematics to visual/verbal humor in comics. From their point of view, this work suggests possibilities for future study rather than fully realized principles in a field where nonlinguistic meaning making is only beginning to be incorporated into linguistic analysis. Therefore, one can often forgive the uneven nature of this undertaking. Stronger concerns arise when problematic or missing information affects a central claim.
The book consists of 12 chapters, organized into two parts. Part I deals with theory and method. The eight chapters in Part II consider multimodal application and analysis. Theoretical interests begin with Hartmut Stöckl's hierarchically structured and networked system of sensory channels (visual/auditory), core modes (image/language), medial variants (static/dynamic), peripheral modes (such as typography), submodes (such as gesture), and features (such as hue). Within this network, Stöckl considers how modes integrate and how they are distinct. One distinction concerning cognitive orientation, which claims images are “based on simultaneous and holistic gestalt-perception” (p. 17), requires immediate discussion. As the seminal work of the cognitive psychologist Allan Paivio (1986) has demonstrated, images are not simultaneously processed; they are simultaneously available for processing. The limitations of vision demand that viewers make “saccadic” jumps from one area of interest to another. Further, eyescan studies on images (Buswell 1935, Norton & Stark 1971) show that individual scan paths do not map onto each other even if individuals often alight on some shared areas of an image – complicating linguistic analysis. These facts might not seem evident because the examples shown contain limited visual information. Additionally, the analysis directs the eye to particular image elements. Although distinction/integration perspectives have much to offer, and in fact have an important place in Paivio's (1986) work, Stöckl's perspective, as potentially useful as it could be, will be affected by corrected distinctions.
On a different track, Peter Muntigl's theoretical perspective argues that gesture is a semiotic system containing semantic, lexicogrammatic, and expression planes as represented in systemic functional linguistic approaches. He backs up his argument with an example of pool players using gesture and language to recount a shot. His work builds to the intriguing proposition that mode – adapted from Ruqaiya Hasan's combination of language role, medium, and sensory channel – activates different semiotic systems, such as gesture, depending on the aspects of mode that are available and/or necessary. However, the argument allows one aspect of activation to live under the radar. As Kress & Van Leeuwen (2001:126) point out, only “when one community invests ‘work’” in forms like gesture can those forms become fully developed. While the example used in this chapter represents a discourse community of pool players, the author's work overall seems to suggest a more general application. Further development of this intriguing proposition might yield even more intriguing conclusions.
The cross-section of theoretical perspectives concludes with Victor Lim Fei's careful look at the difference between semiotic resources and semiotic systems, and in particular, how images live within each. While semiotic resources have content and expression planes, semiotic systems contain potential but still unrealized meaning within each of those planes. Images are semiotic resources, while the lines, shading, and shape that underlie images are semiotic systems. Meaning is realized when unifying relationships hold elements together to resemble such things as faces. Semiotic resources, of all types, offer “differing degrees of arbitrariness between the signifiers and the signifieds” (59). Words are the more arbitrary building blocks of meaning, whereas icons are less arbitrary. Both can be ambiguous until surrounded by co-text. What constitutes co-text in images is left to the imagination, as is the nature of the vocabulary icons might produce, or the purpose of lines or shapes that stubbornly maintain a sense of ambiguity even in the presence of language. However, the author himself allays concern by stating in both the abstract and the conclusion that he is well aware of the preliminary nature of this work.
Part I ends with an excellent contribution to method. Its authors, John Bateman, Judy Delin, and Renate Henschel, propose a design for empirically evaluating multimodal claims, which they fear are sometimes based on “impressionistic interpretations” (67) that do not necessarily hold up under closer scrutiny. The authors' project, known as Genre and Modality (GeM), builds on state-of-the-art corpus preparation that can address specific claims about images by using participant feedback. Additionally, problems in producing annotated visual/verbal texts are addressed. Although these authors admit to concerns about empirical approaches to multimodality, they continue in the hope their findings may limit problematic theories.
Part II, “Analyses and applications,” allows contributions to spread across an eclectic array of multimodal interests. Kay L. O'Halloran begins with a fascinating look at multisemiotic presentation in mathematics. This work takes a historical look at the opportunities and limitations in language, symbol, and visual display, which both enrich and restrict the constructions of reality that mathematics attempts to explain. The argument becomes even more compelling when visual dynamics, a later development, again alter what mathematicians can consider. But multimodality does not always aid learning, as Martin Kaltenbacher shows in the next chapter. The specific example he analyzes is a multimodal CD-ROM aimed at foreign language learners. Kaltenbacher demonstrates why pictures might take learners off course when the thousand words an image paints do not help evoke the new words presented in a new language.
A quick shift to a cultural studies focus in Markus Rheindorf's chapter, engages transdisciplinary analysis by combining cultural studies with the linguistic and semiotic concepts capable of doing the analytic work. Though I claim no expertise in cultural studies, this work seems to be an interesting approach to genre in film, in particular the film Dirty Dancing. In the next chapter, by Christopher Taylor, film is also the subject of analysis, but the focus changes to the problem of translation in subtitling, which is a particularly difficult form of translation because it involves text shortening. Taylor presents a multimodal approach that evaluates subtitling options based on the idea that “equivalent effect” (155) can often be identified in other modalities. The approach is employed in examples ranging from documentary to dark humor.
Translation is a pragmatic activity for Taylor, but Klaus Kaindl, in a carefully reasoned contribution, argues that translation studies done for the purpose of analyzing the humorous effect of comics have for too long focused on examples in which language plays the key role. Kaindl considers, instead, examples in which humor is present only because of their multimodal composition. Translation in comics segues into the interpretation of museum experience by Andrea Hofinger & Eija Ventola, based on Systemic Functional Linguistics (Halliday 1978). The authors' interest in interactions between pictures and spoken language, rather than movement through space, almost quiets the nagging concern that their work could benefit from the addition of an ethnographic element.
Eva Martha Eckkrammer's analysis of medical self-counseling texts and hypertexts as a process of multimodal interaction, while insightful, is problematic. The work builds to a central concern with the lack of multimodality in these self-help texts, leading to a claim that relevant images might improve comprehension and retention. In fact, there is a large body of work (Carney & Levin 2002, Mayer 2002, Plass et al. 1998) that already demonstrates how improvements in learning emerge when relevant images enhance text. To her credit, Eckkrammer herself states that “it will be necessary for linguistics to integrate approaches from other disciplines” (216). The final chapter of the section, also with a health care theme, is not the strongest, although the question addressed is important. The author, Kristin Bührig, considers the problem of informed consent in situations where nonnative patients must make medical decisions. Specifically, can labeled diagrams of the body help untrained interpreters communicate with nonnative patients? While the problem itself is compelling, the study contains too many open variables to produce reliable conclusions.
Even though this book's weaker chapters, and missing or inaccurate information, remain troubling, the volume can still be seen as useful for the array of ideas it presents, some intriguing, others admirable, which also produce an intertextual discussion from one chapter to the next. As that discussion continues outside this book, it might be useful to remember Eckkrammer's advice to look beyond the discipline as well as deep within it.
