1. Introduction

Can the number of shared features between two concepts account for the similarity between the two concepts aligned in metaphors expressed in images versus language?

And what type of information is carried by these semantic features, in relation to the metaphor terms aligned in visual vs. linguistic metaphors? Are there differences in the way these two modalities of expression construct and express metaphors?

Semantic feature norms are basic attributes derived from human experiences with given concepts, which provide insights into core aspects of the content of rich and varied concepts. Such features are typically generated by speakers and standardized by researchers into feature norms, i.e., one-word labels or short sentences that describe a meaningful aspect of a concept, such as perceptual and non-perceptual properties, functions, and taxonomic relations (for example, given the concept “car”, the most frequent semantic features produced by informants typically are “has wheels”, “used for transportation”, and so on). The resulting concept–feature pairs are collected in datasets (such as McRae, Cree, Seidenberg, & McNorgan, Reference McRae, Cree, Seidenberg and McNorgan2005; Vinson & Vigliocco, Reference Vinson and Vigliocco2008; Recchia & Jones, Reference Recchia and Jones2011) and are generally enriched with metadata, such as production frequency and feature type. The latter piece of information is based on different coding schemes, such as, for example, the knowledge-based taxonomy (Wu & Barsalou, Reference Wu and Barsalou2009), the brain region taxonomy (Cree & McRae, Reference Cree and McRae2003), or schemes that have been adapted for the annotation of concrete and abstract concepts (5-category taxonomy in Vinson & Vigliocco, Reference Vinson and Vigliocco2008; 19-category taxonomy in Recchia & Jones, Reference Recchia and Jones2011). Concepts that share many features can be ascribed to the same conceptual category, under the assumption that the degree of similarity between two concepts can be operationalized in terms of feature matching (Tversky, Reference Tversky1977).

The question addressed in this paper is whether (and to what extent) the information encoded in semantic features can account for the similarity between two terms compared in visual or in linguistic metaphors. In this respect, it must be mentioned that a long-standing body of research suggests that the similarity between two terms that are compared in a metaphor can be based on the attributes that belong to the compared entities, which are typically perceptual properties, or on their relational properties, which are typically non-perceptual (e.g., Gentner, Reference Gentner, Vosniadou and Ortony1989). The present study exploits the information encoded in semantic features, which is mainly attributional rather than relational. ^{Footnote 1}

The type of information expressed by the collected features for the metaphor terms will be classified and compared across the two different modalities of expression, using a well-established taxonomy of semantic feature types (Wu & Barsalou, Reference Wu and Barsalou2009). This will allow us to observe different distributions of property types that speakers consider salient when they imagine given concepts that are used in metaphors.

Some methodological clarifications need to be made in relation to this study. First, within the body of literature in metaphor studies, the interaction view suggests that the similarity between two terms compared in a metaphor emerges specifically from their interaction, rather than being pre-existing (e.g., Black, Reference Black and Ortony1979). In this respect, the present study does not aim to cover all the possible cognitive operations that underlie metaphor comprehension. Rather, it is limited to modeling (and trying to predict) the functioning of those metaphors in which the similarity is somehow pre-existing (although latent and non-explicit) and is a part of the conceptual content of the metaphor domains, encoded in the semantic features. A number of studies that are currently in preparation will aim at exploring and modeling alternative types of cognitive operations afforded by metaphor (e.g., emergent similarity vs. pre-existing similarity, as addressed here).

Second, a long-standing body of literature in metaphor studies claims that conceptual metaphors are based on image schemas, (generic concepts such as source – path – goal, containment, or balance) which provide a scaffolding for grounding abstract concepts in human experience (e.g., Johnson, Reference Johnson1987; Lakoff, Reference Lakoff1987; Gibbs, Reference Gibbs2006). To the best of my knowledge there are no studies that explicitly compare and distinguish the type of information and the nature of the semantic representations based on image schemas vs. semantic features. I do believe that both types pertain to the conceptual dimension of meaning, but they provide access to conceptual content at different levels of abstraction, image schemas being more schematic and abstract, and semantic features being more closely tied to the perceptual experiences in which the concepts have been perceived.

In the present study I chose to compare visual metaphors to linguistic metaphors, to investigate how these two modalities afford the construction and representation of metaphor. To the best of my knowledge there are no other studies that aim at comparing the structure and functioning of these two modalities of expression in relation to metaphor. A recent body of work on visual and audiovisual representations has shown in a qualitative fashion how some of the conventional (conceptual) metaphors extracted from language use can be expressed in visuals as well (see, for example, Forceville, Reference Forceville and Fludernik2011; Hidalgo and Kralievic, Reference Hidalgo, Kraljevic, Gonzálvez García, Peña and Pérez-Hernández2011; and Yu, Reference Yu2008), as well as how primary metaphors can be realized visually (Ortiz, Reference Ortiz2011) and even image schemas (Pérez Hernández, Reference Pérez Hernández2014). The approach used by these authors is top-down: given a set of conventional conceptual metaphors, the scholars searched for possible realizations in the visual modality. The purpose of this study is different, in that it aims at observing how each of the two modalities of expression of metaphor functions, independently from one another, in a bottom-up fashion, in order to see potential differences emerge spontaneously. For this reason, the present study should be considered as a first exploratory analysis of these two modalities of expression.

The analyses hereby reported are based on two sets of materials that have been annotated for metaphoricity: a sample of metaphors extracted from the Metaphor Corpus (verbal modality) and a sample of metaphors extracted from the VisMet Corpus materials (visual modality). Given the nature of the two original corpora from which the materials have been sampled, the stimuli used for this analysis are considered to be representative for the two modalities of metaphor expression (respectively: verbal and visual).

Therefore, in this study we did not rely on the same set of conceptual metaphors, manifested in text and in pictures. This, however, would be an interesting (and different) experimental design, that can be tackled in further studies aimed at investigating the relationship between verbal and visual metaphor. Moreover, the results of this investigation, because of the methodology used to perform the analyses, do not prove that visual and verbal metaphors are processed differently in the brain. What the data support, in turn, is that the verbal and visual mode have different affordances for the representation of metaphors.

2. Theoretical background

The importance of semantic features as a proxy of the content of concepts is supported by a variety of studies aimed at unraveling mechanisms of conceptual processing, which suggest that human performance on different tasks involves the activation of such feature-related information. A few examples are:

• • Feature typicality: McRae, Cree, Westmacott, and de Sa (Reference McRae, Cree, Westmacott and de Sa1999) show that participants are faster at verifying that a feature applies to a specific concept if it is strongly rather than weakly correlated with other features of that concept. For example, they verify that “has a long tail” is a feature of rat faster than pony, because such a feature is highly correlated with the other features of rat, but not of pony. This suggests that the typicality of features predicts the retrieval of concepts.

• • Number-of-features (NoF) effects: processing words that have a high number of features (such as ambulance) is easier than processing words with a low number of features (such as ball), in categorization tasks, lexical decision, and naming responses (Pexman, Lupker, & Hino, Reference Pexman, Lupker and Hino2002; Pexman, Holyk, & Monfils, Reference Pexman, Holyk and Monfils2003; Grondin, Lupker, & McRae, Reference Grondin, Lupker and McRae2009). This suggests that words with many features are ‘anchored’ better in our semantic memory.

• • Effects of shared and distinctive features: distinctive features (features shared by only one or two concepts) hold a “privileged status” in meaning processing (Cree, McNorgan, & McRae, Reference Cree, McNorgan and McRae2006). For example, distinctive properties of living things are activated more slowly than shared properties (Randall, Moss, Rodd, Greer, & Tyler, Reference Randall, Moss, Rodd, Greer and Tyler2004).

Studies on semantic features have also been conducted to support the idea that the semantic information carried by these attributes and the semantic information carried by free word associations are different, and that they afford different processing mechanisms. An extensive review of these studies, as well as the implications that they have for conceptual vs. linguistic processing, is presented in Bolognesi, Pilgram, and Van den Heerik (unpublished observations). For example, McRae and Boisvert (Reference McRae and Boisvert1998) and Cree, McRae, and McNorgan (Reference Cree, McRae and McNorgan1999) found evidence of automatic priming for concept pairs that shared semantic features (spider–insect), while the same effect was not found for word pairs that were associatively related according to linguistic word associations (spider–web). Moreover, Wu and Barsalou (Reference Wu and Barsalou2009) showed that free word associations and semantic features that describe given concepts differ, by manipulating the experimental instructions given to participants, who were asked to produce descriptors for given stimuli. By asking participants to produce (i) descriptors for the mentally simulated concept, and (ii) free word associations, they obtained different sets of semantic information. In addition, Barsalou, Santos, Simmons, and Wilson (Reference Barsalou, Santos, Simmons, Wilson, De Vega, Glenberg and Graesser2008) found that, when participants are given a concept and asked to describe it, the linguistic associations peak before the semantic features that reveal a deeper mental simulation of the corresponding referent. Simmons, Hamann, Harenski, Hu, and Barsalou (Reference Simmons, Hamann, Harenski, Hu and Barsalou2008) added evidence from neuroimaging experiments, showing that when participants are given a concept and asked to describe it, they first use word associations (which activate left-hemisphere language areas such as Broca’s area), and then produce semantic features (which activate bilateral posterior areas, typically involved in mental imagery).

Assuming that metaphors are matters of thought (see Conceptual Metaphor Theory, from now on CMT: Lakoff & Johnson, Reference Lakoff and Johnson1980) rather than shallow figures of speech, semantic features, which tap into conceptual knowledge rather than shallow word associations, should provide insights into the similarity between concepts aligned in metaphors, regardless of the modality in which the metaphor is expressed. Qualitative differences between the types of features that account for the similarity of the domains, on the other hand, are expected, because of the peculiarity of each modality of expression.

In this respect, a variety of empirical studies suggest that pictures have privileged access to semantic memory compared to words (Shelton & Caramazza, Reference Shelton and Caramazza1999), and constitute a more direct realization of the underlying concepts compared to words (Binder, Desai, Graves, & Conant, Reference Binder, Desai, Graves and Conant2009). According to the literature, not all the perceptual information cued by visual representations of given concepts is encoded in language, or even available for us to be aware of in the first place, and in fact, many functional neuroimaging studies suggest different patterns of activation during matched word and picture recognition tasks (Gorno-Tempini et al., Reference Gorno-Tempini, Price, Josephs, Vandenberghe, Cappa, Kapur and Frackowiak1998; Moore & Price, Reference Moore and Price1999; Chee et al., Reference Chee, Weekes, Lee, Soon, Schreiber, Hoon and Chee2000; Hasson, Levy, Behrmann, Hendler, & Malach, Reference Hasson, Levy, Behrmann, Hendler and Malach2002; Bright, Moss, & Tyler, Reference Bright, Moss and Tyler2004; Gates & Yoon, Reference Gates and Yoon2005; Reinholz & Pollmann, Reference Reinholz and Pollmann2005). Moreover, as indicated in Binder et al.’s (Reference Binder, Desai, Graves and Conant2009) meta-analysis, different studies argue against a complete overlap between the knowledge systems underlying word and object recognition, based on the existence of patients with profound visual object recognition disorders but relatively intact word comprehension (Warrington, Reference Warrington and Frederiks1985; Farah, Reference Farah1990; Davidoff & De Bleser, Reference Davidoff and De Bleser1994).

Finally, a well-known and empirically supported account of cognition suggests that mental representations afford multiple coding strategies. In particular, the Dual Coding Theory claims that two functionally independent but interconnected multimodal systems provide mental representations of our conceptual knowledge. The two systems represent verbal and non-verbal knowledge, respectively (Paivio, Reference Paivio1971, Reference Paivio1986, Reference Paivio2010).

The present study seeks to explore whether the number of shared features between two concepts can account for their relatedness when the two concepts are aligned in an A-is-B metaphor. For this purpose, I chose to compare two modalities of expression (visual and verbal), and to explore whether the two modalities differ, with respect to the number and type of semantic features that the metaphorical domains evoke and (possibly) share.

Given the research questions stated above, and the literature reviewed in the previous paragraphs, I predict that metaphors expressed through images and metaphors expressed through words might trigger different sets of conceptual knowledge about the concepts involved, because the cognitive processing of visual vs. linguistic stimuli affords different routes to the semantic system, as well as different mental representations. In light of this, different types of semantic features might account for the similarity between metaphor terms of visual vs. verbal metaphors, because the modality in which the metaphor is expressed might direct our attention to different aspects of the content of the source and the target.

In order to do this, I collected thousands of instances of semantic features that describe the concepts compared in visual and verbal metaphors in experimental settings, and then compared the features produced for source–target pairs.

4. Data analysis

4.1. validating the dataset: correlation between the collected data and established databases of semantic features

In order to evaluate the quality of the semantic features collected, a correlation study was conducted to check the correlation between the collected data and two well-known databases of semantic features (MRsf and RJsf). The correlations were calculated on the forty-one concrete concepts that appeared in both METsf and MRsf, as well as on the forty-six concrete and abstract concepts that appeared in both METsf and RJsf. In addition, the correlation was also calculated between MRsf and RJsf (281 overlapping concepts).

The correlation coefficients were calculated as follows: since the number (and type) of features produced for each concept in the different databases varied, the representations of the concepts across the different databases were not pairwise comparable. For example, bomb has seventeen semantic features in METsf, five in RJsf, and fifteen in MRsf. Therefore, it is not possible to calculate the correlation between the semantic representation of bomb in METsf and RJsf, or METsf and MRsf, because the vectors for bomb have different lengths in the three databases. For this reason, the semantic representations were indirectly compared, through their relative similarities, within each database. In particular, following the procedure to create the similarity matrices described in McRae et al. (Reference McRae, Cree, Seidenberg and McNorgan2005), a table of similarities was computed for the overlapping concepts within each database. For example, for the forty-one overlapping concepts in METsf and MRsf, a contingency table was created first, where the forty-one concepts were displayed in the rows and all the features produced for all the forty-one concepts were displayed in the columns. The production frequencies for each concept-feature were reported in the cells. Then, the cosines ^{Footnote 8} were calculated between each pair of rows (for example: bomb–missile: 0.64; bomb–trumpet: 0.06; etc.), and reported in the final table of similarities, where the concepts appeared both in rows and columns, and the relative similarity (cosine) between each pair was displayed in the cells. The correlation coefficients were computed on the resulting tables of similarities, and are reported in Table 1. Since the correlations are very high, it can be claimed that the semantic features collected within the METsf database are methodologically and content-wise valid, because they are fully comparable to state-of-the-art semantic feature databases.

table 1. The average Pearson’s correlation coefficients between concepts that are shared across the databases of semantic features

4.2. computing similarity in terms of shared features between source and target domains

The semantic features were compared between the two conceptual domains aligned in every metaphor. On a qualitative basis it immediately emerged that there were many fewer cases of overlap of semantic features between metaphor terms in verbal metaphors compared to visual metaphors. In particular, only five (out of 50) verbal metaphors shared features between the two compared terms, while in visual metaphors this happened for twenty-six (out of 50) metaphors.

The cosine measurement was used to define the similarity between the two compared concepts in a metaphor, as is usual in this type of research (McRae et al., Reference McRae, Cree, Seidenberg and McNorgan2005; Recchia & Jones, Reference Recchia and Jones2011). The computation of the cosines was performed on the pairs of concept vectors extracted from the contingency table created on the basis of the METsf data (see previous section for the construction of the contingency table). Table 2 displays the average cosines (and therefore the similarity) between metaphor terms in the fifty verbal metaphors and in the fifty visual metaphors.

table 2. Average cosines between metaphorical domains in verbal and visual metaphors

As Table 2 shows, the cosines between metaphorical terms of visual metaphors are higher on average than the cosines between metaphorical terms in linguistic metaphors. The difference between the cosines in verbal and in visual metaphors is statistically significant (t = 3.282, p < .01).

This means that the information carried by the semantic features tends to capture the similarity between metaphorical terms aligned in visual metaphors more than in verbal metaphors. This finding suggests that visual and verbal metaphors indeed function in different ways, and in particular that the similarity between metaphorical domains expressed in the two modalities can be modeled by semantic feature matching quite well for visual metaphors, but that this is not the case for verbal metaphors. An in-depth analysis of the types of semantic features that the informants produced in response to the concepts used in visual vs. verbal metaphors was therefore required in order to see how such features differ.

5. General discussion

The present study explored how the visual and the verbal modality of expression afford metaphor construction and representation, and whether the shared semantic features between two concepts can account for the similarity between them when they are aligned as metaphor terms. The study, therefore, was based on both verbal and visual metaphors, which were compared and contrasted.

Since both metaphors expressed through images and metaphors expressed through words are assumed to be different manifestations of the same phenomenon, and are therefore both conceptual in nature (see CMT: Lakoff & Johnson, Reference Lakoff and Johnson1980), the method adopted from cognitive psychology (i.e., semantic feature norms) was expected to work equally well for both modalities, and possibly to provide different results in terms of types of features that can account for the similarity between domains. This prediction was based on the idea that different modalities of expression would trigger in the viewer/reader different sets of conceptual knowledge about the metaphorical domains, and therefore different types of shared semantic features might account for the similarity between metaphor domains of visual vs. verbal metaphors.

This prediction was partially confirmed: the metaphorical terms identified for the visual metaphors triggered features that mainly describe perceptual and systemic properties of the entity, as well as its function, while the metaphorical terms identified for the verbal metaphors triggered features that mainly describe taxonomic properties and introspections. In particular, using the method of semantic feature norms to explore the relatedness between metaphorical terms proved to be quite a reliable method when applied to visual metaphors, but it did not deliver positive results when it was applied to metaphors expressed through language. As a matter of fact, while for many visual metaphors it was possible to observe overlapping features between the metaphorical domains, this was not the case for verbal metaphors.

The results obtained in this exploratory study, which call for further investigation, could lead to the following explanation. As illustrated at the beginning of this paper, the visual mode of communication has privileged access to memory when compared to the verbal mode. In this sense, semantic features elicited from informants, who were explicitly asked to read a stimulus, create a mental image, and describe its contents, are probably the product of the semantic information which is encoded in the mental imagery. Such information comes into play when two concepts are compared in a metaphor expressed by visual means.

On the other hand, typical verbal metaphors, i.e., metaphor manifestations that are typically expressed indirectly in discourse, might function in a different way, and might need to be grounded in the conceptual system through a different route (rather than through the perceptual information retrieved from the mental imagery). In particular, for the A-is-B verbal correspondences derived from the application of the MIPVU procedure, the similarity between the two terms might need to be retrieved by looking at how the two terms are used in language, rather than how the two terms appear in the mental imagery. The similarity between two terms of a verbal metaphor might be a function of the semantic information that the two terms share in their verbal use. In addition, when the linguistic A-is-B correspondences retrieved through the MIPVU procedure (specifically implemented for linguistic metaphor identification) are analyzed at their conceptual level, and expressed by A-is-B conceptual correspondences (see Lakoff & Johnson, Reference Lakoff and Johnson1980), then the conceptual mappings might emerge in terms of shared semantic features, because our conceptual knowledge comprises information derived from both language and perceptual experiences.

This possible explanation taps into the distinction between a linguistic dimension and a conceptual dimension of meaning, where the linguistic information can be operationalized in terms of language statistics and word co-occurrences (see, for example, Louwerse & Hutchinson, Reference Louwerse and Hutchinson2012), while the conceptual information is formed on the basis of semantic information that we derive from language and the information that we derive from perceptual experience. This distinction seems to be supported in a variety of quite recent studies:

All these studies seem to acknowledge the fact that conceptual knowledge encompasses (at least) two sets of information streams, coming from language and from perceptual experience, and seem to imply that the information conveyed by these two streams of information does not fully overlap. The present exploratory investigation capitalizes on this distinction, suggesting that metaphors expressed through words and metaphors expressed through images might provide access to different information, which we use to make sense of the metaphorical comparison.

A final remark: one of the qualitative observations that emerged from the present investigation, suggests that a fair number of the verbal metaphors taken into account for the analysis seem to relate to structural metaphors such as EMOTION IS A PHYSICAL FORCE and IDEAS ARE OBJECTS, which are highly conventional and share an underlying experiential basis. On the other hand, a fair number of the visual metaphors taken into account seem to be low level, ad-hoc created, and novel, such as MOUTHWASH IS BOMB and RADIO IS A BEGGAR. For these metaphors, the perceptual resemblance is often exploited to make a connection between the two domains. This qualitative observation raises a new research question: How does the modality of expression relate to metaphor conventionality? In particular: Can the modality of expression (visual or verbal) be a predictor for metaphor conventionality? Such questions can be tackled in a future experimental study in which metaphor modality is the independent variable, and metaphor conventionality is the dependent variable.