Verbal and Verbal-Visual Logico-Semantic Relations in Picturebooks: an English-Brazilian Portuguese Parallel Corpus Study

Abstract:

This article presents a methodology to analyze verbal and verbal-visual logico-semantic relations in picturebooks originally written in English and their translations into Brazilian Portuguese. Drawing on systemic-functional theory (Halliday & Matthiessen, 2014) and on Visual Grammar (Kress & van Leeuwen, 2006), verbal and visual texts in picturebooks were annotated and analyzed according to the logico-semantic relations identified in them. Results revealed a higher number of clause complexes than simplexes. As regards the type of logico-semantic relations that tend to occur most frequently between clause complexes in verbal text and between verbal and visual text, in both source and translated texts, Projection_Locution is the most frequent logico-semantic relation between clause complexes, while Expansion_Extension is the most frequent logico-semantic relation between verbal and visual text. Differences between source and target texts are basically related to the number of clause complexes. The study brings an original contribution to systemic functional theory and translation studies, as it shows typical patterns of logico-semantic relations in picturebooks and offers a proposal to categorize logico-semantic relations in verbal text and between visual and verbal text that proves to be fully operational for annotation purposes and text analysis.

Keywords:
Translation; Multimodality; Children’s Literature; Picturebooks; Logico-Semantic Relations

1. Introduction

This article reports on a study of logico-semantic relations in young children's picturebooks. The study drew on systemic functional theory (Halliday & Matthiessen, 2014) and covered two complementary perspectives: 1) logico-semantic relations between clauses in clause complexes in verbal text (hereinafter referred to as verbal LSRs); and 2) logico-semantic relations between clauses in verbal text and images in visual text, henceforth referred to as verbal-visual LSRs. The aim was to explore a methodology to analyze verbal and verbal-visual LSRs and examine those relations in picturebooks. Additionally, the study sought to contribute to a comparison of both verbal and verbal-visual LSRs in picturebooks originally written in English and their translations into Brazilian Portuguese.

2. State of the Art

The translation of children’s literature has been in the research agenda of Translation Studies since the 1960s, having gained prominence in the 1980s with studies on the norms governing the production and reception of translated texts in the target culture (see, for instance, Shavit 1981). With the development of corpus-based translation studies, reliance on manual annotation and inspection of texts gave way to compilation of corpora of children’s literature with a myriad of comparisons enabled between both originals and translations as well as translated and non-translated texts (see as an example Puurtinen’s (1998) work on a parallel and a comparable corpus of children's literature in English, its translation into Finnish, and children’s literature originally written in Finnish). From the 2000s on, a growing interest in multimodality allowed for studies exploring how meaning is created through a complex process of interaction of verbal and visual texts, as seen in Van Meerbegen’s (2009). The author examines the translation of picturebooks from Dutch and Flemish into Swedish with a view to exploring the translation of picturebooks as a process of co-production through which visual components of the source text are woven together with new verbal components of the translated text and bring about a new distinct verbal and visual interaction. Works such as Van Meerbegen’s (2009) seek to integrate a multimodal analysis based on Kress & Van Leeuwen’s grammar of visual design (2006) into descriptive studies of translation (Toury, 1995) in order to verify how social meaning is created through words, images, and the interaction between them.

Outside the discipline of Translation Studies, though also building on Kress & van Leeuwen (2006), language-image interaction has been the object of studies developed within a systemic-functional approach to language (Halliday & Matthiessen, 2014). Unsworth (2006), for one, develops an SFL-informed methodology for multimodal texts to investigate the dynamics of the interaction between text and image which is of particular interest to our present research. Modelled upon ideational meanings, his proposal considers three main types of interaction: “ideational concurrence”, “ideational complementarity”, and “ideational connection”. Ideational concurrence is described by Unsworth (2006) as equivalence in the configuration of experiential functions (Participants and Processes), there being four possible scenarios: 1) Redundancy, when there is convergence of ideational meaning in both verbal and visual modes; 2) Exposition, when text and image construe meanings at an analogous level, neither of them being more or less specific than the other; 3) Instantiation, when one mode presents an instance of what is represented by the other; 4) Homospatiality, when verbal and visual modes construe meaning within a single space and as one single entity. Ideational Complementarity refers to situations when information in verbal and in visual texts differs but jointly contributes to meaning. Complementarity can be subclassified into two categories: i) Augmentation, when a piece of information in one mode complements information present in the other mode; an image can extend verbal text, or verbal text can extend an image; and ii) Divergence, when there is conflicting information in verbal and visual text. Finally, Ideational connection operates in two different ways: a) Projection, which can be verbal or mental, involving quotes or reported speech and thought; and b) Conjunction, which is verbal-visual interaction involving causal, temporal, or spatial conjunctive relations.

Our study builds on Unsworth (2006) for the description of ideational meanings and on Halliday & Matthiessen’s (2014) grammatical categories to analyze verbal LSRs and verbal-visual LSRs. Our theoretical framework will be briefly presented below with the categories used in our study being detailed in the Methodology section.

Clause and image as representation

Systemic-Functional Theory posits that, from the point of view of a linguist, the text “is a rich, many-faceted phenomenon that ‘means’ in many different ways” (Halliday & Matthiessen, 2014, p. 3) and consequently offers several possibilities of analysis. From the perspective of lexicogrammar, the clause is the basic unit of analysis, since “it is in the clause that meanings of different kinds are mapped into an integrated grammatical structure” (Halliday & Matthiessen, 2004, p.10). In other words, the clause is the site where three main systems (TRANSITIVITY, MODE, and THEME) linked to the three SFL metafunctions (ideational/experiential, interpersonal, and textual) operate, simultaneously performing three different functions and, as a consequence, construing, enacting and constructing three kinds of meanings.

Through the system of THEME, the textual metafunction organizes meanings in the clause as message, thematic position being an important choice in the construction of discourse.

Through the system of MODE, the interpersonal metafunction organizes meanings in the clause as exchange.

Through the system of TRANSITIVITY, the ideational/experiential metafunction organizes meanings in the clause as representation.

Our work focuses on the ideational/experiential metafunction, that is, the construal of representational meanings.

Our experience of the world is represented through a configuration of Processes, Participants, and Circumstances, which are grammatically realized by verbal groups, nominal groups, and adverbial groups and prepositional phrases, respectively (Halliday & Matthiessen, 2014), as illustrated in the example in Table 1 below:

Table 1:
Example of grammatical realization of the functions of the TRANSITIVITY system

	A dirty fairy	Fell	down the chimney
Clause Function	Participant	(Material) Process	Circumstance (of place)
Group realization	Nominal group	Verbal group	Prepositional phrase

Analogous to the three metafunctional strands proposed by SFL, Kress & van Leeuwen (2006, p. 20) present a model for reading images based on the assumption that “as a resource for representation, images, like language, will display regularities, which can be made the subject of relatively formal description”. The authors build on the metafunctions posited by Halliday & Matthiessen (2004) in order to describe how visual language: 1) represents the experience (representational meaning); 2) establishes relationships with the observer(s) (interactive meaning); 3) is organized as a visual structure (compositional meaning).

As far as representational meaning is concerned, Kress & van Leeuwen (2006, p. 20) argue that, just like in verbal language, images construe objects and their relations in the world. In other words, we can also identify Participants, Processes, and Circumstances in images. Participants are the most prominent “volumes” in images. Processes can enter into basically two kinds of structures in images: i) narrative patterns: when actions, events, processes of change, and arrangements are represented by vectors (formed by parts of the body or tools, for example) which connect the Participants; ii) conceptual patterns: when Participants are represented regarding their class, structure, or meaning. Circumstances, on the other hand, are Participants which do not directly take part in narrative processes, being mainly references to time, place, and manner in images.

In Figure 1 ² below, we can see one Participant in the foreground and five Participants in the background. The Participant in the foreground-a young boy --is on the beach (a Circumstance). He is sitting and scratching his knee and ankle, Processes represented by the vectors that can be drawn along the Participant’s arms and legs, as shown in Figure 2.

Figure 1:
Example of representational meanings in an image

Figure 2:
Example of Participants and Processes in images modelled as volumes and vectors drawing on Kress & van Leeuwen (2006)

Figure 2 shows Participants, i.e. the “volumes” in the image (Kress & van Leeuwen, 2006) represented in black, and the Processes related to the Participant in the foreground as vectors represented in white. The setting (the beach) is a Circumstance of place and the blue sky construes time of the day (a Circumstance of temporal localization).

Within the scope of the present work, Kress & van Leeuwen’s (2006) model is used in the description and annotation of the ideational meanings of images, a methodological step which, together with the annotation of ideational meanings in clauses, allowed us to classify the verbal-visual LSRs in the analyzed picturebooks, as described in our Methodology.

Verbal LSRs between clauses

Halliday & Matthiessen (2014) group verbal LSRs between clauses into two basic systems: Expansion and Projection. While Expansion connects sequences of events and/or symbolic relationships, Projection relates locutions (what someone says) and/or ideas (what someone thinks).

There are three main types of Expansion: 1) Elaborating; 2) Extension); 3) Enhancing. Elaborating expansion takes place when “[...] one clause elaborates on the meaning of another by further specifying or describing it [...]” (Halliday & Matthiessen, 2014, p. 461). There is no new information in the message brought about by the second clause, but there is further information about what has already been said, as the example in Table 2 illustrates.

Table 2:
Example of clauses with LSR of Elaboration.

ele agora é vegê, [he is now a veggie]	não come mais carne [he no longer eats meat]
Clause 1	Clause 2

• Source: Ahlberg & Ahlberg, 1991

Extension takes place when “[...] one clause extends the meaning of another by adding something new to it. What is added may be just an addition, or else a replacement, or an alternative” (Halliday & Matthiessen, 2014, p. 471). The example in Table 3, taken from the corpus of our research, illustrates this case through a clause complex.

Table 3:
Example of clauses with LSR of Extension.

The car was too small to drive	But he made the best of it.
Clause 1	Clause 2

Enhancing takes place when “[...] one clause (or subcomplex) enhances the meaning of another by qualifying it in one of a number of possible ways: by reference to time, place, manner, cause or condition” (Halliday & Matthiessen, 2014, p. 476). The example in Table 4 illustrates how clause 1 enhances clause 2 by reference to a cause.

Table 4:
Example of clauses with LSR of Enhancing.

Because she was very pretty and rich,	all the princes wanted her to be their Mrs.
Clause 1	Clause 2

Projection takes place when “[...] the secondary clause is projected through the primary clause, which instantiates it as (a) a locution or (b) an idea” (Halliday & Matthiessen, 2014, p. 443). The examples in Tables 5 and 6 illustrate locution and idea, respectively.

Table 5:
Example of clauses with LSR of Projection_locution.

“Guess how much I love you”,	he said.
Clause 1	Clause 2

Table 6:
Example of clauses with LSR of Projection_idea.

“Hmm, that is a lot”,	thought Little Nutbrown Hare.
Clause 1	Clause 2

3. Methodology

3. 1 Corpus

For this work, 287 clauses and 44 images were selected from 2 picturebooks, the originals written in English and their translations into Brazilian Portuguese. Titles, authors, and translators of Source Texts (STs) and Translated Texts (TTs) are shown in Table 7.

Table 7:
Picturebooks composing the analyzed corpus.

Source Text title	Author	Translated Text title	Translator
Guess how much I love you	Sam McBrateney/ Anita Jeram	Adivinha quanto eu te amo	Fernando Nuno
Prince Cinders	Babette Cole	Príncipe Cinderelo	Monica Stahel

3. 2 Methodological procedures

An outline of the methodological steps taken for data collection in our study is presented in Figure 3, followed by a brief explanation of each step.

Figure 3:
Methodological steps.

3. 2. 1 Corpus preparation (verbal and visual texts)

Manual analysis was chosen due to the size and type of corpus. The following steps were taken to prepare texts for analysis: 1) verbal and visual text was scanned; 2) verbal text was saved in.txt files and visual text was saved in.JPG files; 3) clauses in verbal text were segmented and pasted onto spreadsheets for the annotation of verbal LSRs between them; 4) images were numbered and their corresponding numbers were ordered on spreadsheets. Separate spreadsheets were used for each analyzed text in order to map and annotate the verbal-visual LSRs, by bringing together information about the clauses and their corresponding images. On each sheet, 6 tabs were created: ii) two tabs for the annotation of LSRs regarding Participants in STs and TTs; ii) two tabs for the annotation of LSRs concerning Processes in STs and TTs; iii) two tabs for the annotation of LSRs regarding Circumstances in STs and TTs.

3.2.2 Annotation of verbal LSRs between clauses

Verbal LSRs between clauses were annotated on a spreadsheet based on the categories in Halliday & Matthiessen (2014), as shown in Figure 4.

Figure 4:
Screen shot of spreadsheet annotation of verbal LSRs between clauses.

When the clause did not enter into a relation with another clause, it was annotated as simplex. The example in Table 8 illustrates this type of occurrence.

Table 8:
Example of a clause simplex.

Then Little Nutbrown Hare had a good idea.

Simplex

3.2.3 Annotation of ideational meanings in verbal and visual texts

Ideational meanings (Participants, Processes, and Circumstances) construed in the clauses were annotated on a spreadsheet based on the categories in Halliday & Matthiessen (2014), as can be seen in Figure 5.

Figure 5:
Screen shot of spreadsheet annotation of ideational meanings in verbal text.

Representational meanings (Participants, Processes, and Circumstances) in images were annotated based on Kress & van Leeuwen (2006). A spreadsheet with annotation is shown in Figure 6.

Figure 6:
Screen shot of spreadsheet annotation of ideational meanings in visual text.

3.2.4 Annotation of the verbal-visual LSRs

Verbal-visual LSRs were annotated on a spreadsheet which brought together information of ideational meanings of the clauses (on the left) and representational meanings of images (on the right) from the annotations previously carried out in separate spreadsheets, as shown in Figure 7. Verbal-visual LSRs were annotated in the central column of the spreadsheet.

Figure 7:
Screenshot of spreadsheet annotation of verbal-visual LSRs.

If a piece of text did not enter into a relation with an image, or an image did not enter into a relation with a piece of text, it was annotated as simplex.

In order to establish verbal-visual LSRs, we mapped correspondences between categories of verbal and visual text drawing on Kress & van Leeuwen (2006).

The categories proposed in our study were grouped on the basis of two basic systems, as presented below:

Projection: an image projects a second order clause, by presenting speech or thought in bubbles (Figure 8).

Figure 8:
Example of Projection.

Expansion: an image expands the ideational meaning of a clause, or a clause expands the ideational meaning of an image. The same clause can present different kinds of relations of Expansion with an image, as we have different elements construing representations in both verbal and visual texts. Unlike Projection, no speech or thought bubbles occur. There may be Projection in verbal text, but this operates within verbal text, not being projected by visual text, as Figure 8 illustrates.

There are three types of Expansion:

2.1) Elaborating: there is (essentially) no new element being added by the verbal text when related to the visual text, nor new element being added by the visual text when related to the verbal text. That is, Participants, Processes, and Circumstances in the verbal text are shown, specified, or detailed in the visual text or vice-versa.

Elaborating is subclassified into two categories according to which mode elaborates the other, assuming a verbal reading pattern from left to right and top to bottom, as shown below.

i) Verbal text elaborates visual text

Verbal text elaborates visual text when the visual text is the anchor of a message; that is, the visual text is the first ideational element, being placed left and before the verbal text is introduced, or above the verbal text, as shown in Figure 9.

Figure 9:
Example of verbal text elaborating visual text

In the example above, there is Elaboration of Participant (“the dog”), Elaboration of the Process (Relational Process in the verbal text “is” and Conceptual representation in the visual text), and Elaboration of the Circumstance of location: place (“in the bucket”).

ii) Visual text elaborates verbal text

Visual text elaborates verbal text when the verbal text is the anchor of the message; that is, the verbal text is the first ideational element, being placed left and before the visual text is introduced, or above the visual text, as shown in Figure 10.

Figure 10:
Example of visual text elaborating verbal text

In the example above, similarly to that in Figure 9, there is Elaboration of Participant, Process, and Circumstance. The two examples differ in that the first element of the message in Figure 9 is visual text, whereas in Figure 10 it is verbal text.

It is important to mention that position, i.e., whether visual text or verbal text comes first, does play a decisive role in this case, because, as mentioned and illustrated by Figures 9 and 10, when we have Elaboration, all three of the ideational/experiential functions (Participant, Processes and Circumstances) are present in both the verbal and the visual texts, the main difference being the position in which they are placed, assuming a left-to-right and top-to-bottom verbal reading pattern.

2.2) Extension: an image adds new information to verbal text by showing new Participants, Processes, or Circumstances, or verbal text presents Participants, Processes, or Circumstances that are not shown in an image. In this case, the verbal text can extend the meaning by adding new information to or by contradicting the visual text, or vice-versa. Besides, when there is projection between the clauses in verbal text, it is considered that verbal text extends visual text for Participants such as I and you. Extension is further classified into the subcategories below illustrated with examples.

Regarding the LSR of Extension, it is important to mention that the order in which verbal and visual text appear in the picturebook, that is, which comes first and next, above or below, is not decisive to subcategorize Extension. This is due to the fact that what is taken into consideration during annotation is which ideational/experiential function (Participant, Process or Circumstance) will be adding new meaning to the visual or verbal text or contradicting one or the other.

Extension is subclassified into two categories:

i) Verbal text extends visual text

Figure 11:
Example of verbal text extending visual text

In the example in Figure 11 above, there is a projected clause in verbal text, and there is Extension of Participant. “Toto”, the addressee of a command in the verbal text, is extended as being a dog (portrayed in the image) and Extension of Process a Material Process in the verbal text (“get out”), is extended as being a movement of displacement of the bucket, that is not shown in the visual text.

ii) Visual text extends verbal text

Figure 12:
Example of visual text extending verbal text

In the example above (Figure 12), visual text extends verbal text, as the Circumstance of location: place in the verbal text (“in his doggie house”) is contradicted by the visual text, as the dog is not inside a conventional doggie house, but a bucket.

2.3) Enhancing: verbal text presents conjunctive relations of time, place, manner, cause, or condition with visual text, or the other way round. That means that there are Circumstances in the verbal text enhancing the meaning of the visual text, or the visual text can enhance the verbal text with Circumstances. Enhancing was divided into the two subcategories below illustrated with examples.

As far as Enhancing LSRs are concerned, the order in which verbal and visual text appears, that is, what comes first and next, above or below, is not a criterion for subcategorizing Enhancing. The main criterion is the presence of any Circumstantial element in the visual text that is not present in the verbal text and vice-versa.

Enhancing is subclassified into two categories:

i) Verbal text enhances visual text

Figure 13:
Example of verbal text enhancing visual text

In the example in Figure 13, verbal text enhances visual text as there is a Circumstance of cause, which can be construed from the verbal text and thus worded: “Toto is in the bucket because he wants a bath”.

ii) Visual text enhances verbal text

Figure 14:
Example of visual text enhancing verbal text

In the example in Figure 14, visual text enhances verbal text as there is a Circumstance of place in the picture, which is related to a Material Process (“play”) in the verbal text and can thus be worded: Toto is playing in the bucket.

Using the categories hitherto presented, LSRs in original texts and translated texts were annotated. Once the annotation was concluded, data processing was done automatically in a spreadsheet editor in order to obtain quantitative results for frequency of occurrence of annotated categories.

The results obtained are presented in the following section.

4. Results

In this section, we present the main results from the annotation of the verbal LSRs between clauses complexes and the verbal-visual LSRs. The STs results are presented first, followed by the TTs results and the comparison between them. Tables display results in decreasing order in frequency of occurrence.

4.1 Verbal LSRs between clause complexes

4.1.1 STs results

The annotation of verbal LSRs between clauses in clauses complexes revealed that 72% were part of clause complexes, while 28% of the clauses were simplexes, that is, were not linked as to form a clause complex, as shown in Table 9.

Table 9:
Clause types in the ST.

Clause type	Frequency (%)
Clause complex	72%
Simplex	28%
Total	100%

This result suggests that the tendency in the analyzed picturebooks is for clauses to be part of clause complexes and therefore establish some kind of LSR with each other. Types and frequencies of occurrence of LSRs are presented in Table 10:

Table 10:
Verbal LSRs between clauses in the ST.

LSR between clauses	Frequency (%)
projection_locution	37%
expansion_extension	30%
expansion_enhancement	19%
projection_idea	14%
expansion_elaboration	0%
Total	100%

According to Table 10, Projection_locution is the most frequent verbal LSR between clause complexes (37%). This result can be accounted for by the large amount of dialogue in one of the picturebooks (Guess how much I love you) included in the analyzed corpus, as illustrated by the example in Table 11 below.

Table 11:
Example of Projection_Locution.

“But I love you this much	he said
Clause 1	Clause 2
	Projection_locution

As shown in Table 10, Extension is the second most frequent LSR between clause complexes, with a frequency of occurrence of 30%. The example in Table 12 illustrates this type of occurrence.

Table 12:
Exampleof Extension.

“The car was too small to drive	but he made the best of it.
Clause 1	Clause 2
Expansion_extension

Finally, as shown in Table 10, while Expansion_enhancement and Projection_idea were also identified as LSRs between the clause complexes, their frequency of occurrence was relatively lower (19% and 14%, respectively) when compared with Projection_locution and Expansion_Extension (37% and 30%, respectively). On the other hand, no single occurrence of Expansion_elaboration was found in the analyzed clause complexes. Examples of LSRs of Expansion_enhancement and Projection_idea are presented below (Tables 13 and 14).

Table 13:
Example of Expansion_enhancement.

“But when he arrived at The Royal Rave up,	but was too big to fit through the door!
Clause 1	Clause 2
Expansion_enhancement

Table 14:
Example of Projection_idea.

“Hmm, that is a lot,	thought Little Nutbrown Hare.
Clause 1	Clause 2
Projection_idea

4.1.2 TTs results

When it comes to the TTs, the results show that 70% of the clauses were linked forming clause complexes, while 30% of the clauses were simplexes, as shown in Table 15.

Table 15:
Clause types in the TTs.

Clause type	Frequency (%)
Clause complex	70%
Simplex	30%
Total	100%

These frequencies are very close to those found in STs and reveal that clauses in the TTs also tend to be linked through clause complexes and establish LSRs. Projection_locution is the most frequent LSR between clause complexes (35%), followed by Expansion_extension (34%), as presented in Table 16 and illustrated by the examples shown in Tables 17 and 18 below.

Table 16:
LSR between clauses in the TTs.

LSR between clauses	Frequency (%)
projection_locution	35%
expansion_extension	34%
projection_idea	17%
expansion_enhancement	14%
expansion_elaboration	0%
Total	100%

Table 17:
Exampleof Projection_locution.

“Adivinha o quanto eu te amo”	disse ele.
Clause 1	Clause 2
Projection_locution

Table 18:
Example of Expansion_extension.

O carro era muito pequeno,	mesmo assim ele deu um jeito.
Clause 1	Clause 2
Expansion_extension

Table 16 also reveals that Expansion_Extension and Projection_ideas have a slightly higher frequency of occurrence in translated clause complexes (34% and 17% in the TTs against 30% and 14% in the ST, respectively), while Projection_locution and Expansion_enhancement have a slightly lower frequency of occurrence (37% and 19% in the STs against 35% and 14% in the TTs, respectively). These changes in frequencies of occurrence regarding LSRs between ST and TT might be due to shifts in how LSRs are translated into Brazilian Portuguese. They are worth further investigation; however, this exceeds the scope of the present paper.

No single occurrence of Expansion_elaboration was found in the translated clause complexes. Examples of LSRs of Expansion_enhancement and Projection_idea are presented below (Tables 19 and 20).

Table 19:
Example.of Expansion_enhancement.

Mas, quando chegou ao Embalo Real,	percebeu que era muito grande para passar pela porta.
Clause 1	Clause 2
Expansion_enhancement

Table 20:
Example of Projection_idea.

Hum, isso é um bocado,	pensou o Coelhinho.
Clause 1	Clause 2
Projection_ideia

The fact that Elaboration was not found in clause complexes in the analyzed corpus is interesting when compared to the verbal-visual results which will be presented in the next section.

4.2 Verbal-Visual LSRs

In the annotation of verbal-visual LSRs, no occurrences of simplexes between verbal text and visual text were found. In other words, in the analyzed corpus, there is always a relation between text and image. This points to a relevant feature of picturebooks: the strong link between verbal-visual texts, which accounts for their being labelled with that particular name.

4.2.1 STs results

The annotation of the verbal-visual relations revealed only LSRs of Expansion. Among annotated categories, in terms of percentage of total number of occurrences, Expansion_Extension is the main verbal-visual LSR, with 43,7% of the occurrences in the analyzed corpus, as presented in Table 21 below.

Table 21:
Verbal-Visual LSR general results.

Verbal_Visual_LSR	Participants_ST	Processes_ST	Circumstances_ST	Total
Expansion_Extension	47,9%	70,4%	0,0%	43,7%
Expansion_Elaboration	52,1%	29,6%	14,3%	36,2%
Expansion_Enhancement	0,0%	0,0%	85,7%	22,2%
Projection	0,0%	0,0%	0,0%	0,0%
Total	100,0%	100,0%	100,0%	100,0%

It is important to point out that verbal LSRs of Projection were found only in clause complexes in verbal text. This means that there were no elements, i.e. thought and/or speak bubbles that represented LSRs of Projection in the visual texts analyzed. This suggests that the addition of new information (through verbal or visual text) is the main function in the division of labour between verbal and visual channels.

When we compare the results in Table 21 with those regarding LSRs between clauses (Table 9), we observe that Extension is the only LSR that is established both between clauses in clauses complexes and between verbal-visual texts.

When frequency percentages are examined for each TRANSITIVITY function, Elaboration is the most frequent LSR (52,1%) in connection with Participants, while Extension and Enhancement are more frequent in relation to Processes (70,4%) and Circumstances (85,7%), respectively. This suggests that there is a correlation between TRANSITIVITY functions (Participant, Process, and Circumstance) and the three LSRs of Expansion (Elaboration, Extension, and Enhancing, respectively). In other words, each TRANSITIVITY function tends to be expanded, in verbal or visual text, through a specific and different type of LSR of Expansion.

When we consider verbal-visual LSRs according to how each channel establishes these relations, we notice that the most frequent LSR is Extension (29,0%), particularly of verbal text extending the visual one, as shown in Table 22.

Table 22:
Verbal-visual LSR by channel.

Verbal_Visual_LSR	Participants_ST	Processes_ST	Circumstances_ST	Total
Expansion_Extension_verbal_extends_visual	31,0%	47,3%	0,0%	29,0%
Expansion_Elaboration_verbal_elaborates_visual	34,0%	23,0%	9,8%	23,0%
Expansion_Enhancement_visual_enhances_verbal	0,0%	0,0%	60,9%	15,8%
Expansion_Extention_visual_extends_verbal	16,5%	23,1%	0,0%	14,6%
Expansion_Elaboration_visual_elaborates_verbal	18,0%	8,6%	4,5%	11,1%
Expansion_Enhancement_verbal_enhances_visual	0,0%	0,0%	24,8%	6,4%
Total	100%	100%	100%	100%

This means that verbal text tends to add more information by introducing Participants, Processes, and/or Circumstances that were not present in the visual text. In addition, when we consider verbal-visual LSRs in relation to Processes, we notice Process is the function that operates in the addition of new information by the verbal text, which is something that takes place 47,3% of the time. Figure 15 illustrates this finding.

Figure 15:
Verbal text extending visual text through Process.

In Figure 15, we can notice that the visual text-heads of Prince Cinder’s three brothers--is related to two clauses and two Processes in the verbal text (“had” and “teasing”) and two Participants (“he” and “three brothers”). In the visual text there is no meaning construed in terms of the Participants’ teasing. It is only thorough the verbal text, through the Process realized by “teasing", that we get additional information about what Prince Cinders’ brothers do, which is not by any means depicted in the visual text.

As regards Participants, verbal text also plays an important role in LSRs, mainly through Elaboration (34,0%). In other words, the verbal text elaborates the visual text by presenting Participants, who were also depicted in the visual text.

If we turn to verbal-visual LSRs in relation to Circumstances, however, the most frequent LSR is Enhancement, with the visual text enhancing the verbal one 60,9% of the time. This result suggests that Circumstances (i.e. meanings related to time, place, reason, and others) tend to be realized mainly through the visual text in picturebooks.

Figure16:
Visual text enhancing verbal text through Circumstances

Figure 16 presents an example of visual text enhancing verbal text through Circumstances. It is the fact that the houses and trees grow smaller in the distance that shows the reader how far is the distance realized by the Circumstance of Extent (“down the lane as far as the river”) in the verbal text.

Finally, when we consider only the verbal-visual relation or, in other words, what role is played by the verbal and visual channels/texts in the construal of meaning in picturebooks, we can verify that the verbal channel/text plays a considerably larger role, not only in general (58,1%) but also in relation to Participants (65,5%) and Processes (68,3%), as shown in Table 23 below.

Table 23:
Verbal and visual role in the construal of meaning in picturebooks.

Verbal_Visual_relation	Participants_ST	Processes_ST	Circumstances_ST	Total
verbal ^ visual	65,5%	68,3%	34,6%	58,5%
visual ^ verbal	34,5%	31,7%	65,4%	41,5%
Total	100,0%	100,0%	100,0%	100,0%

These numbers suggest that language tends to be more constitutive in the picturebooks in our corpus. In other words, the visual text does not construe meaning on its own but requires verbal text to do so.

4.2.2 TTs results

The following three tables show the main results regarding the TTs. In general, Extension is the main verbal-visual LSR, with 41,2% of the occurrences in the TTs, as shown in Table 24 below. This suggests that as in the STs the addition of new information (through verbal or visual text) is the main function in the division of labour between verbal and visual channels in the TTs.

Table 24:
TTs’ verbal-visual LSR general results.

Verbal_Visual_LSR	Participants_TT	Processes_TT	Circumstances_TT	Total
Expansion_Extension	43,8%	68,4%	0,0%	41,2%
Expansion_Elaboration	56,2%	31,6%	13,5%	36,0%
Expansion_Enhancement	0,0%	0,0%	86,5%	22,8%
Total	100,0%	100,0%	100,0%	100,0%

When we consider verbal-visual LSRs regarding Participants, Processes and Circumstances, Elaboration appears as the most frequent LSR (56,2%) with regards to Participants, while Extension and Enhancement are more frequent in relation to Processes (68,4%) and Circumstances (86,5%), respectively.

Although these results might suggest that there are no significant differences in the verbal-visual LSRs between English source texts and Brazilian Portuguese translations, it is interesting to note that, as regards Participants in the TTs, there was a slight increase in the frequency of occurrence for the category Elaboration (from 53,1% in the STs to 56,2% in the TTs). This increase co-occurs with a slight decrease in the frequency of occurrence for Extension (from 47,9% in the STs to 43,8% in the TTs). There was also a slight increase in the occurrence of Elaboration regarding Processes: from 29,6% in the STs to 31,6% in the TTs. This increase co-occurs with a slight decrease in the frequency of occurrence for Extension in relation to Processes: from 70,4% in the STs to 68,4% in the TTs.

While these results might be explained by the fact that the TTs have more verbal text (more clauses) than the ST, they could also be pointing to where possible differences can be found in the TTs, which merits further investigation.

When we turn to verbal-visual LSRs according to how they are established and translated into the TT, we find that Extension is also the most frequent LSR (27,6%), with verbal text extending visual text, mostly through Processes added as new information (50,0% of the time), as presented in the Table 25 below:

Table 25:
TT’s verbal-visual LSR by channel.

Verbal_Visual_LSR	Participants_TT	Processes_TT	Circumstances_TT	Total
Expansion_Extension_verbal_extends_visual	25,3%	50,0%	0,0%	27,6%
Expansion_Elaboration_verbal_elaborates_visual	32,6%	21,3%	9,5%	22,4%
Expansion_Enhancement_visual_enhances_verbal	0,0%	0,0%	58,7%	15,5%
Expansion_Extention_visual_elaborates_verbal	23,6%	10,3%	4,0%	13,6%
Expansion_Elaboration_visual_extends_verbal	18,5%	18,4%	0,0%	13,6%
Expansion_Enhancement_verbal_enhances_visual	0,0%	0,0%	27,8%	7,3%
Total	100%	100%	100%	100%

When it comes to Participants, verbal text elaborated visual text 32,6% of the time, while visual text enhanced the verbal one in 58,7% of the occurrences regarding Circumstances.

We can also see that the slight increase in the frequency of occurrence for Elaboration was due to more cases of visual text elaborating verbal text. Similarly, the slight increase in the occurrence of Elaboration regarding Processes is also related to more cases of visual text elaborating verbal text. As argued before, these increases are worth further investigation, given that they can point to where differences can be found in the translated text: they, however, exceed the scope of the present paper.

Finally, when we consider the role played by verbal and visual texts in the construal of meaning in the TTs, we find that the verbal channel/texts also plays a considerably larger role in general (57,3%) and in relation to Participants (57,9%) and Processes (71,3%), as shown in Table 26. These numbers reiterate language’s more constitutive role in the analyzed picturebooks, and indicate no significant changes in the verbal-visual division of labour in the TTs.

Table 26: TTs’
verbal and visual role in the construal of meaning in picturebooks.

Verbal_Visual_relation	Participants_TT	Processes_TT	Circumstances_TT	Total
verbal ^ visual	57,9%	71,3%	37,3%	57,3%
visual ^ verbal	42,1%	28,7%	62,7%	42,7%
Total	100,0%	100,0%	100,0%	100,0%

5. Conclusion

This article aimed to present and explore a methodology to analyze verbal and verbal-visual LSRs in picturebooks originally written in English and their translations into Brazilian Portuguese. The main findings of our analysis can be thus summarized.

As regards LSRs between clauses, Projection_locution was the most frequent one, followed by Extension, in the STs and in the TTs as well. It is interesting to note, however, that while Elaboration was the most frequent verbal-visual LSR regarding Participants, it did not occur between clause complexes in the analysed corpus. This result might suggest that Elaboration is one of the LSRs that tends to be established only through verbal-visual LSRs in picturebooks.

Regarding verbal-visual LSRs, both in the STs and in the TTs, no simplexes of verbal text or visual text were found. This result confirms that the division of labor between visual and verbal texts: 1) is an important feature in picturebooks; 2) can be mapped through LSRs, more specifically those of Expansion. Extension is the most frequent LSR, not only in general but also in relation to Processes. Elaboration is the most frequent one with regard to Participants, while Enhancement occurs more frequently when we consider Circumstances. In addition, we find that verbal text tends to elaborate visual text through Participants and to extend it through Processes, while visual text tends to enhance verbal text through Circumstances. Finally, when we consider the division of labor between verbal and visual channels/texts, we find that verbal text plays a considerably larger role than visual text in the construal of meaning in the analyzed corpus. This suggests that language has a more constitutive role in picturebooks.

A further contribution of our study to research on multimodality in both text analysis and translation studies is its methodological proposal to categorize LSRs in verbal text and LSRs in visual and verbal texts, which proved to be fully operational for annotation purposes and text analysis.

References

• Ahlberg, J., & Ahlberg, A. (1991). The Jolly Christmas Postman New York: LB kids.
• Cole, B.Prince Cinders London: Puffin Books, 1997.
• Cole, B.Príncipe Cinderelo Tradução de Monica Stahel. São Paulo: Martins Fontes Editora Ltda., 2006.
• Halliday, M. A., & Matthiessen, C. (2004). An introduction to functional grammar (3 ed.). London: Edward Arnold.
• Halliday, M. A., & Matthiessen, C. (2014). Halliday's Introduction to Functional Grammar (4th. ed.). Oxford: Routledge.
• Knowles, M., & Malmkjaer, K. (1998). Language and control in children's literature London and New York: Routledge.
• Kress, G., & Van Leeuwen, T. (2006). Reading images: the grammar of visual design (2ª ed.). London & New York, Great Britain: Routlegde.
• McBrantney, S.; Jeram, A.Guess How Much I Love You London: Candlewick Press, 1996.
• McBrantney, S.; Jeram, A.Adivinha quanto eu te amo Tradução de Fernando Nuno. São Paulo: Martins Fontes Editora Ltda., 2011.
• Puurtinen, T. (1998). Syntax, Readability and ideology in children's literature. Meta: journal des traducteurs / Meta: Translators' Journal, 4 (XLIII), 1-10.
• Shavit, Z. (1981). Translation of children's literature as a function of its position in the literary polysystem.Poetics Today, 2(4), 171-179.
• Toury, G. (1995). Descriptive Translation Studies and beyond Amsterdam and Philadelphia: John Benjamins.
• Unsworth, L. (2006). Towards a metalanguage for multiliteracies education: Describing the meaning-making resources of language-image interaction. English Teaching: Practice and Critique, 5, 55-76.
• Van Meerbergen, S. (2009). Dutch picture books in Swedish Translation: Towards a model for multimodal analysis. Translation and (Trans)formation of Identities. Selected papers of the CETRA Research Seminar in Translation Studies, pp. 1-20.

• 1
  Unless otherwise stated, all examples of verbal text were retrieved from our corpus of picturebooks.

• 2
  All Figures belong to the Authors’ personal archive.

Publication Dates

• Publication in this collection
  Jan 2018

History

• Received
  07 Apr 2017
• Accepted
  14 Aug 2017