2. Related works

Araújo et al. (Reference Araujo, Ferreira, Silva, Lemos, Neto, Omaia, de Souza Filho and Tavares2013) describe an architecture for machine translation of Brazilian Portuguese contents to Libras in the real-time open domain, such as Digital TV. The system called LibrasTV receives a closed-caption stream as input and generates a Libras window on the TV screen as output. The machine translation is performed in real time and uses some strategies to reduce translation time. These include a Libras dictionary with pre-rendered signs and an efficient mechanism for machine translation of Brazilian Portuguese contents to Libras gloss. The translation mechanism combines morphosyntactic transfer rules, defined by specialists; a statistical method of data compression to classify input tokens (words) in Brazilian Portuguese; and a textual simplification that reduces entry complexity before applying these translation rules. Araújo et al. (Reference Araujo, Ferreira, Silva, Oliveira, Falcóo, Domingues, Martins, Portela, Nobrega, de Souza Filho, Tavares and Duarte2014) modify the architecture presented in Araújo et al. (Reference Araujo, Ferreira, Silva, Lemos, Neto, Omaia, de Souza Filho and Tavares2013), expanding the usage scenarios to the Web and Digital Cinema. Human computing strategies are also incorporated to facilitate the creation of Libras signs and the definition of syntactic rules.

The work of Boulares and Jemni (Reference Boulares and Jemni2013) presents a machine translator from English to ASL (American Sign Language) through mobile devices, to make information accessible for Deaf people anywhere. The system is proposed for an open translation domain and uses a rule-based translation strategy. The system receives an SMS (Short Message Service) as input and generates an MMS (Multimedia Message Service) containing an ASL video represented by an avatar as output. The authors report that one of the difficulties of this system is that SMS is not always received quickly, which can lead to a delay in the translation.

Huenerfauth (Reference Huenerfauth2004) presents a proposal of machine translation of English texts to an ASL animation and test the boundaries of traditional architectural designs for machine translation. It involves a multi-pathway translation architecture that incorporates direct, transfer-based, and interlanguage-based translation approaches into a single system. He also models semantic-pragmatic elements called “classifier predicates.”

The Huenerfauth (Reference Huenerfauth2005a) project uses representations of speech, semantics, syntax, and (sign) phonology adapted for ASL generation. In particular, this system intends to translate contents, use classifiers, and mark the signing space. In Huenerfauth (Reference Huenerfauth2008a), he presents a hybrid rule-based machine translation approach combining transfer-based, interlanguage-based, and direct translation. As a mark of the Huenerfauth works, there is always the concern of the modeling classifiers in the sign language translation system.

However, one issue reported by the author (Huenerfauth Reference Huenerfauth2008a) is the difficulty of designing an interlanguage-based solution for an open (or general) domain. Typically, they are only feasible for specific (or limited) areas. Besides, there are English sentences where in-depth semantic analysis is not required. In this case, we could use a transfer-based translation approach.

The formal model named Partition/Constitute (P/C) is a method for representing a linguistic sign containing several parameters (Huenerfauth Reference Huenerfauth2005b). This formalism is compared to representations used in research related to gesture animation and was used in the development of an English-to-ASL machine translation system.

Morrissey and Way (Reference Morrissey and Way2014) use a statistical method for translating English into Irish Sign Language (ISL), German Sign Language (DGS), and a gloss annotation in DGS.Footnote ⁱ To perform this task, the authors use four aligned linguistic corpora, which make possible the translation of the 12 pairs of languages. This study also allows the conversion of both: spoken language-to-SL and SL-to-spoken language.

López-Ludeña et al. (Reference Lopez-Ludena, Gonzalez-Morcillo, Lopez, Ferreiro, Ferreiros and San-Segundo2014a) propose the generation of Spanish Sign Language (LSE) gloss with a rule-based translation strategy. The system architecture consists of an analyzer, a transfer module, and a generator. The analyzer is responsible for identifying the morphological dependencies in the spoken language and the SL, generating a dependency tree that represents the functional relations between the words. The transfer module maps the dependencies in the spoken language and the SL and performs the lexical (lexical-morphological and semantic) and structural replacements. Finally, the generator is responsible for the ordering of words based on the LSE dependency tree, generating an LSE gloss. It is important to emphasize that the López-Ludeña et al. (Reference Lopez-Ludena, Gonzalez-Morcillo, Lopez, Ferreiro, Ferreiros and San-Segundo2014a) solution is for a restricted area, specifically for attending some public services (e.g., withdrawal of driver’s license).

The hybrid translation strategy (rules-based and statistical) used by the López-Ludeña et al. (Reference Lopez-Ludena, Gonzalez-Morcillo, Lopez, Ferreiro, Ferreiros and San-Segundo2014b) had higher accuracy than rules-based and statistical approaches. However, the authors make a caveat that the dependence of a bilingual corpus may represent a bottleneck for its adoption, since LSE is a modern language and not yet fully consolidated. Another observation verified from the validation of their system is that it has higher accuracy in the translation of short sentences, that is, with few signs.

Rayner et al. (Reference Rayner, Bouillon, Gerlach, Strasly, Tsourakis and Ebling2016) present an open platform for rule-based speech-to-sign translation. The platform Lite Speech2Sign aims to combine components for speech recognition, translation, and sign language animation. It is a similar approach used in VLibras. This kind of modularization of the sequence of processes involved (speech-to-text, text-to-gloss, and gloss-to-sign) allows us to incorporate advances in specific technologies used at each stage of the process. The fundamental differences between the platforms are translation approaches and the sign language animation strategy. Lite Speech2Sign adopts transfer-based machine translation and synthesis from form notation, whereas VLibras uses rule-based with syntactic-semantic adequacy and handcrafted animation. While the first strategy allows greater ease in the prototyping new simple translation grammars, the second one can provide a better result.

In the opposite direction, we can cite several initiatives translating continuous sign language to text, such as the SignSpeak Project (Reference Dreuw, Forster, Gweth, Stein, Ney, Martinez, Llahi, Crasborn, Ormel, Du, Hoyoux, Piater, Moya and Wheatley2010) and, more recently, the work of Kim and O’Neill-Brown (Reference Kim and O’Neill-Brown2019). The overall goal of SignSpeak is to develop new vision-based technology for translating continuous sign language to text to improve their Deaf communication community with the hearing people and the other way.

Kim and O’Neill-Brown research aim the challenges of building an automated sign language recognizer (ASLR), due to the lack of sufficient quantities of annotated ASL-English parallel corpora for training, testing, and developing. In particular, the authors explore a range of different techniques for automatically generating synthetic data from existing datasets to improve the accuracy of ASLR. In common with our approach, both projects consider a textual representation of the recognized signs as an intermediate step before message translation and rendering.

As identified by Kim and O’Neill-Brown (Reference Kim and O’Neill-Brown2019) and Morrissey and Way (Reference Morrissey and Way2014), considering the difficulty of having annotated sign language-spoken language parallel corpora, especially for general (or open) domain, in this paper, we chose to develop a rule-based translation approach from BP to Libras, instead of a statistical or neural approach. Besides, unlike the works of Huenerfath (Reference Huenerfauth2005a, Reference Huenerfauth2005b, Reference Huenerfauth2008a) and López-Ludeña et al. (Reference Lopez-Ludena, Gonzalez-Morcillo, Lopez, Ferreiro, Ferreiros and San-Segundo2014a, Reference Lopez-Ludena, Gonzalez-Morcillo, Lopez, Ferreiro, Ferreiros and San-Segundo2014b), which have solutions addressed to limited (or specific) domains, we designed our approach addressing general domain texts from BP into Libras, treating some syntactic-semantic adequacy.

Additionally, we also model a formal language to describe the translation rules from Brazilian Portuguese to Libras, and a simple string-based notation for Libras gloss. This notation helps simplify the sign language animation implementation, which has to display a sequence of signs (simple or compound) from the gloss. This is an alternative path with respect to notations like the one proposed by Huenerfauth, which requires to render a 3D animation coordinating multiple channels (e.g., right and left hand, head tilt, eye gaze, among others) and makes its implementation more complex. We discuss this issue in more detail in Section 5.2.

Finally, the approach by Araújo et al. (Reference Araujo, Ferreira, Silva, Lemos, Neto, Omaia, de Souza Filho and Tavares2013, Reference Araujo, Ferreira, Silva, Oliveira, Falcóo, Domingues, Martins, Portela, Nobrega, de Souza Filho, Tavares and Duarte2014) presents some limitations, not addressing some essential aspects of a machine translation solution to Libras, such as the treatment of verbal tense, semantic appropriateness of the use of adverbs of intensity and negation, treatment of connecting verbs, among others. As the main contribution of this work, our proposal for a new machine translation from BP into Libras mechanism seeks to address these aspects. The comparison between our proposal and the proposal by Araújo et al. will be further detailed in Section 4.

3. Linguistic aspects of libras

Libras is a natural human language. It has grammatical rules common to the Deaf people in Brazil, linguistic variation according to their specific community (dialects), and its own phonological, morphological, syntactic, and semantic-pragmatic system. Besides, Libras is a cultural expression of the Brazilian Deaf community.

Concerning the phonological aspects, the minimum units of their linguistic system are the quiremas.Footnote ^j The quirema is a formational unit of the sign, that is, it is composed of the articulation between the parameters of the sign language. The sign consists of five basic parameters (Figure 1), that are handshape (HS) (Subfigure 1D), location (L) (Subfigure 1C), hand movement (M) (Subfigure 1E), orientation (Or) (Subfigure 1A), and non-manual aspects of signs (NM) (Subfigure 1B) (Quadros and Karnopp Reference Quadros and Karnopp2004).

Figure 1. Parameters of the sign language (Choi et al. Reference Choi, Vieira, de Oliveira and Nakasato2011).

In terms of sign language morphology, the complexity of morphemes is a result of non-concatenated processes, that is, the root is enriched with movements and contours in the signing space (Quadros and Karnopp Reference Quadros and Karnopp2004).

The syntactic aspects of Libras consider the sign execution space because the syntactic relations make use of the pronominal and nominal system for this purpose. According to Felipe (Reference Felipe2007), since SLs are gestural-visual, they use the three-dimensionality of the space as a grammatical element. Non-manual aspects are also relevant in Libras syntax, since utterances with verbs with an agreement in their formation must have this mark (Quadros and Karnopp Reference Quadros and Karnopp2004). However, it is not only in this situation that the non-manual aspects must appear. We also use them in marking the location of referents.

The semantic and pragmatic aspects are related to the linguistic meaning of the utterance. In Libras, the syntactic elements of the “verb” influenced them. According to Silva (Reference Silva2006), the verb is also quite influential in the semantic relations of the language. It happens due to the importance of the meaning of the verb to understand its behavior and to predict its syntactic properties. Thus, we classify the verbs considering their morphological and semantic characteristics in Libras.

4. The proposed approach

As mentioned in Section 2, the VLibras platform proposed by Araújo et al. (Reference Araujo, Ferreira, Silva, Oliveira, Falcóo, Domingues, Martins, Portela, Nobrega, de Souza Filho, Tavares and Duarte2014) has some limitations in its machine translation component, especially regarding the handling of syntactic and semantic issues. Araújo et al. (Reference Araujo, Ferreira, Silva, Lemos, Neto, Omaia, de Souza Filho and Tavares2013) translation strategy designed their solution to translate general domain content efficiently (quickly). To perform this task, it combines statistical compression methods to classify input tokens, textual simplification strategies to reduce the complexity of input text, and a set of morphological and syntactic rules defined by Libras specialists. Figure 2 shows the architecture of the machine translation component described by Araújo et al. (Reference Araujo, Ferreira, Silva, Lemos, Neto, Omaia, de Souza Filho and Tavares2013, Reference Araujo, Ferreira, Silva, Oliveira, Falcóo, Domingues, Martins, Portela, Nobrega, de Souza Filho, Tavares and Duarte2014).

Figure 2. Architecture of the machine translation component defined by Araújo et al. (Reference Araujo, Ferreira, Silva, Lemos, Neto, Omaia, de Souza Filho and Tavares2013, Reference Araujo, Ferreira, Silva, Oliveira, Falcóo, Domingues, Martins, Portela, Nobrega, de Souza Filho, Tavares and Duarte2014).

According to Figure 1, their machine translation component is given in five steps. In the first step, the tokenizer splits the text in BP into a sequence of words (or tokens). Afterward, it classifies the tokens into morphological categories (Morphological-Syntactic Classifier step), using the PPM-C [29], a statistical data compression method based on N-order Markov models. Then, they simplifies the text by removing prepositions and articles (Remove Tokens step). Afterward, they replaces some words (Lexical Replacement step) to adapt the meaning of the sentence rewritten to Libras. For example, the words casa (house), lar (home), habitação (habitation), and residência (residence) in BP have the same sign in Libras: CASA (HOME). Non-inflected gloss verbs replace the BP verbs. Finally, they perform a dactylology replacement for fingerspelling the proper names and technical terms. The output generated is a Libras gloss.

Thus, the solution of Araújo et al. (Reference Araujo, Ferreira, Silva, Lemos, Neto, Omaia, de Souza Filho and Tavares2013, Reference Araujo, Ferreira, Silva, Oliveira, Falcóo, Domingues, Martins, Portela, Nobrega, de Souza Filho, Tavares and Duarte2014). presents some limitations, not addressing some essential aspects of a machine translation solution to Libras, such as the treatment of verbal tense, semantic appropriateness of the use of adverbs of intensity and negation, treatment of copula verbs, among others.

To address these limitations, in this work, we propose the development of a new mechanism to perform machine translation to Libras addressing these aspects. We also model a formal language to describe the translation rules from Brazilian Portuguese to Libras, making the translation process flexible, configurable, and able to deal with other linguistic aspects. Othero (Reference Othero2009) corroborates the importance of this type of language when he states that “the study of language syntax may be essential for its computational treatment at several other levels of linguistic description, and usually, the computational implementation work of a language involves the syntactic or morphosyntactic treatment of that language.”

Table 1 makes a summary comparison between the aspects of translation addressed in our proposal and the solution proposed by Araújo et al. (Reference Araujo, Ferreira, Silva, Lemos, Neto, Omaia, de Souza Filho and Tavares2013, Reference Araujo, Ferreira, Silva, Oliveira, Falcóo, Domingues, Martins, Portela, Nobrega, de Souza Filho, Tavares and Duarte2014).

Table 1. Comparison between the aspects of translation addressed in our proposal and the solution proposed by Araújo et al. (Reference Araujo, Ferreira, Silva, Lemos, Neto, Omaia, de Souza Filho and Tavares2013, Reference Araujo, Ferreira, Silva, Oliveira, Falcóo, Domingues, Martins, Portela, Nobrega, de Souza Filho, Tavares and Duarte2014)

Figure 3 illustrates the proposed machine translation mechanism. According to Figure 3, it works as follows. Initially, we apply a process of morphological and syntactic classification in the input text in BP. This process may produce two types of output: (1) a syntax tree or (2) a null tree. It produces a syntax tree for sentences recognized by the classifier grammar and generates a null tree when the sentences is unrecognized by the classifier grammar. Then, the Morphological Adequacy and Syntactic Adequacy modules apply a set of translation rules defined by specialists based on the proposed rule description language (see Section 4.1). When the classification module returns a null tree, the Adequacy module uses the Morphological classification (e.g., a POS-tagging process). In the latter case, the component applies only translation rules defined at the morphological level. As a result, the translation component generates a Libras gloss.

Figure 3. Architecture of the proposed machine translation mechanism.

Afterward, we apply post-processing in this gloss. This process is responsible for performing some translation refinements, such as the treatment of numbers, plural, among others. The post-processing process is important because it reduces the possibility of fingerspellingFootnote ^k during the presentation of visual gestures by the Libras generation service. More details about the implementation of a prototype of this machine translation mechanism will be presented in Section 5.2.

An essential contribution of our work is that we use a simpler Libras gloss representation to treat the primary sign language specificities described in Section 3. More specifically, we simplify Libras gloss as a sequence of tokens. Besides, we also include “special tokens” to incorporate expressions and the specific aspects of sign languages, such as directional verb agreement, incorporation of facial expressions during the signing, and intensification of signs. For example, in Libras, the adverb NÃO (NO) can modify the signing of the verb TER (HAVE) in a sentence. In this case, our translation rules address this issue, generating a special token NÃO_TER (NOT_HAVE), which will have a different signing than the NÃO (NO) and TER (HAVE) signs separately. Thus, we can represent this modification in the TER sign generated by the adverb NÃO, using a special sign NÃO_TER.

We can also address the treatment of the directional verb agreement using this representation. For example, the sentence “EU PERGUNTO PARA VOCÊ” (I ASK YOU) is translated to a special Libras gloss “1S_PERGUNTAR_3S” (1S_ASK_3S), which represents the directional verb PERGUNTAR (ASK) with the directionality of the first person singular (represented by 1s) to the third person singular (represented by 3s).

This simpler representation of Libras gloss has the advantage of making the translation process from spoken languages into sign languages similar to a translation process between two spoken languages. It converts to a sequence of tokens in a source language to another sequence of tokens in the target language. As it uses a more straightforward and closer representation of the current bilingual corpus for machine translation, it could also simplify the creation of bilingual text-to-gloss translation corpus. Consequently, it can also possibly reuse some machine translation techniques currently available for use in this type of problem, when a bilingual text-to-gloss corpus is available.

Another advantage is that it reduces the sign language animation component’s complexity, which converts Libras gloss into a Libras representation (e.g., video or animation). When we use a multiple channel output representation (e.g., right hand, left hand, head tilt, eye gaze), as proposed by Huenerfauth (Reference Huenerfauth2005b), we would have to render a 3D animation coordinating multiple channels, which makes this component more complex.

Finally, it is also important to point out that our solution also supports syntactic-semantic adequacy. According to Felipe (Reference Felipe2006), the incorporation, simultaneous activity, and directional verb agreement grammatical aspects, treated by our solution, are syntactic-semantic aspects of the sign languages, since they change the form (syntactic) and the meaning (semantics) of the sentence. Additionally, as our solution has a rule description language, we can incorporate translation rules to address other syntactic-semantic aspects.

In Section 4.1, we will present the rule description language developed for describing the translation rules used by our Adequacy module.

4.1. Rule description language

In the rule description language developed in this work, we define a sentence S formally as:

\begin{equation*}S = <sg_1,sg_2,sg_3,\ldots, sg_n>,\end{equation*}

where $sg_1, \ldots, sg_n$ are noun, verbal, adverbial, adjectival, numeral, quantitative, prepositional, determinant, or possessive phrases. Each phrase $sg_i$ is composed of subphrases (non-terminals) and/or words (terminals) and is formally defined as:

\begin{equation*}sg_i= <title, specific, action, newpos, newprop, newtoken, newtokenpos>,\end{equation*}

where title represents the morphological and/or syntactic class of the phrase; specific is an optional field that indicates the existence of restrictions in the rule application, that is, the rule is only applied to a specific subphrases and/or word; action indicates the type of operation to be performed by the rule; newpos represents the new position of the subphrase and/or word after the application of the rule. The value “-1” is used when the subphrase and/or word should be removed of the sentence; newprop is an optional field that indicates changes in the subphrase and/or word (e.g, incorporate the adverb of negation into adjectives, verbs, and adverbs); newtoken is also an optional field that indicates the addition of a new subphrase and/or word to the sentence (e.g, when a sentence does not present the adverb of time in Libras, the signs FUTURE and PAST must be added to indicate a reference to the future or the past tense, respectively); newtokenpos is also an optional field that represents the position of the new subphrase and/or word added in the sentence.

From the formalization described above, we define an XML representation to represent the attributes of translation rules. Each rule has a <counter> attribute that represents the number subphrases and/or words involved in that rule. It has also a <active> flag that indicates whether the solution should apply the rule or not, which facilitates the tests of the translation rules and the Libras specialists’ validation.

An example of the XML representation of a translation rule is shown in Figure 4. It indicates that a Brazilian Portuguese sentence with “Verbal Phrase (VP) + Noun Phrase (NP)” that is in reverse order must be translated to direct order (“Noun Phrase (NP) + Verbal Phrase (VP)”) in Libras.

Figure 4. Example of the translation rule “S (VP NP) $ \to $ S (NP VP)” described with the proposed rule description language.

Another example of a translation rule is shown in Figure 5. It indicates that in a BP sentence in the future tense, the verb must be modified to the infinitive form, followed by the sign FUTURE in Libras.

Figure 5. Example of the translation rule with sentence processing in future tense: (VP (VB-R ^*) $\to$ VP (VB ^* <FUTURE>), where ^* represents any phrase and/or word that is after the verb.

5. Proof of concept

As a proof of concept of the proposed solution, initially, we modeled a preliminary grammar using the rule description language described in Section 4.1. This grammar consists of a set of formal machine translation rules from Brazilian Portuguese (BP) to Libras, defined by Libras specialists. Afterward, we implemented a prototype of the proposed translation mechanism using this modeled grammar and integrated it with the VLibras Suite (Araujo et al. Reference Araujo, Ferreira, Silva, Oliveira, Falcóo, Domingues, Martins, Portela, Nobrega, de Souza Filho, Tavares and Duarte2014). Sections 5.1 and 5.2 present more details about the modeled grammar and the prototype of the proposed translation mechanism, respectively. Section 5.3 describes the integration of this prototype into VLibras.

5.1. Definition of a grammar from BP to libras

Table 2 presents the grammar we modeled to translate content from Brazilian Portuguese to Libras. As mentioned in Section 4, we use a Libras gloss representation, which includes “special tokens” to address the specific aspects of sign languages, such as directional verb agreement, incorporation of facial expressions during the signing, intensification of signs, among others.

Table 2. Grammar proposed using the rule description language

\[{\text{a}}\] P, R, D (present, future and past).

According to Table 2, the rule (2) intends to eliminate the Brazilian Portuguese connecting verb. In BP, the connecting verbs are not significant and do not indicate action. These verbs appear in the verbal phrase S(VP) that have a verb (VB-^*) followed by an adjectival phrase (AP). In this case, the corresponding VP in Libras will be formed only by the AP (See Table 2, rule 2).

Another rule example is the elimination of prepositions. In Libras, some prepositions are not signing and consequently must be eliminated in the translation process. We identify this rule as (3) in Table 2, which indicates that: when we have a prepositional phrase (PP) composed by a specific preposition (P$_x$) and a definite phrase (DP) or an adverbial phrase (AdvP), the (P$_x$) is excluded.

When we have a common noun of two genders, which represents nouns having the same writing in both males and females, we can apply the rule (4). When dealing with a common noun of two genders, the sign HOMEM (MAN) or MULHER (WOMAN) is added in the Libras interpretation to sign the gender. Then, the rule (4) indicates that whether a masculine (DA$_{(m)}$) or definite feminine article (DA$_{(f)\!}$) is followed by a common noun of two genders (SN$_c$) in BP, we remove the definite article and add a HOMEM or MULHER sign to represent the gender, respectively.

5.2. Implementation of the proposed translator mechanism

To validate and develop a proof of concept of the proposed approach, we implement the proposed machine translation mechanism and integrate it into VLibras (Araujo et al. Reference Araujo, Ferreira, Silva, Oliveira, Falcóo, Domingues, Martins, Portela, Nobrega, de Souza Filho, Tavares and Duarte2014). We chose VLibras because it is an open-source BP-to-Libras translator, designed for three usage scenarios (Digital TV, Web, and Digital Cinema), and performs translation for an open and general domain.

We developed the proposed translation component in the Python programming language, according to the architecture presented in Figure 2. The Aelius toolFootnote ^l (Alencar Reference Alencar2010), an open-source POS-tagger library for Brazilian Portuguese, performs the morphosyntactic classification of the input sentences. The Morphological Classification step generates a list of tuples, where each tuple is composed of the word and its respective morphological classification. From this list, it performs a Syntactic Classification. As mentioned in Section 4, this process may produce a syntax tree (if the classifier recognizes the sentence) or a null tree (if the classifier has unrecognized the sentence). The null tree can happen in two situations: the sentence structure is not adequate in BP or due to a limitation of the grammar used by the parser.

Afterward, the Morphological and Syntactic Adequacy modules apply some translation rules defined by Libras specialists based on the grammar and language presented in Sections 5.1 and 4.1, respectively. In the scope of this proof of concept, we define some rules for the treatment of verbal tense, treatment of common nouns of two genders, elimination of articles, semantic appropriateness of the use of adverbs of intensity and negation, treatment of connecting verbs, among others. We present some examples of the application of these rules in Table 3.

Table 3. Some examples of the use of translation rules in the proposed translation component. The first column shows the input sentence in BP and the output generated in Libras gloss. The second column represents the translation rules applied to generate that output

Finally, we apply the post-processing module to the treatment of numerals, plural terms, among others. More specifically, it performs the following steps:

• Synonym treatment: performs simple substitution of synonyms, using a list of synonyms in BP whose signing is the same in Libras. Ex: RESIDÊNCIA $\rightarrow$ CASA (RESIDENCE $\rightarrow$ HOME).

• Treatment of cardinal numbers: performs substitution of the number writing for its algebraic representation. Ex: DOZE $\rightarrow$ 12 (TWELVE $\rightarrow$ 12).

• Treatment of ordinal numbers: performs substitution of the algebraic writing (and the extended writing of the numbers from 10º) to the long writing and referencing only the sequence of numbers from 0 to 9. Ex: 12º or DÉCIMO SEGUNDO $\rightarrow$ PRIMEIRO SEGUNDO (12º or TWELFTH $\rightarrow$ FIRST SECOND).

• Treatment of gender: performs substitution of pronouns, adjectives, and adverbs for their corresponding masculine version, since the signing in Libras is similar. Ex: BONITA $\rightarrow$ BONITO (BEAUTIFUL $\rightarrow$ BEAUTIFUL).

• Treatment of plural terms: performs substitution of pronouns, adjectives, and adverbs in the plural for their corresponding singular representation, since Libras signing does not modify them in plural form. Ex: MEUS $\rightarrow$ MEU (MY $\rightarrow$ MY).

Figures 6 and 7 illustrate how the proposed machine translation module works for the input sentence “Ele comprou o carro” (He bought the car). Figure 6 shows the syntax tree generated by the morphosyntactic classification process. In this case, after this step, the Adequacy module is invoked and applies rules 5 and 7 (see Table 2), followed by the post-processing module. Figure 7 shows Libras gloss generated as output as well as its labeling. We then pass this gloss to the Animation component, which will convert it to a Libras animation or video.

Figure 6. A syntax tree example for the input sentence “Ele comprou o carro” (He bought the car).

Figure 7. Output generated by the proposed Machine Translation Process.

The generated syntactic tree has the sentence (S) initially formed by a nominal phrase (NP - “ELE”) and a verbal phrase (VP - “COMPROU O CARRO”). Since NP is an end node, that is, it does not have any other phrase in its constitution, the morphological classification of the word “ELE” is PRO (PRONOME). VB is not only composed by a terminal node, and therefore its classification generates a terminal node (VBar - “COMPROU”) and another nominal phrase (NP - “O CARRO”). The NP produced from the VP is a terminal node with the letter “O” morphologically classified as a determinant (DET) and then as an article (D), while the word “CARRO” is classified as a name (NOM) and then as a noun (N). Finally, the terminal node VBar is classified morphologically as a verb (V) and later as VB_D (verb in the past), since this morphological class carries information about when the action happened.

It is important to point out the output of this component is just a sequence of signs. As mentioned in Section 4, one of the main reasons for this is to reduce the complexity of the Synthesis (or Animation) component, which converts Libras gloss into a Libras 3D animation, since when we use a multiple channel output representation, we would have to render a 3D animation coordinating multiple channels. Thus, the Animation component has to display a sequence of signs (simple or composite) from Libras gloss, preventing it from making specific adaptations (e.g., including non-manual expressions) in the presented signs.

5.3. Integration of the proposed translator into VLibras

The integration of the proposed translator component into VLibras is illustrated in Figure 8. We inserted the new translator component after the components that filter and extract the text in BP (i.e., the Speech Recognition and Subtitle Extractor components), and before the Synchronizer component. Thus, the translator component receives an input text in BP and generates a Libras gloss sent to the Synchronizer component.

Figure 8. Schematic view of the integration of the proposed translation component into VLibras.

More specifically, the system works as follows. Initially, a user submits a digital content (text, video, subtitle, or audio) to the VLibras service. A web server receives this content and sends it to a component that processes it and extracts a text in BP from it. For example, if the content is an audio or a video, it triggers a speech recognition component to generate an input text in BP. Otherwise, if the content is a subtitle, it triggers the Subtitle Extractor to extract the text from this subtitle. Then, it sends the text to the proposed translation mechanism, which converts it to Libras gloss. Then, this gloss is sent to the Synchronizer, which, together with the Animation component, is responsible for generating a Libras video or animation.

Afterward, if the input stream is a video, Libras video can be mixed in the original video, generating an accessible copy of this video, that is, the user video with a Libras window. Then, we return this accessible video to the user. Another option is only to generate the Libras video or animation as output, usually when the input is a text, an audio, or a subtitle.

It is also important to point out that we currently implemented this proposed machine translation mechanism and put it in production on VLibras.Footnote ^m Currently, VLibras is installed on thousands of Brazilian websites and is accessed by millions of users daily. It also helps to show the feasibility of the solution and its contribution in a real use scenario. VLibras’ current sign dictionary has about 15,000 signs and a significant portion of them consisting of “special glosses” that address the specific aspects of sign language mentioned in Sections 3 and 4.

In Section 6, we will present some results taken from the proposed approach to evaluate the translated contents.