2. Problem definition and motivation

Both NMT and SMT require a large-scale high-quality parallel corpus for training a good-quality machine translation (MT) system. Absence of such corpora makes NMT suffer from the adequacy problem (Koehn and Knowles Reference Koehn and Knowles2017). Creating a high-quality large-scale parallel corpus is expensive as it requires time, money, and professionals to translate a large amount of texts. As a result, many of the existing large-scale parallel corpora are limited to some specific languages and domains.

The quality of any MT system can be characterized by its adequacy and fluency. The long-dominant SMT has been found to be good at handling adequacy, but lacks in fluency. Recently, NMT has become the new state of the paradigm to MT. However, it has been reported that NMT sacrifices adequacy at the cost of fluency (Koehn and Knowles Reference Koehn and Knowles2017). The performance of an NMT system greatly depends on the amount of parallel corpus: more the data better is the performance. However, having sufficient corpus for training an NMT system is a challenge. The adequacy is a direct measure of how well an NMT system learns mapping between the symbols in source and target languages. However, NMT fails to capture these mappings when the parallel corpus is not enough. This is often the case that sufficient parallel corpus is not available for many language pairs as well as for some restricted domains. So, in order to help the NMT models learn the mapping between words in the absence of sufficient parallel corpus, we extract phrases from the original training data and add to it (i.e., the original training corpus) for better evidences. This, in turn, provides an implicit knowledge about the mappings between the source–target pairs.

In our current work, we propose an effective approach for the translation of a variety of texts and languages. Phrases extracted from SMT are fed as input to the training of NMT. Firstly, we build the NMT systems for the resource-scarce Indian languages, and secondly, we translate the old English texts to the modern. India is a multilingual country with great linguistic and cultural diversities. The resources and tools in the form of parallel corpus, morphological analyzers, parts-of-speech tagger, etc. are readily not available in the required measures. Translating old text to the modern text is very important for various purposes. Human languages are constantly evolving and changing over time to reflect sociocultural changes, fit current conventions, mores, expressions, and needs. This change in a language often requires “rewriting” the old texts for the modern readers in the same language. In line with global trends, old texts are increasingly available in the forms that computer can process. These ever expanding records (e.g., historical records, scanned books, academic papers, large-scale corpora, and maps)—either digitally born or reconstructed through digitization pipelines—are too big to be “rewritten” manually. We pose this rewriting of old text as an MT problem and use our proposed method to improve this rewriting.

3. Related work

Having enough parallel corpus is a big challenge in NMT, and it is very unlikely to have millions of parallel sentences for every language pair. A few attempts have been made to build NMT systems for the low-resource language pairs (Sennrich, Haddow, and Birch Reference Sennrich, Haddow and Birch2016a; Zhang and Zong Reference Zhang and Zong2016; Gulcehre et al. Reference Gulcehre, Firat, Xu, Cho and Bengio2017), which incorporated huge monolingual corpus in the source and target sides.

Sennrich et al. (Reference Sennrich, Haddow and Birch2016a) have incorporated monolingual data on the target side to investigate two methods of filling the source side of the monolingual data. In the first method, they have used a dummy source sentence for every target sentence, and in the second method, they used a synthetic source sentence obtained via back translation. They claimed that the second method is more effective. However, if there is not enough parallel data, quality of back translation is again a problem.

Zhang and Zong (Reference Zhang and Zong2016) explored the effect of incorporating large-scale source-side monolingual data in NMT in different ways. In the first approach, inspired by Sennrich et al. (Reference Sennrich, Haddow and Birch2016a), they first built a baseline model and then obtained parallel synthetic data by translating the monolingual data. These parallel data along with the original data are used for training an attention-based GRU system. The second method used the multitask learning framework to generate the target translation and reorder the source-side sentences at the same time. They claimed that usage of source-side monolingual data in NMT is more effective than that of SMT.

Gulcehre et al. (Reference Gulcehre, Firat, Xu, Cho and Bengio2017) have proposed two alternative methods to integrate monolingual data on the target side, namely shallow fusion and deep fusion. In shallow fusion, the top K hypotheses (produced by NMT) in each time step t are re-scored using the weighted sum of the scores given by the NMT (trained on parallel data) and a recurrent neural network-based language model (RNNLM). Whereas in deep fusion, hidden states obtained at each time step t of RNNLM and NMT are concatenated and the output is generated from that concatenated state.

In recent times, Arthur, Neubig, and Nakamura (Reference Arthur, Neubig and Nakamura2016) have proposed a model to incorporate translation lexicons through calculating lexical predictive probability and adding this probability to the input of the Softmax. Feng et al. (Reference Feng, Zhang, Zhang, Wang and Abel2017) proposed a method that extracted phrase translation dictionary from the corpus using word alignment, and the phrase translation probability is used in the NMT model to construct the local memory.

Zoph et al. (Reference Zoph, Yuret, May and Knight2016) applied transfer learning for low-resource NMT. They trained a model on high-resource language pair, and then the learned parameters were used for training a low-resource language pair. However, it requires selecting both the high- and low-resource language pairs to be of similar types (i.e., closer to each other). So this approach may not work if the language pairs are distant. There are two fundamental differences between our proposed approach and theirs. Unlike theirs, we do not use any large amount of additional parallel corpus, rather we use a relatively smaller corpus.

Wang et al. (Reference Wang, Pham, Dai and Neubig2018) proposed simple data augmentation technique through randomly replacing words in both the source sentence and the target sentence with other random words from their corresponding vocabularies. He et al. (Reference He, He, Wu and Wang2016) have incorporated SMT features such as translation model, language model (LM) under log-linear framework during the beam search of decoding step.

Wang et al. (Reference Wang, Lu, Tu, Li, Xiong and Zhang2017), Wang, Tu, and Zhang (Reference Wang, Tu and Zhang2018) have proposed NMT model advised by SMT, where at each decoding step, SMT offers additional recommendations and the recommendations are scored with a classifier for combining with the NMT model in an end-to-end manner. Zhao et al. (Reference Zhao, Wang, Zhang and Zong2018) also have used phrase table as recommendation through adding bonus to words worthy of recommendation, for helping NMT in predicting adequate words.

NMT always shows weakness in translating the rare words. Fadaee, Bisazza, and Monz (Reference Fadaee, Bisazza and Monz2017) have proposed an approach for handling rare words through data augmentation for English–German language pair. Their approach also made use of huge monolingual corpus for generating sentence pairs containing rare words, and these generated sentence pairs are used for training the NMT models. Although the said pair does not fall under the low-resource category, they created simulated low-resource settings to perform the experiments and claimed to have achieved substantial improvement on the BLEU score. However, in our experimental settings, we truly use the low-resource languages, for which having a large monolingual corpus is also a challenge.

Song et al. (Reference Song, Zhang, Yu, Luo, Wang and Zhang2019) have investigated a data augmentation method for constraining NMT with pre-specified translations. In this method, source sentences in the training data are code-switched by replacing source phrases with their target translations allowing the model to learn lexicon translations by copying source-side target words. However, in our approach, we do not code-switch the training data, instead we use phrase pairs as training data.

Most of the earlier works related to NMT in low-resource scenario tried to incorporate monolingual data either in the source or target side. The effect of adding monolingual data in NMT is similar to that of building LM on a large-scale monolingual data in SMT. It makes output more fluent; however, NMT always lacks in generating adequate output. Adding monolingual data do not contribute much in improving the adequacy. A number of attempts related to low-resource NMT also tried to expand the training data by adding the back-translated monolingual data. At first, a model is trained using the available training data, and then the monolingual corpus is passed for translation. Quality of the parallel data obtained from the translation of monolingual data depends on the size of the original parallel data. If the data size is very small, then the translated data may not help much. However, the effect of adding source–target phrases into the training data is less explored (Reference Sen, Hasanuzzaman, Ekbal, Bhattacharyya and WaySen et al. in press).

Recently, unsupervised NMT (Lample et al. Reference Lample, Ott, Conneau, Denoyer and Ranzato2018; Artetxe, Labaka, and Agirre Reference Artetxe, Labaka and Agirre2018; Ren et al. Reference Ren, Zhang, Liu, Zhou and Ma2019; Lample and Conneau Reference Lample and Conneau2019), semi-supervised NMT (Zhang et al. Reference Zhang, Liu, Li, Zhou and Chen2018), and unsupervised pre-training (Ramachandran, Liu, and Le Reference Ramachandran, Liu and Le2017) have been emerged and shown promising results on the related languages. These techniques require a huge amount of monolingual data. It has been shown that pure unsupervised technique does not work for distance language pairs, for example, the pairs we are dealing with in this work (Guzmán et al. Reference Guzmán, Chen, Ott, Pino, Lample, Koehn, Chaudhary and Ranzato2019).

4. Proposed method

Our focus is on low-resource scenario, and in order to handle this situation, we add the phrase pairs extracted from the training corpus as feedback to the NMT framework during training. Our proposed approach is not specific to any NMT architecture. We perform experiments on two state-of-the-art neural networks, namely attention-based (Bahdanau et al. Reference Bahdanau, Cho and Bengio2015) GRU and transformer (Vaswani et al. Reference Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin2017) models. Here, we briefly describe the two networks and then present the details of the proposed method.

Figure 1. Attention-based GRU NMT architecture.

4.1 Attention-based GRU

The goal of NMT is to translate a sequence of source words into a sequence of target words with the help of a large neural network. The basic architecture of an NMT, shown in Figure 1, uses two recurrent neural networks, one is called encoder and other is known as the decoder. The encoder converts the source sentence into a dense fixed-length vector, and then the decoder generates target sentence from that vector. But the main drawback of this encoder–decoder approach is that it fails drastically as length of the input sentence grows. The encoder–decoder approach assumes that the encoder can encode the whole sentence into a fixed-length vector, which is not realistic, specifically for the longer sentences. To mitigate this drawback, Bahdanau et al. (Reference Bahdanau, Cho and Bengio2015) came up with an idea which focused on the whole input sentence, while generating the outputs.

Formally, given a sequence of source words x (= $x_1, x_2, x_3, ..., x_{T_{x}}$ ) and the previously translated $i-1$ words y (= $y_1, y_2, y_3, ..., y_{i-1}$ ), the conditional probability of the ith output $y_i$ is calculated as

(1) \begin{equation}p(y_i| \textbf{\textit{x}}, y_{<(i-1)}) = {\rm softmax}(W_ot_i)\end{equation}

where $t_i$ , the input to the softmax, is computed as

(2) \begin{equation}t_i = {\rm tanh}(W_{s}s_i+W_{e}y_{i-1}+W_{c}c_i)\end{equation}

where $W_{s}$ , $W_{e}$ , $W_{c}$ , and $W_o$ are the model parameters. The hidden state $s_i$ in the decoder at time step i is computed as

(3) \begin{equation}s_i = g(s_{i-1}, {y_{i-1}}, c_i)\end{equation}

Here, g is a nonlinear transform function, which is usually a long short-term memory (LSTM) (Hochreiter and Schmidhuber Reference Hochreiter and Schmidhuber1997) or a GRU (Cho et al. Reference Cho, Van Merriënboer, Bahdanau and Bengio2014), and $c_i$ is the context vector at time step i, which is calculated as a weighted sum of the input annotations $h_j$ :

(4) \begin{equation}c_i = \sum_{j=1}^{T_{x}} \alpha_{ij}\mathrm{h}_j\end{equation}

where $T_{x}$ is the length of the source sequence and $h_j$ is the encoder hidden state at jth time step and computed using a nonlinear transformation (such as GRU and LSTM) function as

(5) \begin{equation}h_j = f(h_{j-1}, {x_{j}})\end{equation}

The normalized weight $\alpha_{ij}$ for ${h}_j$ is calculated as

(6) \begin{eqnarray}\hspace*{-41pt}\alpha_{ij} &=& \frac{\mathrm{exp}(e_{ij})}{\sum_{k=1}^{T_{x}}\mathrm{exp}(e_{ik})}\end{eqnarray}

(7) \begin{eqnarray}e_{ij} &=& V^{T}_{a}{\rm tanh}(U_{a}s_{i-1}+W_{a}h_{j})\end{eqnarray}

where $V_a$ , $U_a$ , and $W_a$ are the trainable parameters. All of the parameters in the NMT model are optimized to maximize the following conditional log-likelihood of the N parallel sentences

(8) \begin{equation}\ell(\theta) = \frac{1}{N}\sum^N_{n=1}{\sum^{T_{y}}_{i=1}{\mathrm{log}p(y_{i}|s_i,y_{i-1},c_i)}}\end{equation}

where $T_{y}$ is the length of the target sequence.

4.2 Transformer network

In recurrent network, the representation at time step i is dependent of previous time stamps. Vaswani et al. (Reference Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin2017) proposed the transformer network, block diagrammatic representation is shown in Figure 2, which completely depends on self-attention and removes the recurrent operations found in the previous NMT approach—allowing for parallelization of computing at all time stamps in encoder–decoder. However, in the absence of recurrence, to capture the token position within the input sentence, a positional encoding is added with each input embedding before passing to the encoder. The encoder consists of several identical layers. Each layer is composed of mainly two sublayers: a multihead self-attention layer and a position-wise feed-forward network layer. The decoder consists of several identical layers like encoder and operates in a similar to encoder. In addition to the two sublayers in encoder, decoder inserts a third sublayer, which performs multihead attention over the output of the encoder. Each of these sublayers is followed by a layer called normalization. The decoder works similar to the decoder in Bahdanau et al. (Reference Bahdanau, Cho and Bengio2015) and generates one token at a time using a softmax layer. For more specific details of the network, please refer to Vaswani et al. (Reference Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin2017).

Figure 2. Transformer architecture.

Mathematically, as in Vaswani et al. (Reference Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin2017), positional encoding is defined as

(9) \begin{eqnarray} PE_{(pos,2i)} = {\rm sin}(pos / 10,000^{2i/d}) \end{eqnarray}

(10) \begin{eqnarray} \hspace*{-8pt}PE_{(pos,2i+1)} = {\rm cos}(pos / 10,000^{2i/d})\end{eqnarray}

where pos is the position, and i is the ith dimension of a d dimensional input vector. Self-attention is defined as

(11) \begin{equation} \mathrm{Attention}(Q, K, V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V\end{equation}

where Q, K, and V are the queries, keys, values packed together into matrices. $d_k$ is the ratio of d to number of heads denoted as h. One multihead is a concatenation of multiple h heads, defined as

(12) \begin{eqnarray} \mathrm{MultiHead}(Q, K, V) & = \mathrm{Concat}(\mathrm{head_1}, ..., \mathrm{head_{\it h}})W^O\end{eqnarray}

(13) \begin{eqnarray} \text{where}\ \mathrm{head_{\it i}} &= \mathrm{Attention}(QW^Q_i, KW^K_i, VW^V_i)\hspace*{-42.5pt}\end{eqnarray}

where the projections are parameter matrices $W^Q_i, W^K_i, W^O \in \mathbb{R}^{d \times d_k}$ . Feed-forward network in each layer of the encoder and decoder is formulated as

(14) \begin{equation} \mathrm{FFN}(x)=\max(0, xW_1 + b_1) W_2 + b_2\end{equation}

where the projections are parameter matrices $W_1, W_2 \in \mathbb{R}^{d \times d}$ .

4.3 Data augmentation

The NMT models are extremely data hungry, and in the absence of large training corpus, it does not learn the model parameters properly. In our work, we propose an approach for training NMT models using small corpora, especially under a situation for translating domain-specific small corpora.

The overall process flow of our architecture is depicted in Figure 3. The core idea is to provide more information about the alignment between the source and target phrases. When a sentence pair is passed through an encoder–decoder, the sentence pair does not carry any information about the mapping between the source and target phrases. The model learns the translation mappings implicitly by predicting and rectifying the error over a large parallel corpus. However, the model fails to learn the association between the phrases when the corpus size is small. So, apart from feeding sentence pairs into the network, we also feed phrase pairs as the training examples. This gives an illusion of having a larger corpus.

Figure 3. Our proposed phrase injection NMT approach.

In order to perform this feedback mechanism, we first extract parallel phrases from the corpus and then add these parallel phrases in the training set. To extract parallel phrases, we use the Moses (Koehn et al. Reference Koehn, Hoang, Birch, Callison-Burch, Federico, Bertoldi, Cowan, Shen, Moran, Zens, Dyer, Bojar, Constantin and Herbst2007) SMT system. We train a source–target phrase-based SMT (Koehn et al. Reference Koehn, Och and Marcu2003) and extract all the phrase pairs from the phrase table. Out of these, many parallel phrases are not sound, that is, there can be many incorrect source–target alignments (Koehn et al. Reference Koehn, Och and Marcu2003). We set different conditions while choosing the phrases from the phrase table. Assume that every source phrase e is aligned to a set of target phrases $F = (f_1, f_2, . . ., f_n)$ . Note that n may vary for each source phrase. So, for each source phrase e in the phrase table, we extract three sets of parallel phrases:

1. 1. First set ( $Set_{p\geq0.5}$ ): set of parallel phrases ${(e,f_t)}$ provided $P(f_t|e)\geq0.5$ ;

2. 2. Second set ( $Set_{p=1.0}$ ): set of parallel phrases ${(e,f_t)}$ provided $P(f_t|e)=1.0$ ;

3. 3. Third set ( $Set_{all}$ ): for this set, we consider all the phrase pairs from the phrase table.

Since the number of phrase pairs is larger than the number of original parallel sentences, to maintain a fair ratio between them, we use the following formula for combining phrase pairs with the original training set.

(15) \begin{equation}\begin{split}\text{Augmented corpus} & = N \times \text{original corpus} + \text{extracted phrase pairs}\end{split}\end{equation}

We combine the extracted set (of parallel phrases) with N times of the original corpus, where N is calculated as

\begin{equation}\nonumber N = \frac{\text{Number of extracted phrase pairs}}{\text{Number of original parallel sentences}}\end{equation}

Without this, training set will contain mostly the phrases, and as the phrases are smaller in length, they may make the model biased towards the phrase length.

5. Datasets

For experiments, we use English–Hindi and Hind–Bengali parallel corpora from the multilingual Indian Language Corpora Initiative (ILCI) (Jha Reference Jha2010). The ILCI parallel corpora are from the two domains: Health (ILCI-H) and Tourism (ILCI-T), each of these comprising of 25k parallel sentences. These corpora have insufficient number of parallel sentences compared to the other language pairs found in literature. Indian languages do not have sufficient corpus required to train an NMT system, and thus they fall under the low-resource category as the standard NMT requires millions of parallel sentences for training. For experimentation, we randomly split each corpus into three sets: Train, Test, and Dev. Details are shown in Table 1.

Table 1. Dataset statistics showing the number of sentences and tokens. By old, we refer to old English and by mod, we refer to modern English

For judicial domain data, we use IIT Bombay English-Hindi parallel corpus (Kunchukuttan, Mehta, and Bhattacharyya Reference Kunchukuttan, Mehta and Bhattacharyya2018). It consists of parallel sentences from miscellaneous domains and out of which only 7561 parallel sentences belong to the judicial domain. For experiments, we randomly split these judicial domain parallel sentences into Train, Dev, and Test sets consisting of 5561, 1000, and 1000, respectively.

As old English texts, we use the publicly available The Homilies of the Anglo-Saxon Church Footnote a by Ælfric of Eynsham (c.950–c.1010) who was a prolific author in old English and its translation by Benjamin Thorpe (c.1782–c.1870) as modern English texts. We call it Old English-Modern English (OE-ME) corpus.

The OE–ME corpus is tiny in size, and it has 720 parallel paragraphs in 40 sections. Most of the parallel paragraphs have equal number of OE–ME parallel sentences which help in aligning the parallel sentences. Some parallel paragraphs are discarded as they do not have equal number of OE–ME sentences to avoid misalignment which gives rise to a total of 3716 parallel sentences. We do not use any sentence aligner as only a few sentences are discarded. We randomly split it into three sets: Train, Test, and Dev containing 2716, 500, and 500 sentences, respectively. For tokenization, we use tokenizer.perl which is part of the Moses SMT system. Details of the datasets are presented in Table 1.

Table 2. Vocabulary size for different language pairs

O, Old; M, Modern.

Table 3. Training data sizes for different models after adding phrases

E, English; H, Hindi; B, Bengali; O, old; M, modern.

6. Experimental setup

Attention-based GRU models: We use Nematus (Sennrich et al. Reference Sennrich, Firat, Cho, Birch, Haddow, Hitschler, Junczys-Dowmunt, Läubli, Miceli Barone, Mokry and Nadejde2017) for training the NMT models. Our neural models are trained on word level. We create vocabulary from the training set for the different systems. The size of the vocabularies used in training the models is shown in Table 2. The augmented data size for each model are shown in Table 3. We set the embedding size as 128, hidden size as 256, and the learning rate as 0.001. Note that we tried higher embedding and hidden dimensions but did not work as the training data size is very small. Encoder and decoder are two-layered GRU blocks. The models are trained with mini-batch size of 40, and we restrict the maximum sentence length to 80. We use the Adam optimizer (Kingma and Ba Reference Kingma and Ba2015) for optimizing the models. The training stops on meeting the early stopping criteria. We use the early stopping based on BLEU measure with an early stopping patience value 10. All the models run for 110–130k (approx.) updates before the early stopping. For decoding, we set the beam size as 3. For the other parameters, default values were used.

Transformer-based models: For training the models, we use Sockeye (Hieber et al. Reference Hieber, Domhan, Denkowski, Vilar, Sokolov, Clifton and Post2017), a toolkit for NMT. We set default the embedding dimension of 512, hidden dimension of 512, learning rate of 0.0002, and dropout rate of 0.2. Number of layers in each of encoder and decoder is 6. Number of multihead attention is 8. We use Adam optimizer (Kingma and Ba Reference Kingma and Ba2015) optimizer. We keep a mini-batch size of 2000 words.Footnote b

Phrase extraction: We use the Moses (Koehn et al. Reference Koehn, Hoang, Birch, Callison-Burch, Federico, Bertoldi, Cowan, Shen, Moran, Zens, Dyer, Bojar, Constantin and Herbst2007) toolkit for training a phrase-based SMT system. The phrase table, generated during training, is used for extracting the phrases. For training, we keep the following settings in the Moses: grow-diag-final-and heuristics for word alignment, msd-bidirectional-fe for reordering model, and 4-gram LM with modified Kneser–Ney smoothing (Kneser and Ney Reference Kneser and Ney1995) using KenLM (Heafield Reference Heafield2011). However, we note that the order of LM does not affect the phrase table.

We train the following three types of NMT models for each of health, tourism, and judicial domain corpora.

1. 1. Baseline: The NMT model is trained only on the original parallel corpus.

2. 2. Baseline + $Set_{p\geq0.5}$ : The NMT is trained on the original parallel corpus along with phrase pair set $Set_{p\geq0.5}$ (see Section 4.3).

3. 3. Baseline + $Set_{p=1.0}$ : The NMT is trained on the original parallel corpus along with phrase pair set $Set_{p=1.0}$ (see Section 4.3).

4. 4. Baseline + $Set_{all}$ : The NMT model is trained on the original parallel corpus along with phrase pair set $Set_{all}$ (see Section 4.3).

The number of training examples for the systems as mentioned above are shown in Table 3.