NICT_LOGO.JPG KYOTO-U_LOGO.JPG

WAT

The Workshop on Asian Translation
Evaluation Results

[EVALUATION RESULTS TOP] | [BLEU] | [RIBES] | [AMFM] | [HUMAN (WAT2022)] | [HUMAN (WAT2021)] | [HUMAN (WAT2020)] | [HUMAN (WAT2019)] | [HUMAN (WAT2018)] | [HUMAN (WAT2017)] | [HUMAN (WAT2016)] | [HUMAN (WAT2015)] | [HUMAN (WAT2014)] | [EVALUATION RESULTS USAGE POLICY]

BLEU


# Team Task Date/Time DataID BLEU
Method
Other
Resources
System
Description
juman kytea mecab moses-
tokenizer
stanford-
segmenter-
ctb
stanford-
segmenter-
pku
indic-
tokenizer
unuse myseg kmseg
1NLPHutINDIC21en-or2021/03/19 16:29:304596------12.81---NMTNoTransformer with target language tag trained using all languages PMI data. Then fine-tuned using all en-or data.
2ORGANIZERINDIC21en-or2021/04/08 17:24:284800------ 9.08---NMTNoBilingual baseline trained on PMI data. Transformer base. LR=10-3
3NICT-5INDIC21en-or2021/04/21 15:44:245285------15.01---NMTNoPretrain MBART on IndicCorp and FT on bilingual PMI data. Beam search. Model is bilingual.
4SRPOLINDIC21en-or2021/04/21 19:21:195321------12.33---SMTNoBase transformer on all WAT21 data
5NICT-5INDIC21en-or2021/04/22 11:53:175360------16.69---NMTNoMBART+MNMT. Beam 4.
6gaurvarINDIC21en-or2021/04/25 20:00:385584------ 2.20---NMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages
7sakuraINDIC21en-or2021/05/01 11:34:255888------17.81---NMTNoFine-tuning of multilingual mBART one2many model with training corpus.
8gaurvarINDIC21en-or2021/05/01 19:32:305932------ 2.60---NMTNo
9mcairtINDIC21en-or2021/05/03 17:20:415996------17.71---NMTNomultilingual model(one to many model) trained on all WAT 2021 data by using base transformer.
10IIIT-HINDIC21en-or2021/05/03 18:10:236011------20.15---NMTNoMNMT system (En-XX) trained via exploiting lexical similarity on PMI+CVIT parallel corpus, then improved using back translation on PMI monolingual data followed by fine tuning.
11CFILTINDIC21en-or2021/05/04 01:05:156048------18.22---NMTNoMultilingual(One-to-Many(En-XX)) NMT model based on Transformer with shared encoder and decoder.
12coastalINDIC21en-or2021/05/04 01:39:216084------15.66---NMTNoseq2seq model trained on all WAT2021 data
13sakuraINDIC21en-or2021/05/04 04:13:426157------17.88---NMTNoPre-training multilingual mBART one2many model with training corpus followed by finetuning on PMI Parallel.
14SRPOLINDIC21en-or2021/05/04 15:21:356238------19.94---NMTNoEnsemble of one-to-many on all data. Pretrained on BT, finetuned on PMI
15SRPOLINDIC21en-or2021/05/04 16:27:266264------19.15---NMTNoOne-to-many on all data. Pretrained on BT, finetuned on PMI
16IITP-MTINDIC21en-or2021/05/04 18:05:326293------12.57---NMTNoOne-to-Many model trained on all training data with base Transformer. All indic language data is romanized. Model fine-tuned on BT PMI monolingual corpus.
17NICT-5INDIC21en-or2021/06/25 11:38:496489------19.20---NMTNoUsing PMI and PIB data for fine-tuning on a mbart model trained for over 5 epochs. MNMT model.

Notice:

Back to top

RIBES


# Team Task Date/Time DataID RIBES
Method
Other
Resources
System
Description
juman kytea mecab moses-
tokenizer
stanford-
segmenter-
ctb
stanford-
segmenter-
pku
indic-
tokenizer
unuse myseg kmseg
1NLPHutINDIC21en-or2021/03/19 16:29:304596------0.693696---NMTNoTransformer with target language tag trained using all languages PMI data. Then fine-tuned using all en-or data.
2ORGANIZERINDIC21en-or2021/04/08 17:24:284800------0.638520---NMTNoBilingual baseline trained on PMI data. Transformer base. LR=10-3
3NICT-5INDIC21en-or2021/04/21 15:44:245285------0.716665---NMTNoPretrain MBART on IndicCorp and FT on bilingual PMI data. Beam search. Model is bilingual.
4SRPOLINDIC21en-or2021/04/21 19:21:195321------0.714550---SMTNoBase transformer on all WAT21 data
5NICT-5INDIC21en-or2021/04/22 11:53:175360------0.734028---NMTNoMBART+MNMT. Beam 4.
6gaurvarINDIC21en-or2021/04/25 20:00:385584------0.380253---NMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages
7sakuraINDIC21en-or2021/05/01 11:34:255888------0.741763---NMTNoFine-tuning of multilingual mBART one2many model with training corpus.
8gaurvarINDIC21en-or2021/05/01 19:32:305932------0.431373---NMTNo
9mcairtINDIC21en-or2021/05/03 17:20:415996------0.743984---NMTNomultilingual model(one to many model) trained on all WAT 2021 data by using base transformer.
10IIIT-HINDIC21en-or2021/05/03 18:10:236011------0.750260---NMTNoMNMT system (En-XX) trained via exploiting lexical similarity on PMI+CVIT parallel corpus, then improved using back translation on PMI monolingual data followed by fine tuning.
11CFILTINDIC21en-or2021/05/04 01:05:156048------0.738397---NMTNoMultilingual(One-to-Many(En-XX)) NMT model based on Transformer with shared encoder and decoder.
12coastalINDIC21en-or2021/05/04 01:39:216084------0.727477---NMTNoseq2seq model trained on all WAT2021 data
13sakuraINDIC21en-or2021/05/04 04:13:426157------0.740263---NMTNoPre-training multilingual mBART one2many model with training corpus followed by finetuning on PMI Parallel.
14SRPOLINDIC21en-or2021/05/04 15:21:356238------0.751086---NMTNoEnsemble of one-to-many on all data. Pretrained on BT, finetuned on PMI
15SRPOLINDIC21en-or2021/05/04 16:27:266264------0.749740---NMTNoOne-to-many on all data. Pretrained on BT, finetuned on PMI
16IITP-MTINDIC21en-or2021/05/04 18:05:326293------0.714731---NMTNoOne-to-Many model trained on all training data with base Transformer. All indic language data is romanized. Model fine-tuned on BT PMI monolingual corpus.
17NICT-5INDIC21en-or2021/06/25 11:38:496489------0.747217---NMTNoUsing PMI and PIB data for fine-tuning on a mbart model trained for over 5 epochs. MNMT model.

Notice:

Back to top

AMFM


# Team Task Date/Time DataID AMFM
Method
Other
Resources
System
Description
unuse unuse unuse unuse unuse unuse unuse unuse unuse unuse
1NLPHutINDIC21en-or2021/03/19 16:29:304596------0.736638---NMTNoTransformer with target language tag trained using all languages PMI data. Then fine-tuned using all en-or data.
2ORGANIZERINDIC21en-or2021/04/08 17:24:284800------0.714530---NMTNoBilingual baseline trained on PMI data. Transformer base. LR=10-3
3NICT-5INDIC21en-or2021/04/21 15:44:245285------0.748319---NMTNoPretrain MBART on IndicCorp and FT on bilingual PMI data. Beam search. Model is bilingual.
4SRPOLINDIC21en-or2021/04/21 19:21:195321------0.723507---SMTNoBase transformer on all WAT21 data
5NICT-5INDIC21en-or2021/04/22 11:53:175360------0.757804---NMTNoMBART+MNMT. Beam 4.
6gaurvarINDIC21en-or2021/04/25 20:00:385584------0.591864---NMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages
7sakuraINDIC21en-or2021/05/01 11:34:255888------0.767385---NMTNoFine-tuning of multilingual mBART one2many model with training corpus.
8gaurvarINDIC21en-or2021/05/01 19:32:305932------0.611704---NMTNo
9mcairtINDIC21en-or2021/05/03 17:20:415996------0.763064---NMTNomultilingual model(one to many model) trained on all WAT 2021 data by using base transformer.
10IIIT-HINDIC21en-or2021/05/03 18:10:236011------0.735718---NMTNoMNMT system (En-XX) trained via exploiting lexical similarity on PMI+CVIT parallel corpus, then improved using back translation on PMI monolingual data followed by fine tuning.
11CFILTINDIC21en-or2021/05/04 01:05:156048------0.768399---NMTNoMultilingual(One-to-Many(En-XX)) NMT model based on Transformer with shared encoder and decoder.
12coastalINDIC21en-or2021/05/04 01:39:216084------0.758199---NMTNoseq2seq model trained on all WAT2021 data
13sakuraINDIC21en-or2021/05/04 04:13:426157------0.769884---NMTNoPre-training multilingual mBART one2many model with training corpus followed by finetuning on PMI Parallel.
14SRPOLINDIC21en-or2021/05/04 15:21:356238------0.771831---NMTNoEnsemble of one-to-many on all data. Pretrained on BT, finetuned on PMI
15SRPOLINDIC21en-or2021/05/04 16:27:266264------0.771493---NMTNoOne-to-many on all data. Pretrained on BT, finetuned on PMI
16IITP-MTINDIC21en-or2021/05/04 18:05:326293------0.737576---NMTNoOne-to-Many model trained on all training data with base Transformer. All indic language data is romanized. Model fine-tuned on BT PMI monolingual corpus.
17NICT-5INDIC21en-or2021/06/25 11:38:496489------0.000000---NMTNoUsing PMI and PIB data for fine-tuning on a mbart model trained for over 5 epochs. MNMT model.

Notice:

Back to top

HUMAN (WAT2022)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description

Notice:
Back to top

HUMAN (WAT2021)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description
1NLPHutINDIC21en-or2021/03/19 16:29:304596UnderwayNMTNoTransformer with target language tag trained using all languages PMI data. Then fine-tuned using all en-or data.
2NICT-5INDIC21en-or2021/04/21 15:44:245285UnderwayNMTNoPretrain MBART on IndicCorp and FT on bilingual PMI data. Beam search. Model is bilingual.
3NICT-5INDIC21en-or2021/04/22 11:53:175360UnderwayNMTNoMBART+MNMT. Beam 4.
4gaurvarINDIC21en-or2021/04/25 20:00:385584UnderwayNMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages
5gaurvarINDIC21en-or2021/05/01 19:32:305932UnderwayNMTNo
6mcairtINDIC21en-or2021/05/03 17:20:415996UnderwayNMTNomultilingual model(one to many model) trained on all WAT 2021 data by using base transformer.
7IIIT-HINDIC21en-or2021/05/03 18:10:236011UnderwayNMTNoMNMT system (En-XX) trained via exploiting lexical similarity on PMI+CVIT parallel corpus, then improved using back translation on PMI monolingual data followed by fine tuning.
8CFILTINDIC21en-or2021/05/04 01:05:156048UnderwayNMTNoMultilingual(One-to-Many(En-XX)) NMT model based on Transformer with shared encoder and decoder.
9coastalINDIC21en-or2021/05/04 01:39:216084UnderwayNMTNoseq2seq model trained on all WAT2021 data
10sakuraINDIC21en-or2021/05/04 04:13:426157UnderwayNMTNoPre-training multilingual mBART one2many model with training corpus followed by finetuning on PMI Parallel.
11SRPOLINDIC21en-or2021/05/04 15:21:356238UnderwayNMTNoEnsemble of one-to-many on all data. Pretrained on BT, finetuned on PMI
12SRPOLINDIC21en-or2021/05/04 16:27:266264UnderwayNMTNoOne-to-many on all data. Pretrained on BT, finetuned on PMI
13IITP-MTINDIC21en-or2021/05/04 18:05:326293UnderwayNMTNoOne-to-Many model trained on all training data with base Transformer. All indic language data is romanized. Model fine-tuned on BT PMI monolingual corpus.

Notice:
Back to top

HUMAN (WAT2020)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description

Notice:
Back to top

HUMAN (WAT2019)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description

Notice:
Back to top

HUMAN (WAT2018)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description

Notice:
Back to top

HUMAN (WAT2017)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description

Notice:
Back to top

HUMAN (WAT2016)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description

Notice:
Back to top

HUMAN (WAT2015)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description

Notice:
Back to top

HUMAN (WAT2014)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description

Notice:
Back to top

EVALUATION RESULTS USAGE POLICY

When you use the WAT evaluation results for any purpose such as:
- writing technical papers,
- making presentations about your system,
- advertising your MT system to the customers,
you can use the information about translation directions, scores (including both automatic and human evaluations) and ranks of your system among others. You can also use the scores of the other systems, but you MUST anonymize the other system's names. In addition, you can show the links (URLs) to the WAT evaluation result pages.

NICT (National Institute of Information and Communications Technology)
Kyoto University
Last Modified: 2018-08-02