NICT_LOGO.JPG KYOTO-U_LOGO.JPG

WAT

The Workshop on Asian Translation
Evaluation Results

[EVALUATION RESULTS TOP] | [BLEU] | [RIBES] | [AMFM] | [HUMAN (WAT2022)] | [HUMAN (WAT2021)] | [HUMAN (WAT2020)] | [HUMAN (WAT2019)] | [HUMAN (WAT2018)] | [HUMAN (WAT2017)] | [HUMAN (WAT2016)] | [HUMAN (WAT2015)] | [HUMAN (WAT2014)] | [EVALUATION RESULTS USAGE POLICY]

BLEU


# Team Task Date/Time DataID BLEU
Method
Other
Resources
System
Description
juman kytea mecab moses-
tokenizer
stanford-
segmenter-
ctb
stanford-
segmenter-
pku
indic-
tokenizer
unuse myseg kmseg
1NLPHutINDIC21or-en2021/03/19 16:31:184597---18.92------NMTNoTransformer trained using all or-en data and all hi-en data. Then fine-tuned using all or-en data.
2ORGANIZERINDIC21or-en2021/04/08 17:24:424801---16.35------NMTNoBilingual baseline trained on PMI data. Transformer base. LR=10-3
3NICT-5INDIC21or-en2021/04/21 15:44:475286---25.81------NMTNoPretrain MBART on IndicCorp and FT on bilingual PMI data. Beam search. Model is bilingual.
4SRPOLINDIC21or-en2021/04/21 19:33:015331---30.07------NMTNoBase transformer on all WAT21 data
5NICT-5INDIC21or-en2021/04/22 11:53:285361---27.93------NMTNoMBART+MNMT. Beam 4.
6gaurvarINDIC21or-en2021/04/25 18:18:265541---12.32------NMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages
7gaurvarINDIC21or-en2021/04/25 18:34:485550---13.71------NMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages
8gaurvarINDIC21or-en2021/04/25 18:48:405561---13.05------NMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages
9gaurvarINDIC21or-en2021/04/25 19:01:235571---13.69------NMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages
10sakuraINDIC21or-en2021/04/30 22:55:205876---32.67------NMTNoFine-tuning of multilingual mBART many2many model with training corpus.
11IIIT-HINDIC21or-en2021/05/03 18:15:026022---34.11------NMTNoMNMT system (XX-En) trained via exploiting lexical similarity on PMI+CVIT parallel corpus, then improved using back translation on PMI monolingual data followed by fine tuning.
12CFILTINDIC21or-en2021/05/04 01:16:296058---30.46------NMTNoMultilingual(Many-to-One(XX-En)) NMT model based on Transformer with shared encoder and decoder.
13coastalINDIC21or-en2021/05/04 01:46:056107---19.61------NMTNoseq2seq model trained on all WAT2021 data
14CFILT-IITBINDIC21or-en2021/05/04 01:54:166119---25.05------NMTNoMultilingual NMT (Many to One): Transformer based model with shared encoder-decoder and shared BPE vocabulary trained using all indic language data converted to same script
15CFILT-IITBINDIC21or-en2021/05/04 01:59:276128---26.34------NMTNoMultilingual NMT (Many to One): Transformer based model with shared encoder-decoder and shared BPE vocabulary trained using all Indo-Aryan languages data converted to same script
16sakuraINDIC21or-en2021/05/04 13:19:366208---32.82------NMTNoPre-training multilingual mBART many2many model with training corpus followed by finetuning on PMI Parallel.
17SRPOLINDIC21or-en2021/05/04 15:27:426248---37.06------NMTNoEnsemble of many-to-one on all data. Pretrained on BT, finetuned on PMI
18SRPOLINDIC21or-en2021/05/04 16:32:356274---36.04------NMTNoMany-to-one on all data. Pretrained on BT, finetuned on PMI
19IITP-MTINDIC21or-en2021/05/04 18:07:206294---31.19------NMTNoMany-to-One model trained on all training data with base Transformer. All indic language data is romanized. Model fine-tuned on BT PMI monolingual corpus.
20mcairtINDIC21or-en2021/05/04 19:21:486338---29.96------NMTNomultilingual model(many to one) trained on all WAT 2021 data by using base transformer.
21NICT-5INDIC21or-en2021/06/21 12:04:386478---32.08------NMTNoUsing PMI and PIB data for fine-tuning on am mbart model trained for over 5 epochs.
22NICT-5INDIC21or-en2021/06/25 11:49:546499---33.09------NMTNoUsing PMI and PIB data for fine-tuning on a mbart model trained for over 5 epochs. MNMT model.

Notice:

Back to top

RIBES


# Team Task Date/Time DataID RIBES
Method
Other
Resources
System
Description
juman kytea mecab moses-
tokenizer
stanford-
segmenter-
ctb
stanford-
segmenter-
pku
indic-
tokenizer
unuse myseg kmseg
1NLPHutINDIC21or-en2021/03/19 16:31:184597---0.720916------NMTNoTransformer trained using all or-en data and all hi-en data. Then fine-tuned using all or-en data.
2ORGANIZERINDIC21or-en2021/04/08 17:24:424801---0.679781------NMTNoBilingual baseline trained on PMI data. Transformer base. LR=10-3
3NICT-5INDIC21or-en2021/04/21 15:44:475286---0.762604------NMTNoPretrain MBART on IndicCorp and FT on bilingual PMI data. Beam search. Model is bilingual.
4SRPOLINDIC21or-en2021/04/21 19:33:015331---0.789470------NMTNoBase transformer on all WAT21 data
5NICT-5INDIC21or-en2021/04/22 11:53:285361---0.769634------NMTNoMBART+MNMT. Beam 4.
6gaurvarINDIC21or-en2021/04/25 18:18:265541---0.604738------NMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages
7gaurvarINDIC21or-en2021/04/25 18:34:485550---0.634313------NMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages
8gaurvarINDIC21or-en2021/04/25 18:48:405561---0.668638------NMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages
9gaurvarINDIC21or-en2021/04/25 19:01:235571---0.662493------NMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages
10sakuraINDIC21or-en2021/04/30 22:55:205876---0.801734------NMTNoFine-tuning of multilingual mBART many2many model with training corpus.
11IIIT-HINDIC21or-en2021/05/03 18:15:026022---0.795132------NMTNoMNMT system (XX-En) trained via exploiting lexical similarity on PMI+CVIT parallel corpus, then improved using back translation on PMI monolingual data followed by fine tuning.
12CFILTINDIC21or-en2021/05/04 01:16:296058---0.772850------NMTNoMultilingual(Many-to-One(XX-En)) NMT model based on Transformer with shared encoder and decoder.
13coastalINDIC21or-en2021/05/04 01:46:056107---0.737380------NMTNoseq2seq model trained on all WAT2021 data
14CFILT-IITBINDIC21or-en2021/05/04 01:54:166119---0.754313------NMTNoMultilingual NMT (Many to One): Transformer based model with shared encoder-decoder and shared BPE vocabulary trained using all indic language data converted to same script
15CFILT-IITBINDIC21or-en2021/05/04 01:59:276128---0.761082------NMTNoMultilingual NMT (Many to One): Transformer based model with shared encoder-decoder and shared BPE vocabulary trained using all Indo-Aryan languages data converted to same script
16sakuraINDIC21or-en2021/05/04 13:19:366208---0.800209------NMTNoPre-training multilingual mBART many2many model with training corpus followed by finetuning on PMI Parallel.
17SRPOLINDIC21or-en2021/05/04 15:27:426248---0.816956------NMTNoEnsemble of many-to-one on all data. Pretrained on BT, finetuned on PMI
18SRPOLINDIC21or-en2021/05/04 16:32:356274---0.812816------NMTNoMany-to-one on all data. Pretrained on BT, finetuned on PMI
19IITP-MTINDIC21or-en2021/05/04 18:07:206294---0.794791------NMTNoMany-to-One model trained on all training data with base Transformer. All indic language data is romanized. Model fine-tuned on BT PMI monolingual corpus.
20mcairtINDIC21or-en2021/05/04 19:21:486338---0.798326------NMTNomultilingual model(many to one) trained on all WAT 2021 data by using base transformer.
21NICT-5INDIC21or-en2021/06/21 12:04:386478---0.787803------NMTNoUsing PMI and PIB data for fine-tuning on am mbart model trained for over 5 epochs.
22NICT-5INDIC21or-en2021/06/25 11:49:546499---0.788801------NMTNoUsing PMI and PIB data for fine-tuning on a mbart model trained for over 5 epochs. MNMT model.

Notice:

Back to top

AMFM


# Team Task Date/Time DataID AMFM
Method
Other
Resources
System
Description
unuse unuse unuse unuse unuse unuse unuse unuse unuse unuse
1NLPHutINDIC21or-en2021/03/19 16:31:184597---0.740606------NMTNoTransformer trained using all or-en data and all hi-en data. Then fine-tuned using all or-en data.
2ORGANIZERINDIC21or-en2021/04/08 17:24:424801---0.730819------NMTNoBilingual baseline trained on PMI data. Transformer base. LR=10-3
3NICT-5INDIC21or-en2021/04/21 15:44:475286---0.780431------NMTNoPretrain MBART on IndicCorp and FT on bilingual PMI data. Beam search. Model is bilingual.
4SRPOLINDIC21or-en2021/04/21 19:33:015331---0.794017------NMTNoBase transformer on all WAT21 data
5NICT-5INDIC21or-en2021/04/22 11:53:285361---0.782917------NMTNoMBART+MNMT. Beam 4.
6gaurvarINDIC21or-en2021/04/25 18:18:265541---0.726829------NMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages
7gaurvarINDIC21or-en2021/04/25 18:34:485550---0.725121------NMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages
8gaurvarINDIC21or-en2021/04/25 18:48:405561---0.718668------NMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages
9gaurvarINDIC21or-en2021/04/25 19:01:235571---0.721531------NMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages
10sakuraINDIC21or-en2021/04/30 22:55:205876---0.808239------NMTNoFine-tuning of multilingual mBART many2many model with training corpus.
11IIIT-HINDIC21or-en2021/05/03 18:15:026022---0.804930------NMTNoMNMT system (XX-En) trained via exploiting lexical similarity on PMI+CVIT parallel corpus, then improved using back translation on PMI monolingual data followed by fine tuning.
12CFILTINDIC21or-en2021/05/04 01:16:296058---0.793769------NMTNoMultilingual(Many-to-One(XX-En)) NMT model based on Transformer with shared encoder and decoder.
13coastalINDIC21or-en2021/05/04 01:46:056107---0.727657------NMTNoseq2seq model trained on all WAT2021 data
14CFILT-IITBINDIC21or-en2021/05/04 01:54:166119---0.770941------NMTNoMultilingual NMT (Many to One): Transformer based model with shared encoder-decoder and shared BPE vocabulary trained using all indic language data converted to same script
15CFILT-IITBINDIC21or-en2021/05/04 01:59:276128---0.780009------NMTNoMultilingual NMT (Many to One): Transformer based model with shared encoder-decoder and shared BPE vocabulary trained using all Indo-Aryan languages data converted to same script
16sakuraINDIC21or-en2021/05/04 13:19:366208---0.805953------NMTNoPre-training multilingual mBART many2many model with training corpus followed by finetuning on PMI Parallel.
17SRPOLINDIC21or-en2021/05/04 15:27:426248---0.817318------NMTNoEnsemble of many-to-one on all data. Pretrained on BT, finetuned on PMI
18SRPOLINDIC21or-en2021/05/04 16:32:356274---0.814871------NMTNoMany-to-one on all data. Pretrained on BT, finetuned on PMI
19IITP-MTINDIC21or-en2021/05/04 18:07:206294---0.803226------NMTNoMany-to-One model trained on all training data with base Transformer. All indic language data is romanized. Model fine-tuned on BT PMI monolingual corpus.
20mcairtINDIC21or-en2021/05/04 19:21:486338---0.795586------NMTNomultilingual model(many to one) trained on all WAT 2021 data by using base transformer.
21NICT-5INDIC21or-en2021/06/21 12:04:386478---0.000000------NMTNoUsing PMI and PIB data for fine-tuning on am mbart model trained for over 5 epochs.
22NICT-5INDIC21or-en2021/06/25 11:49:546499---0.000000------NMTNoUsing PMI and PIB data for fine-tuning on a mbart model trained for over 5 epochs. MNMT model.

Notice:

Back to top

HUMAN (WAT2022)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description

Notice:
Back to top

HUMAN (WAT2021)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description
1NLPHutINDIC21or-en2021/03/19 16:31:184597UnderwayNMTNoTransformer trained using all or-en data and all hi-en data. Then fine-tuned using all or-en data.
2NICT-5INDIC21or-en2021/04/21 15:44:475286UnderwayNMTNoPretrain MBART on IndicCorp and FT on bilingual PMI data. Beam search. Model is bilingual.
3NICT-5INDIC21or-en2021/04/22 11:53:285361UnderwayNMTNoMBART+MNMT. Beam 4.
4gaurvarINDIC21or-en2021/04/25 18:34:485550UnderwayNMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages
5gaurvarINDIC21or-en2021/04/25 19:01:235571UnderwayNMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages
6sakuraINDIC21or-en2021/04/30 22:55:205876UnderwayNMTNoFine-tuning of multilingual mBART many2many model with training corpus.
7IIIT-HINDIC21or-en2021/05/03 18:15:026022UnderwayNMTNoMNMT system (XX-En) trained via exploiting lexical similarity on PMI+CVIT parallel corpus, then improved using back translation on PMI monolingual data followed by fine tuning.
8CFILTINDIC21or-en2021/05/04 01:16:296058UnderwayNMTNoMultilingual(Many-to-One(XX-En)) NMT model based on Transformer with shared encoder and decoder.
9coastalINDIC21or-en2021/05/04 01:46:056107UnderwayNMTNoseq2seq model trained on all WAT2021 data
10CFILT-IITBINDIC21or-en2021/05/04 01:54:166119UnderwayNMTNoMultilingual NMT (Many to One): Transformer based model with shared encoder-decoder and shared BPE vocabulary trained using all indic language data converted to same script
11CFILT-IITBINDIC21or-en2021/05/04 01:59:276128UnderwayNMTNoMultilingual NMT (Many to One): Transformer based model with shared encoder-decoder and shared BPE vocabulary trained using all Indo-Aryan languages data converted to same script
12SRPOLINDIC21or-en2021/05/04 15:27:426248UnderwayNMTNoEnsemble of many-to-one on all data. Pretrained on BT, finetuned on PMI
13SRPOLINDIC21or-en2021/05/04 16:32:356274UnderwayNMTNoMany-to-one on all data. Pretrained on BT, finetuned on PMI
14IITP-MTINDIC21or-en2021/05/04 18:07:206294UnderwayNMTNoMany-to-One model trained on all training data with base Transformer. All indic language data is romanized. Model fine-tuned on BT PMI monolingual corpus.
15mcairtINDIC21or-en2021/05/04 19:21:486338UnderwayNMTNomultilingual model(many to one) trained on all WAT 2021 data by using base transformer.

Notice:
Back to top

HUMAN (WAT2020)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description

Notice:
Back to top

HUMAN (WAT2019)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description

Notice:
Back to top

HUMAN (WAT2018)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description

Notice:
Back to top

HUMAN (WAT2017)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description

Notice:
Back to top

HUMAN (WAT2016)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description

Notice:
Back to top

HUMAN (WAT2015)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description

Notice:
Back to top

HUMAN (WAT2014)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description

Notice:
Back to top

EVALUATION RESULTS USAGE POLICY

When you use the WAT evaluation results for any purpose such as:
- writing technical papers,
- making presentations about your system,
- advertising your MT system to the customers,
you can use the information about translation directions, scores (including both automatic and human evaluations) and ranks of your system among others. You can also use the scores of the other systems, but you MUST anonymize the other system's names. In addition, you can show the links (URLs) to the WAT evaluation result pages.

NICT (National Institute of Information and Communications Technology)
Kyoto University
Last Modified: 2018-08-02