NICT_LOGO.JPG KYOTO-U_LOGO.JPG

WAT

The Workshop on Asian Translation
Evaluation Results

[EVALUATION RESULTS TOP] | [BLEU] | [RIBES] | [AMFM] | [HUMAN (WAT2022)] | [HUMAN (WAT2021)] | [HUMAN (WAT2020)] | [HUMAN (WAT2019)] | [HUMAN (WAT2018)] | [HUMAN (WAT2017)] | [HUMAN (WAT2016)] | [HUMAN (WAT2015)] | [HUMAN (WAT2014)] | [EVALUATION RESULTS USAGE POLICY]

BLEU


# Team Task Date/Time DataID BLEU
Method
Other
Resources
System
Description
juman kytea mecab moses-
tokenizer
stanford-
segmenter-
ctb
stanford-
segmenter-
pku
indic-
tokenizer
unuse myseg kmseg
1SRPOLINDIC21kn-en2021/05/04 15:26:086245---40.34------NMTNoEnsemble of many-to-one on all data. Pretrained on BT, finetuned on PMI
2SRPOLINDIC21kn-en2021/05/04 16:31:246271---39.01------NMTNoMany-to-one on all data. Pretrained on BT, finetuned on PMI
3sakuraINDIC21kn-en2021/05/04 13:14:166205---35.46------NMTNoPre-training multilingual mBART many2many model with training corpus followed by finetuning on PMI Parallel.
4NICT-5INDIC21kn-en2021/06/25 11:48:366496---35.11------NMTNoUsing PMI and PIB data for fine-tuning on a mbart model trained for over 5 epochs. MNMT model.
5IIIT-HINDIC21kn-en2021/05/03 18:13:306018---34.69------NMTNoMNMT system (XX-En) trained via exploiting lexical similarity on PMI+CVIT parallel corpus, then improved using back translation on PMI monolingual data followed by fine tuning.
6sakuraINDIC21kn-en2021/04/30 22:45:495873---34.11------NMTNoFine-tuning of multilingual mBART many2many model with training corpus.
7NICT-5INDIC21kn-en2021/06/21 14:32:566482---31.74------NMTNoUsing PMI and PIB data for fine-tuning on am mbart model trained for over 5 epochs.
8IITP-MTINDIC21kn-en2021/05/04 17:53:286286---31.24------NMTNoMany-to-One model trained on all training data with base Transformer. All indic language data is romanized. Model fine-tuned on BT PMI monolingual corpus.
9mcairtINDIC21kn-en2021/05/04 20:02:546374---31.16------NMTNomultilingual model(many to one) trained on all WAT 2021 data by using base transformer.
10coastalINDIC21kn-en2021/05/04 05:41:596165---31.04------NMTNomT5 trained only on PMI
11NICT-5INDIC21kn-en2021/04/22 11:51:575355---30.87------NMTNoMBART+MNMT. Beam 4.
12SRPOLINDIC21kn-en2021/04/21 19:31:425328---30.67------NMTNoBase transformer on all WAT21 data
13CFILTINDIC21kn-en2021/05/04 01:13:126055---30.23------NMTNoMultilingual(Many-to-One(XX-En)) NMT model based on Transformer with shared encoder and decoder.
14NICT-5INDIC21kn-en2021/04/21 15:43:005280---29.29------NMTNoPretrain MBART on IndicCorp and FT on bilingual PMI data. Beam search. Model is bilingual.
15CFILT-IITBINDIC21kn-en2021/05/04 02:00:496131---24.18------NMTNoMultilingual NMT (Many to One): Transformer based model with shared encoder-decoder and shared BPE vocabulary trained using all Dravidian languages data converted to same script
16CFILT-IITBINDIC21kn-en2021/05/04 01:55:246121---24.01------NMTNoMultilingual NMT (Many to One): Transformer based model with shared encoder-decoder and shared BPE vocabulary trained using all indic language data converted to same script
17coastalINDIC21kn-en2021/05/04 01:44:396101---21.60------NMTNoseq2seq model trained on all WAT2021 data
18ORGANIZERINDIC21kn-en2021/04/08 17:22:334795---20.33------NMTNoBilingual baseline trained on PMI data. Transformer base. LR=10-3
19NLPHutINDIC21kn-en2021/03/19 16:22:464593---17.72------NMTNoTransformer with source language and target language tags trained using all languages PMI data. Then fine-tuned using kn-en PMI data.
20gaurvarINDIC21kn-en2021/04/25 18:32:335547---13.87------NMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages
21gaurvarINDIC21kn-en2021/04/25 18:59:055568---13.86------NMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages
22gaurvarINDIC21kn-en2021/04/25 18:45:485558---13.45------NMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages
23gaurvarINDIC21kn-en2021/04/25 18:15:235537---12.98------NMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages

Notice:

Back to top

RIBES


# Team Task Date/Time DataID RIBES
Method
Other
Resources
System
Description
juman kytea mecab moses-
tokenizer
stanford-
segmenter-
ctb
stanford-
segmenter-
pku
indic-
tokenizer
unuse myseg kmseg
1SRPOLINDIC21kn-en2021/05/04 15:26:086245---0.840458------NMTNoEnsemble of many-to-one on all data. Pretrained on BT, finetuned on PMI
2SRPOLINDIC21kn-en2021/05/04 16:31:246271---0.837287------NMTNoMany-to-one on all data. Pretrained on BT, finetuned on PMI
3sakuraINDIC21kn-en2021/05/04 13:14:166205---0.820898------NMTNoPre-training multilingual mBART many2many model with training corpus followed by finetuning on PMI Parallel.
4sakuraINDIC21kn-en2021/04/30 22:45:495873---0.815837------NMTNoFine-tuning of multilingual mBART many2many model with training corpus.
5coastalINDIC21kn-en2021/05/04 05:41:596165---0.811950------NMTNomT5 trained only on PMI
6NICT-5INDIC21kn-en2021/06/25 11:48:366496---0.811862------NMTNoUsing PMI and PIB data for fine-tuning on a mbart model trained for over 5 epochs. MNMT model.
7IITP-MTINDIC21kn-en2021/05/04 17:53:286286---0.806170------NMTNoMany-to-One model trained on all training data with base Transformer. All indic language data is romanized. Model fine-tuned on BT PMI monolingual corpus.
8IIIT-HINDIC21kn-en2021/05/03 18:13:306018---0.804694------NMTNoMNMT system (XX-En) trained via exploiting lexical similarity on PMI+CVIT parallel corpus, then improved using back translation on PMI monolingual data followed by fine tuning.
9mcairtINDIC21kn-en2021/05/04 20:02:546374---0.803525------NMTNomultilingual model(many to one) trained on all WAT 2021 data by using base transformer.
10NICT-5INDIC21kn-en2021/06/21 14:32:566482---0.800615------NMTNoUsing PMI and PIB data for fine-tuning on am mbart model trained for over 5 epochs.
11SRPOLINDIC21kn-en2021/04/21 19:31:425328---0.798426------NMTNoBase transformer on all WAT21 data
12NICT-5INDIC21kn-en2021/04/22 11:51:575355---0.796119------NMTNoMBART+MNMT. Beam 4.
13NICT-5INDIC21kn-en2021/04/21 15:43:005280---0.793521------NMTNoPretrain MBART on IndicCorp and FT on bilingual PMI data. Beam search. Model is bilingual.
14CFILTINDIC21kn-en2021/05/04 01:13:126055---0.772913------NMTNoMultilingual(Many-to-One(XX-En)) NMT model based on Transformer with shared encoder and decoder.
15coastalINDIC21kn-en2021/05/04 01:44:396101---0.766410------NMTNoseq2seq model trained on all WAT2021 data
16CFILT-IITBINDIC21kn-en2021/05/04 02:00:496131---0.759045------NMTNoMultilingual NMT (Many to One): Transformer based model with shared encoder-decoder and shared BPE vocabulary trained using all Dravidian languages data converted to same script
17CFILT-IITBINDIC21kn-en2021/05/04 01:55:246121---0.758489------NMTNoMultilingual NMT (Many to One): Transformer based model with shared encoder-decoder and shared BPE vocabulary trained using all indic language data converted to same script
18ORGANIZERINDIC21kn-en2021/04/08 17:22:334795---0.717654------NMTNoBilingual baseline trained on PMI data. Transformer base. LR=10-3
19NLPHutINDIC21kn-en2021/03/19 16:22:464593---0.710551------NMTNoTransformer with source language and target language tags trained using all languages PMI data. Then fine-tuned using kn-en PMI data.
20gaurvarINDIC21kn-en2021/04/25 18:45:485558---0.683906------NMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages
21gaurvarINDIC21kn-en2021/04/25 18:59:055568---0.674282------NMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages
22gaurvarINDIC21kn-en2021/04/25 18:32:335547---0.648733------NMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages
23gaurvarINDIC21kn-en2021/04/25 18:15:235537---0.629379------NMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages

Notice:

Back to top

AMFM


# Team Task Date/Time DataID AMFM
Method
Other
Resources
System
Description
unuse unuse unuse unuse unuse unuse unuse unuse unuse unuse
1SRPOLINDIC21kn-en2021/05/04 15:26:086245---0.823730------NMTNoEnsemble of many-to-one on all data. Pretrained on BT, finetuned on PMI
2SRPOLINDIC21kn-en2021/05/04 16:31:246271---0.820355------NMTNoMany-to-one on all data. Pretrained on BT, finetuned on PMI
3sakuraINDIC21kn-en2021/05/04 13:14:166205---0.809702------NMTNoPre-training multilingual mBART many2many model with training corpus followed by finetuning on PMI Parallel.
4coastalINDIC21kn-en2021/05/04 05:41:596165---0.806951------NMTNomT5 trained only on PMI
5sakuraINDIC21kn-en2021/04/30 22:45:495873---0.805112------NMTNoFine-tuning of multilingual mBART many2many model with training corpus.
6mcairtINDIC21kn-en2021/05/04 20:02:546374---0.799216------NMTNomultilingual model(many to one) trained on all WAT 2021 data by using base transformer.
7IITP-MTINDIC21kn-en2021/05/04 17:53:286286---0.798540------NMTNoMany-to-One model trained on all training data with base Transformer. All indic language data is romanized. Model fine-tuned on BT PMI monolingual corpus.
8SRPOLINDIC21kn-en2021/04/21 19:31:425328---0.792687------NMTNoBase transformer on all WAT21 data
9NICT-5INDIC21kn-en2021/04/22 11:51:575355---0.792622------NMTNoMBART+MNMT. Beam 4.
10IIIT-HINDIC21kn-en2021/05/03 18:13:306018---0.790977------NMTNoMNMT system (XX-En) trained via exploiting lexical similarity on PMI+CVIT parallel corpus, then improved using back translation on PMI monolingual data followed by fine tuning.
11NICT-5INDIC21kn-en2021/04/21 15:43:005280---0.782087------NMTNoPretrain MBART on IndicCorp and FT on bilingual PMI data. Beam search. Model is bilingual.
12CFILTINDIC21kn-en2021/05/04 01:13:126055---0.778602------NMTNoMultilingual(Many-to-One(XX-En)) NMT model based on Transformer with shared encoder and decoder.
13coastalINDIC21kn-en2021/05/04 01:44:396101---0.773141------NMTNoseq2seq model trained on all WAT2021 data
14CFILT-IITBINDIC21kn-en2021/05/04 01:55:246121---0.751223------NMTNoMultilingual NMT (Many to One): Transformer based model with shared encoder-decoder and shared BPE vocabulary trained using all indic language data converted to same script
15CFILT-IITBINDIC21kn-en2021/05/04 02:00:496131---0.744802------NMTNoMultilingual NMT (Many to One): Transformer based model with shared encoder-decoder and shared BPE vocabulary trained using all Dravidian languages data converted to same script
16ORGANIZERINDIC21kn-en2021/04/08 17:22:334795---0.692019------NMTNoBilingual baseline trained on PMI data. Transformer base. LR=10-3
17gaurvarINDIC21kn-en2021/04/25 18:59:055568---0.687810------NMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages
18gaurvarINDIC21kn-en2021/04/25 18:45:485558---0.687726------NMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages
19gaurvarINDIC21kn-en2021/04/25 18:32:335547---0.686757------NMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages
20NLPHutINDIC21kn-en2021/03/19 16:22:464593---0.679617------NMTNoTransformer with source language and target language tags trained using all languages PMI data. Then fine-tuned using kn-en PMI data.
21gaurvarINDIC21kn-en2021/04/25 18:15:235537---0.675607------NMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages
22NICT-5INDIC21kn-en2021/06/21 14:32:566482---0.000000------NMTNoUsing PMI and PIB data for fine-tuning on am mbart model trained for over 5 epochs.
23NICT-5INDIC21kn-en2021/06/25 11:48:366496---0.000000------NMTNoUsing PMI and PIB data for fine-tuning on a mbart model trained for over 5 epochs. MNMT model.

Notice:

Back to top

HUMAN (WAT2022)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description

Notice:
Back to top

HUMAN (WAT2021)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description
1NLPHutINDIC21kn-en2021/03/19 16:22:464593UnderwayNMTNoTransformer with source language and target language tags trained using all languages PMI data. Then fine-tuned using kn-en PMI data.
2NICT-5INDIC21kn-en2021/04/21 15:43:005280UnderwayNMTNoPretrain MBART on IndicCorp and FT on bilingual PMI data. Beam search. Model is bilingual.
3NICT-5INDIC21kn-en2021/04/22 11:51:575355UnderwayNMTNoMBART+MNMT. Beam 4.
4gaurvarINDIC21kn-en2021/04/25 18:45:485558UnderwayNMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages
5gaurvarINDIC21kn-en2021/04/25 18:59:055568UnderwayNMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages
6sakuraINDIC21kn-en2021/04/30 22:45:495873UnderwayNMTNoFine-tuning of multilingual mBART many2many model with training corpus.
7IIIT-HINDIC21kn-en2021/05/03 18:13:306018UnderwayNMTNoMNMT system (XX-En) trained via exploiting lexical similarity on PMI+CVIT parallel corpus, then improved using back translation on PMI monolingual data followed by fine tuning.
8CFILTINDIC21kn-en2021/05/04 01:13:126055UnderwayNMTNoMultilingual(Many-to-One(XX-En)) NMT model based on Transformer with shared encoder and decoder.
9CFILT-IITBINDIC21kn-en2021/05/04 01:55:246121UnderwayNMTNoMultilingual NMT (Many to One): Transformer based model with shared encoder-decoder and shared BPE vocabulary trained using all indic language data converted to same script
10CFILT-IITBINDIC21kn-en2021/05/04 02:00:496131UnderwayNMTNoMultilingual NMT (Many to One): Transformer based model with shared encoder-decoder and shared BPE vocabulary trained using all Dravidian languages data converted to same script
11coastalINDIC21kn-en2021/05/04 05:41:596165UnderwayNMTNomT5 trained only on PMI
12SRPOLINDIC21kn-en2021/05/04 15:26:086245UnderwayNMTNoEnsemble of many-to-one on all data. Pretrained on BT, finetuned on PMI
13SRPOLINDIC21kn-en2021/05/04 16:31:246271UnderwayNMTNoMany-to-one on all data. Pretrained on BT, finetuned on PMI
14IITP-MTINDIC21kn-en2021/05/04 17:53:286286UnderwayNMTNoMany-to-One model trained on all training data with base Transformer. All indic language data is romanized. Model fine-tuned on BT PMI monolingual corpus.
15mcairtINDIC21kn-en2021/05/04 20:02:546374UnderwayNMTNomultilingual model(many to one) trained on all WAT 2021 data by using base transformer.

Notice:
Back to top

HUMAN (WAT2020)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description

Notice:
Back to top

HUMAN (WAT2019)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description

Notice:
Back to top

HUMAN (WAT2018)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description

Notice:
Back to top

HUMAN (WAT2017)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description

Notice:
Back to top

HUMAN (WAT2016)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description

Notice:
Back to top

HUMAN (WAT2015)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description

Notice:
Back to top

HUMAN (WAT2014)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description

Notice:
Back to top

EVALUATION RESULTS USAGE POLICY

When you use the WAT evaluation results for any purpose such as:
- writing technical papers,
- making presentations about your system,
- advertising your MT system to the customers,
you can use the information about translation directions, scores (including both automatic and human evaluations) and ranks of your system among others. You can also use the scores of the other systems, but you MUST anonymize the other system's names. In addition, you can show the links (URLs) to the WAT evaluation result pages.

NICT (National Institute of Information and Communications Technology)
Kyoto University
Last Modified: 2018-08-02