NICT_LOGO.JPG KYOTO-U_LOGO.JPG

WAT

The Workshop on Asian Translation
Evaluation Results

[EVALUATION RESULTS TOP] | [BLEU] | [RIBES] | [AMFM] | [HUMAN (WAT2022)] | [HUMAN (WAT2021)] | [HUMAN (WAT2020)] | [HUMAN (WAT2019)] | [HUMAN (WAT2018)] | [HUMAN (WAT2017)] | [HUMAN (WAT2016)] | [HUMAN (WAT2015)] | [HUMAN (WAT2014)] | [EVALUATION RESULTS USAGE POLICY]

BLEU


# Team Task Date/Time DataID BLEU
Method
Other
Resources
System
Description
juman kytea mecab moses-
tokenizer
stanford-
segmenter-
ctb
stanford-
segmenter-
pku
indic-
tokenizer
unuse myseg kmseg
1SRPOLINDIC21ta-en2021/05/04 15:28:466250---36.13------NMTNoEnsemble of many-to-one on all data. Pretrained on BT, finetuned on PMI
2SRPOLINDIC21ta-en2021/05/04 16:33:256276---35.06------NMTNoMany-to-one on all data. Pretrained on BT, finetuned on PMI
3sakuraINDIC21ta-en2021/05/04 13:21:246210---31.94------NMTNoPre-training multilingual mBART many2many model with training corpus followed by finetuning on PMI Parallel.
4NICT-5INDIC21ta-en2021/06/25 11:50:446501---31.60------NMTNoUsing PMI and PIB data for fine-tuning on a mbart model trained for over 5 epochs. MNMT model.
5sakuraINDIC21ta-en2021/04/30 23:00:525878---31.09------NMTNoFine-tuning of multilingual mBART many2many model with training corpus.
6NICT-5INDIC21ta-en2021/06/21 12:05:266480---30.29------NMTNoUsing PMI and PIB data for fine-tuning on am mbart model trained for over 5 epochs.
7IIIT-HINDIC21ta-en2021/05/03 18:16:016024---29.61------NMTNoMNMT system (XX-En) trained via exploiting lexical similarity on PMI+CVIT parallel corpus, then improved using back translation on PMI monolingual data followed by fine tuning.
8CFILTINDIC21ta-en2021/05/04 01:18:096060---29.34------NMTNoMultilingual(Many-to-One(XX-En)) NMT model based on Transformer with shared encoder and decoder.
9mcairtINDIC21ta-en2021/05/04 19:35:586346---28.04------NMTNomultilingual model(many to one) trained on all WAT 2021 data by using base transformer.
10SRPOLINDIC21ta-en2021/04/21 19:33:545333---28.01------NMTNoBase transformer on all WAT21 data
11IITP-MTINDIC21ta-en2021/05/04 18:16:186304---27.76------NMTNoMany-to-One model trained on all training data with base Transformer. All indic language data is romanized. Model fine-tuned on BT PMI monolingual corpus.
12NICT-5INDIC21ta-en2021/04/22 11:54:115365---26.90------NMTNoMBART+MNMT. Beam 4.
13coastalINDIC21ta-en2021/05/04 05:43:546169---26.69------NMTNomT5 trained only on PMI
14NICT-5INDIC21ta-en2021/04/21 15:45:595290---24.72------NMTNoPretrain MBART on IndicCorp and FT on bilingual PMI data. Beam search. Model is bilingual.
15CFILT-IITBINDIC21ta-en2021/05/04 02:01:236132---22.75------NMTNoMultilingual NMT (Many to One): Transformer based model with shared encoder-decoder and shared BPE vocabulary trained using all Dravidian languages data converted to same script
16CFILT-IITBINDIC21ta-en2021/05/04 01:55:456122---21.37------NMTNoMultilingual NMT (Many to One): Transformer based model with shared encoder-decoder and shared BPE vocabulary trained using all indic language data converted to same script
17coastalINDIC21ta-en2021/05/04 01:47:066110---19.04------NMTNoseq2seq model trained on all WAT2021 data
18ORGANIZERINDIC21ta-en2021/04/08 17:25:524805---16.07------NMTNoBilingual baseline trained on PMI data. Transformer base. LR=10-3
19NLPHutINDIC21ta-en2021/05/03 00:13:565984---15.40------NMTNoTransformer with source and target language tags trained using all languages PMI data. Then fine tuned using all ta-en data.
20NLPHutINDIC21ta-en2021/03/20 00:18:364617---14.77------NMTNoTransformer with source and target language tags trained using all languages PMI data. Then fine tuned using ta-en PMI data.
21gaurvarINDIC21ta-en2021/04/25 19:03:525573---13.77------NMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages
22gaurvarINDIC21ta-en2021/04/25 18:49:585563---13.36------NMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages
23gaurvarINDIC21ta-en2021/04/25 18:36:455552---12.47------NMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages
24gaurvarINDIC21ta-en2021/04/25 18:17:095539---11.39------NMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages

Notice:

Back to top

RIBES


# Team Task Date/Time DataID RIBES
Method
Other
Resources
System
Description
juman kytea mecab moses-
tokenizer
stanford-
segmenter-
ctb
stanford-
segmenter-
pku
indic-
tokenizer
unuse myseg kmseg
1SRPOLINDIC21ta-en2021/05/04 15:28:466250---0.822312------NMTNoEnsemble of many-to-one on all data. Pretrained on BT, finetuned on PMI
2SRPOLINDIC21ta-en2021/05/04 16:33:256276---0.815951------NMTNoMany-to-one on all data. Pretrained on BT, finetuned on PMI
3sakuraINDIC21ta-en2021/04/30 23:00:525878---0.806993------NMTNoFine-tuning of multilingual mBART many2many model with training corpus.
4sakuraINDIC21ta-en2021/05/04 13:21:246210---0.806968------NMTNoPre-training multilingual mBART many2many model with training corpus followed by finetuning on PMI Parallel.
5NICT-5INDIC21ta-en2021/06/25 11:50:446501---0.803273------NMTNoUsing PMI and PIB data for fine-tuning on a mbart model trained for over 5 epochs. MNMT model.
6NICT-5INDIC21ta-en2021/06/21 12:05:266480---0.797120------NMTNoUsing PMI and PIB data for fine-tuning on am mbart model trained for over 5 epochs.
7coastalINDIC21ta-en2021/05/04 05:43:546169---0.794380------NMTNomT5 trained only on PMI
8mcairtINDIC21ta-en2021/05/04 19:35:586346---0.793839------NMTNomultilingual model(many to one) trained on all WAT 2021 data by using base transformer.
9IITP-MTINDIC21ta-en2021/05/04 18:16:186304---0.788181------NMTNoMany-to-One model trained on all training data with base Transformer. All indic language data is romanized. Model fine-tuned on BT PMI monolingual corpus.
10SRPOLINDIC21ta-en2021/04/21 19:33:545333---0.788057------NMTNoBase transformer on all WAT21 data
11IIIT-HINDIC21ta-en2021/05/03 18:16:016024---0.785332------NMTNoMNMT system (XX-En) trained via exploiting lexical similarity on PMI+CVIT parallel corpus, then improved using back translation on PMI monolingual data followed by fine tuning.
12CFILTINDIC21ta-en2021/05/04 01:18:096060---0.784291------NMTNoMultilingual(Many-to-One(XX-En)) NMT model based on Transformer with shared encoder and decoder.
13NICT-5INDIC21ta-en2021/04/22 11:54:115365---0.780120------NMTNoMBART+MNMT. Beam 4.
14NICT-5INDIC21ta-en2021/04/21 15:45:595290---0.766631------NMTNoPretrain MBART on IndicCorp and FT on bilingual PMI data. Beam search. Model is bilingual.
15CFILT-IITBINDIC21ta-en2021/05/04 02:01:236132---0.756364------NMTNoMultilingual NMT (Many to One): Transformer based model with shared encoder-decoder and shared BPE vocabulary trained using all Dravidian languages data converted to same script
16coastalINDIC21ta-en2021/05/04 01:47:066110---0.748475------NMTNoseq2seq model trained on all WAT2021 data
17CFILT-IITBINDIC21ta-en2021/05/04 01:55:456122---0.747748------NMTNoMultilingual NMT (Many to One): Transformer based model with shared encoder-decoder and shared BPE vocabulary trained using all indic language data converted to same script
18NLPHutINDIC21ta-en2021/05/03 00:13:565984---0.702428------NMTNoTransformer with source and target language tags trained using all languages PMI data. Then fine tuned using all ta-en data.
19ORGANIZERINDIC21ta-en2021/04/08 17:25:524805---0.690144------NMTNoBilingual baseline trained on PMI data. Transformer base. LR=10-3
20NLPHutINDIC21ta-en2021/03/20 00:18:364617---0.679740------NMTNoTransformer with source and target language tags trained using all languages PMI data. Then fine tuned using ta-en PMI data.
21gaurvarINDIC21ta-en2021/04/25 18:49:585563---0.677433------NMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages
22gaurvarINDIC21ta-en2021/04/25 19:03:525573---0.660037------NMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages
23gaurvarINDIC21ta-en2021/04/25 18:36:455552---0.604598------NMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages
24gaurvarINDIC21ta-en2021/04/25 18:17:095539---0.581790------NMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages

Notice:

Back to top

AMFM


# Team Task Date/Time DataID AMFM
Method
Other
Resources
System
Description
unuse unuse unuse unuse unuse unuse unuse unuse unuse unuse
1SRPOLINDIC21ta-en2021/05/04 15:28:466250---0.806540------NMTNoEnsemble of many-to-one on all data. Pretrained on BT, finetuned on PMI
2SRPOLINDIC21ta-en2021/05/04 16:33:256276---0.803595------NMTNoMany-to-one on all data. Pretrained on BT, finetuned on PMI
3sakuraINDIC21ta-en2021/04/30 23:00:525878---0.796074------NMTNoFine-tuning of multilingual mBART many2many model with training corpus.
4sakuraINDIC21ta-en2021/05/04 13:21:246210---0.790353------NMTNoPre-training multilingual mBART many2many model with training corpus followed by finetuning on PMI Parallel.
5mcairtINDIC21ta-en2021/05/04 19:35:586346---0.790184------NMTNomultilingual model(many to one) trained on all WAT 2021 data by using base transformer.
6SRPOLINDIC21ta-en2021/04/21 19:33:545333---0.788439------NMTNoBase transformer on all WAT21 data
7IITP-MTINDIC21ta-en2021/05/04 18:16:186304---0.786587------NMTNoMany-to-One model trained on all training data with base Transformer. All indic language data is romanized. Model fine-tuned on BT PMI monolingual corpus.
8coastalINDIC21ta-en2021/05/04 05:43:546169---0.786098------NMTNomT5 trained only on PMI
9CFILTINDIC21ta-en2021/05/04 01:18:096060---0.785098------NMTNoMultilingual(Many-to-One(XX-En)) NMT model based on Transformer with shared encoder and decoder.
10NICT-5INDIC21ta-en2021/04/22 11:54:115365---0.772249------NMTNoMBART+MNMT. Beam 4.
11coastalINDIC21ta-en2021/05/04 01:47:066110---0.764269------NMTNoseq2seq model trained on all WAT2021 data
12NICT-5INDIC21ta-en2021/04/21 15:45:595290---0.758282------NMTNoPretrain MBART on IndicCorp and FT on bilingual PMI data. Beam search. Model is bilingual.
13IIIT-HINDIC21ta-en2021/05/03 18:16:016024---0.750297------NMTNoMNMT system (XX-En) trained via exploiting lexical similarity on PMI+CVIT parallel corpus, then improved using back translation on PMI monolingual data followed by fine tuning.
14CFILT-IITBINDIC21ta-en2021/05/04 02:01:236132---0.745090------NMTNoMultilingual NMT (Many to One): Transformer based model with shared encoder-decoder and shared BPE vocabulary trained using all Dravidian languages data converted to same script
15CFILT-IITBINDIC21ta-en2021/05/04 01:55:456122---0.742311------NMTNoMultilingual NMT (Many to One): Transformer based model with shared encoder-decoder and shared BPE vocabulary trained using all indic language data converted to same script
16gaurvarINDIC21ta-en2021/04/25 19:03:525573---0.688325------NMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages
17gaurvarINDIC21ta-en2021/04/25 18:49:585563---0.687892------NMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages
18ORGANIZERINDIC21ta-en2021/04/08 17:25:524805---0.675969------NMTNoBilingual baseline trained on PMI data. Transformer base. LR=10-3
19gaurvarINDIC21ta-en2021/04/25 18:36:455552---0.674214------NMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages
20NLPHutINDIC21ta-en2021/05/03 00:13:565984---0.669984------NMTNoTransformer with source and target language tags trained using all languages PMI data. Then fine tuned using all ta-en data.
21NLPHutINDIC21ta-en2021/03/20 00:18:364617---0.663932------NMTNoTransformer with source and target language tags trained using all languages PMI data. Then fine tuned using ta-en PMI data.
22gaurvarINDIC21ta-en2021/04/25 18:17:095539---0.663642------NMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages
23NICT-5INDIC21ta-en2021/06/21 12:05:266480---0.000000------NMTNoUsing PMI and PIB data for fine-tuning on am mbart model trained for over 5 epochs.
24NICT-5INDIC21ta-en2021/06/25 11:50:446501---0.000000------NMTNoUsing PMI and PIB data for fine-tuning on a mbart model trained for over 5 epochs. MNMT model.

Notice:

Back to top

HUMAN (WAT2022)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description

Notice:
Back to top

HUMAN (WAT2021)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description
1NICT-5INDIC21ta-en2021/04/21 15:45:595290UnderwayNMTNoPretrain MBART on IndicCorp and FT on bilingual PMI data. Beam search. Model is bilingual.
2NICT-5INDIC21ta-en2021/04/22 11:54:115365UnderwayNMTNoMBART+MNMT. Beam 4.
3gaurvarINDIC21ta-en2021/04/25 18:49:585563UnderwayNMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages
4gaurvarINDIC21ta-en2021/04/25 19:03:525573UnderwayNMTNoMulti Task Multi Lingual T5 trained for Multiple Indic Languages
5sakuraINDIC21ta-en2021/04/30 23:00:525878UnderwayNMTNoFine-tuning of multilingual mBART many2many model with training corpus.
6NLPHutINDIC21ta-en2021/05/03 00:13:565984UnderwayNMTNoTransformer with source and target language tags trained using all languages PMI data. Then fine tuned using all ta-en data.
7IIIT-HINDIC21ta-en2021/05/03 18:16:016024UnderwayNMTNoMNMT system (XX-En) trained via exploiting lexical similarity on PMI+CVIT parallel corpus, then improved using back translation on PMI monolingual data followed by fine tuning.
8CFILTINDIC21ta-en2021/05/04 01:18:096060UnderwayNMTNoMultilingual(Many-to-One(XX-En)) NMT model based on Transformer with shared encoder and decoder.
9CFILT-IITBINDIC21ta-en2021/05/04 01:55:456122UnderwayNMTNoMultilingual NMT (Many to One): Transformer based model with shared encoder-decoder and shared BPE vocabulary trained using all indic language data converted to same script
10CFILT-IITBINDIC21ta-en2021/05/04 02:01:236132UnderwayNMTNoMultilingual NMT (Many to One): Transformer based model with shared encoder-decoder and shared BPE vocabulary trained using all Dravidian languages data converted to same script
11coastalINDIC21ta-en2021/05/04 05:43:546169UnderwayNMTNomT5 trained only on PMI
12SRPOLINDIC21ta-en2021/05/04 15:28:466250UnderwayNMTNoEnsemble of many-to-one on all data. Pretrained on BT, finetuned on PMI
13SRPOLINDIC21ta-en2021/05/04 16:33:256276UnderwayNMTNoMany-to-one on all data. Pretrained on BT, finetuned on PMI
14IITP-MTINDIC21ta-en2021/05/04 18:16:186304UnderwayNMTNoMany-to-One model trained on all training data with base Transformer. All indic language data is romanized. Model fine-tuned on BT PMI monolingual corpus.
15mcairtINDIC21ta-en2021/05/04 19:35:586346UnderwayNMTNomultilingual model(many to one) trained on all WAT 2021 data by using base transformer.

Notice:
Back to top

HUMAN (WAT2020)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description

Notice:
Back to top

HUMAN (WAT2019)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description

Notice:
Back to top

HUMAN (WAT2018)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description

Notice:
Back to top

HUMAN (WAT2017)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description

Notice:
Back to top

HUMAN (WAT2016)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description

Notice:
Back to top

HUMAN (WAT2015)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description

Notice:
Back to top

HUMAN (WAT2014)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description

Notice:
Back to top

EVALUATION RESULTS USAGE POLICY

When you use the WAT evaluation results for any purpose such as:
- writing technical papers,
- making presentations about your system,
- advertising your MT system to the customers,
you can use the information about translation directions, scores (including both automatic and human evaluations) and ranks of your system among others. You can also use the scores of the other systems, but you MUST anonymize the other system's names. In addition, you can show the links (URLs) to the WAT evaluation result pages.

NICT (National Institute of Information and Communications Technology)
Kyoto University
Last Modified: 2018-08-02