NICT_LOGO.JPG KYOTO-U_LOGO.JPG

WAT

The Workshop on Asian Translation
Evaluation Results

[EVALUATION RESULTS TOP] | [BLEU] | [RIBES] | [AMFM] | [HUMAN (WAT2022)] | [HUMAN (WAT2021)] | [HUMAN (WAT2020)] | [HUMAN (WAT2019)] | [HUMAN (WAT2018)] | [HUMAN (WAT2017)] | [HUMAN (WAT2016)] | [HUMAN (WAT2015)] | [HUMAN (WAT2014)] | [EVALUATION RESULTS USAGE POLICY]

BLEU


# Team Task Date/Time DataID BLEU
Method
 Other
Resources
 System
Description
juman kytea mecab moses-
tokenizer		 stanford-
segmenter-
ctb		 stanford-
segmenter-
pku		 indic-
tokenizer		 unuse myseg kmseg
1ORGANIZERMMEVTEXT24en-hi2020/08/27 21:02:123586------39.78---NMTNoTransformer base model
2ODIANLPMMEVTEXT24en-hi2020/09/14 18:52:373711------40.85---NMTYesTransformer model (used IITB as an additional resource for training)
3CNLP-NITSMMEVTEXT24en-hi2020/09/18 16:08:143897------38.84---NMTYesPretrained monolingual data (IITB) and fine-tuned with parallel data in training using BRNN model.
4MMEVTEXT24en-hi2020/09/19 04:50:414030------33.83---NMTNoFor the text-only Eng-Hindi translation task, we use an adaptation of the NMT-Keras code to suit our problem. Here, the focus is on long term translation as well as active learning strategies. The t
5CNLP-NITS-PPMMEVTEXT24en-hi2021/04/28 00:20:195733------37.01---NMTYesPretrained monolingual data (IITB) and fine-tuned with parallel data (WAT21 train data+ Extracted Phrase pairs from WAT21 train data +IITB train data) in training using BRNN model.
6VoltaMMEVTEXT24en-hi2021/05/25 13:55:196427------44.12---NMTYesFinetuned mBART (Used IITB corpus for data augmentation)
7nlp_novicesMMEVTEXT24en-hi2022/07/10 23:57:476733------43.10---NMTYesFinetuned Transformers over OPUS Corpora additionally
8CNLP-NITS-PPMMEVTEXT24en-hi2022/07/11 12:24:456739------37.00---NMTNoTransliteration-based phrase pairs augmentation in training using BRNN-based NMT
9SILO_NLPMMEVTEXT24en-hi2022/07/12 04:13:506836------36.20---NMTNoFine-tuning with pre-trained mBART-50 model
10ODIAGENMMEVTEXT24en-hi2023/07/03 02:52:227087------44.60---NMTNoFine-tuning Transformer using NLLB-200 from Facebook
11ODIAGENMMEVTEXT24en-hi2023/07/06 12:27:297109------44.60---NMTNoFine-tuning Transformer using NLLB-200 from Facebook
1200-7MMEVTEXT24en-hi2024/08/11 13:13:287322------43.30---NMTYes
13ODIAGENMMEVTEXT24en-hi2024/08/11 18:14:567335------41.60---OtherNoLLM based (Mistral-7B fine-tuning)
14DCU_NMTMMEVTEXT24en-hi2024/08/11 22:54:117347------40.20---NMTNoThe baseline model trained in English-Hindi direction using Fairseq NMT for text only.
15DCU_NMTMMEVTEXT24en-hi2024/08/11 22:55:417348------42.70---NMTYesThe En-Hi system trained using additional 8k data extracted from Flicker.

Notice:

Back to top

RIBES


# Team Task Date/Time DataID RIBES
Method
 Other
Resources
 System
Description
juman kytea mecab moses-
tokenizer		 stanford-
segmenter-
ctb		 stanford-
segmenter-
pku		 indic-
tokenizer		 unuse myseg kmseg
1ORGANIZERMMEVTEXT24en-hi2020/08/27 21:02:123586------0.776892---NMTNoTransformer base model
2ODIANLPMMEVTEXT24en-hi2020/09/14 18:52:373711------0.790908---NMTYesTransformer model (used IITB as an additional resource for training)
3CNLP-NITSMMEVTEXT24en-hi2020/09/18 16:08:143897------0.793416---NMTYesPretrained monolingual data (IITB) and fine-tuned with parallel data in training using BRNN model.
4MMEVTEXT24en-hi2020/09/19 04:50:414030------0.753529---NMTNoFor the text-only Eng-Hindi translation task, we use an adaptation of the NMT-Keras code to suit our problem. Here, the focus is on long term translation as well as active learning strategies. The t
5CNLP-NITS-PPMMEVTEXT24en-hi2021/04/28 00:20:195733------0.795302---NMTYesPretrained monolingual data (IITB) and fine-tuned with parallel data (WAT21 train data+ Extracted Phrase pairs from WAT21 train data +IITB train data) in training using BRNN model.
6VoltaMMEVTEXT24en-hi2021/05/25 13:55:196427------0.821469---NMTYesFinetuned mBART (Used IITB corpus for data augmentation)
7nlp_novicesMMEVTEXT24en-hi2022/07/10 23:57:476733------0.816860---NMTYesFinetuned Transformers over OPUS Corpora additionally
8CNLP-NITS-PPMMEVTEXT24en-hi2022/07/11 12:24:456739------0.795302---NMTNoTransliteration-based phrase pairs augmentation in training using BRNN-based NMT
9SILO_NLPMMEVTEXT24en-hi2022/07/12 04:13:506836------0.785673---NMTNoFine-tuning with pre-trained mBART-50 model
10ODIAGENMMEVTEXT24en-hi2023/07/03 02:52:227087------0.829217---NMTNoFine-tuning Transformer using NLLB-200 from Facebook
11ODIAGENMMEVTEXT24en-hi2023/07/06 12:27:297109------0.829213---NMTNoFine-tuning Transformer using NLLB-200 from Facebook
1200-7MMEVTEXT24en-hi2024/08/11 13:13:287322------0.812578---NMTYes
13ODIAGENMMEVTEXT24en-hi2024/08/11 18:14:567335------0.821154---OtherNoLLM based (Mistral-7B fine-tuning)
14DCU_NMTMMEVTEXT24en-hi2024/08/11 22:54:117347------0.798470---NMTNoThe baseline model trained in English-Hindi direction using Fairseq NMT for text only.
15DCU_NMTMMEVTEXT24en-hi2024/08/11 22:55:417348------0.817949---NMTYesThe En-Hi system trained using additional 8k data extracted from Flicker.

Notice:

Back to top

AMFM


# Team Task Date/Time DataID AMFM
Method
 Other
Resources
 System
Description
unuse unuse unuse unuse unuse unuse unuse unuse unuse unuse
1ORGANIZERMMEVTEXT24en-hi2020/08/27 21:02:123586------0.796570---NMTNoTransformer base model
2ODIANLPMMEVTEXT24en-hi2020/09/14 18:52:373711------0.808340---NMTYesTransformer model (used IITB as an additional resource for training)
3CNLP-NITSMMEVTEXT24en-hi2020/09/18 16:08:143897------0.804250---NMTYesPretrained monolingual data (IITB) and fine-tuned with parallel data in training using BRNN model.
4MMEVTEXT24en-hi2020/09/19 04:50:414030------0.767900---NMTNoFor the text-only Eng-Hindi translation task, we use an adaptation of the NMT-Keras code to suit our problem. Here, the focus is on long term translation as well as active learning strategies. The t
5CNLP-NITS-PPMMEVTEXT24en-hi2021/04/28 00:20:195733------0.642785---NMTYesPretrained monolingual data (IITB) and fine-tuned with parallel data (WAT21 train data+ Extracted Phrase pairs from WAT21 train data +IITB train data) in training using BRNN model.
6VoltaMMEVTEXT24en-hi2021/05/25 13:55:196427------0.838180---NMTYesFinetuned mBART (Used IITB corpus for data augmentation)
7nlp_novicesMMEVTEXT24en-hi2022/07/10 23:57:476733------0.000000---NMTYesFinetuned Transformers over OPUS Corpora additionally
8CNLP-NITS-PPMMEVTEXT24en-hi2022/07/11 12:24:456739------0.000000---NMTNoTransliteration-based phrase pairs augmentation in training using BRNN-based NMT
9SILO_NLPMMEVTEXT24en-hi2022/07/12 04:13:506836------0.000000---NMTNoFine-tuning with pre-trained mBART-50 model
10ODIAGENMMEVTEXT24en-hi2023/07/03 02:52:227087------0.000000---NMTNoFine-tuning Transformer using NLLB-200 from Facebook
11ODIAGENMMEVTEXT24en-hi2023/07/06 12:27:297109------0.000000---NMTNoFine-tuning Transformer using NLLB-200 from Facebook
1200-7MMEVTEXT24en-hi2024/08/11 13:13:287322------0.000000---NMTYes
13ODIAGENMMEVTEXT24en-hi2024/08/11 18:14:567335------0.000000---OtherNoLLM based (Mistral-7B fine-tuning)
14DCU_NMTMMEVTEXT24en-hi2024/08/11 22:54:117347------0.000000---NMTNoThe baseline model trained in English-Hindi direction using Fairseq NMT for text only.
15DCU_NMTMMEVTEXT24en-hi2024/08/11 22:55:417348------0.000000---NMTYesThe En-Hi system trained using additional 8k data extracted from Flicker.

Notice:

Back to top

HUMAN (WAT2022)


# Team Task Date/Time DataID HUMAN
Method
 Other
Resources
System
Description
1nlp_novicesMMEVTEXT24en-hi2022/07/10 23:57:476733UnderwayNMTYesFinetuned Transformers over OPUS Corpora additionally
2CNLP-NITS-PPMMEVTEXT24en-hi2022/07/11 12:24:456739UnderwayNMTNoTransliteration-based phrase pairs augmentation in training using BRNN-based NMT
3SILO_NLPMMEVTEXT24en-hi2022/07/12 04:13:506836UnderwayNMTNoFine-tuning with pre-trained mBART-50 model

Notice:
Back to top

HUMAN (WAT2021)


# Team Task Date/Time DataID HUMAN
Method
 Other
Resources
System
Description
1CNLP-NITS-PPMMEVTEXT24en-hi2021/04/28 00:20:195733UnderwayNMTYesPretrained monolingual data (IITB) and fine-tuned with parallel data (WAT21 train data+ Extracted Phrase pairs from WAT21 train data +IITB train data) in training using BRNN model.

Notice:
Back to top

HUMAN (WAT2020)


# Team Task Date/Time DataID HUMAN
Method
 Other
Resources
System
Description
1ODIANLPMMEVTEXT24en-hi2020/09/14 18:52:373711UnderwayNMTYesTransformer model (used IITB as an additional resource for training)
2CNLP-NITSMMEVTEXT24en-hi2020/09/18 16:08:143897UnderwayNMTYesPretrained monolingual data (IITB) and fine-tuned with parallel data in training using BRNN model.
3MMEVTEXT24en-hi2020/09/19 04:50:414030UnderwayNMTNoFor the text-only Eng-Hindi translation task, we use an adaptation of the NMT-Keras code to suit our problem. Here, the focus is on long term translation as well as active learning strategies. The t

Notice:
Back to top

HUMAN (WAT2019)


# Team Task Date/Time DataID HUMAN
Method
 Other
Resources
System
Description

Notice:
Back to top

HUMAN (WAT2018)


# Team Task Date/Time DataID HUMAN
Method
 Other
Resources
System
Description

Notice:
Back to top

HUMAN (WAT2017)


# Team Task Date/Time DataID HUMAN
Method
 Other
Resources
System
Description

Notice:
Back to top

HUMAN (WAT2016)


# Team Task Date/Time DataID HUMAN
Method
 Other
Resources
System
Description

Notice:
Back to top

HUMAN (WAT2015)


# Team Task Date/Time DataID HUMAN
Method
 Other
Resources
System
Description

Notice:
Back to top

HUMAN (WAT2014)


# Team Task Date/Time DataID HUMAN
Method
 Other
Resources
System
Description

Notice:
Back to top

EVALUATION RESULTS USAGE POLICY

When you use the WAT evaluation results for any purpose such as:
- writing technical papers,
- making presentations about your system,
- advertising your MT system to the customers,
you can use the information about translation directions, scores (including both automatic and human evaluations) and ranks of your system among others. You can also use the scores of the other systems, but you MUST anonymize the other system's names. In addition, you can show the links (URLs) to the WAT evaluation result pages.

NICT (National Institute of Information and Communications Technology)
Kyoto University
Last Modified: 2018-08-02