WAT

The Workshop on Asian Translation

Evaluation Results

[HOME] | [EVALUATION RESULTS] | [AUTOMATIC EVALUATION PROCEDURES] | [EVALUATION RESULTS USAGE POLICY] | [TECHNICAL COLLABORATOR]

EVALUATION RESULTS

+ Document-level translation tasks:
[ASPEC]
Task: ASPECen-ja	Evaluation Result
Task: ASPECja-en	Evaluation Result
Task: ASPECzh-ja	Evaluation Result
Task: ASPECja-zh	Evaluation Result
[BSD Corpus (Document-level Business Scene Dialogue Translation)]**
Task: BSDen-ja	Evaluation Result
Task: BSDja-en	Evaluation Result
[JIJI Corpus]
Task: JIJIen-ja	Evaluation Result
Task: JIJIja-en	Evaluation Result
Task: JIJICen-ja	Evaluation Result
Task: JIJICja-en	Evaluation Result
[NICT-SAP (Multilingual Multi-domain evaluation task: Unstructured document translation (IT tasks 2022))]**
Task: SOFTWARE22en-hi	Evaluation Result
Task: SOFTWARE22hi-en	Evaluation Result
Task: SOFTWARE22en-id	Evaluation Result
Task: SOFTWARE22id-en	Evaluation Result
Task: SOFTWARE22en-ja	Evaluation Result
Task: SOFTWARE22ja-en	Evaluation Result
Task: SOFTWARE22en-ko	Evaluation Result
Task: SOFTWARE22ko-en	Evaluation Result
Task: SOFTWARE22en-ms	Evaluation Result
Task: SOFTWARE22ms-en	Evaluation Result
Task: SOFTWARE22en-th	Evaluation Result
Task: SOFTWARE22th-en	Evaluation Result
Task: SOFTWARE22en-vi	Evaluation Result
Task: SOFTWARE22vi-en	Evaluation Result
Task: SOFTWARE22en-zf	Evaluation Result
Task: SOFTWARE22zf-en	Evaluation Result
Task: SOFTWARE22en-zh	Evaluation Result
Task: SOFTWARE22zh-en	Evaluation Result
[NICT-SAP (Multilingual Multi-domain evaluation task: Structured document translation (IT tasks 2022))] *
Task: SWSTR22en-ja	Evaluation Result
Task: SWSTR22ja-en	Evaluation Result
Task: SWSTR22en-ko	Evaluation Result
Task: SWSTR22ko-en	Evaluation Result
Task: SWSTR22en-zf	Evaluation Result
Task: SWSTR22zf-en	Evaluation Result
Task: SWSTR22en-zh	Evaluation Result
Task: SWSTR22zh-en	Evaluation Result
[NICT-SAP (Multilingual Multi-domain evaluation task (IT tasks 2021))]
Task: SOFTWAREen-hi	Evaluation Result
Task: SOFTWAREhi-en	Evaluation Result
Task: SOFTWAREen-id	Evaluation Result
Task: SOFTWAREid-en	Evaluation Result
Task: SOFTWAREen-ms	Evaluation Result
Task: SOFTWAREms-en	Evaluation Result
Task: SOFTWAREen-th	Evaluation Result
Task: SOFTWAREth-en	Evaluation Result
[NICT-SAP (Multilingual Multi-domain evaluation task (Wikinews tasks) (Asian Language Treebank (ALT)))]**
Task: ALT20en-hi	Evaluation Result
Task: ALT20hi-en	Evaluation Result
Task: ALT20en-id	Evaluation Result
Task: ALT20id-en	Evaluation Result
Task: ALT20en-ms	Evaluation Result
Task: ALT20ms-en	Evaluation Result
Task: ALT20en-th	Evaluation Result
Task: ALT20th-en	Evaluation Result
------------------------------
+ Multimodal translation tasks:
[Video Guided MT Task for Ambiguous Subtitles (2022)]
Task: VIDEOGASja-en	Evaluation Result
[Bengali, Hindi, and Malayalam Visual Genome (2023)]**
Task: HINDENMMEVTEXT23en-bn	Evaluation Result
Task: HINDENMMEVHI23en-bn	Evaluation Result
Task: HINDENMMEVMM23en-bn	Evaluation Result
Task: HINDENMMCHTEXT23en-bn	Evaluation Result
Task: HINDENMMCHHI23en-bn	Evaluation Result
Task: HINDENMMCHMM23en-bn	Evaluation Result
Task: HINDENMMEVTEXT23en-hi	Evaluation Result
Task: HINDENMMEVHI23en-hi	Evaluation Result
Task: HINDENMMEVMM23en-hi	Evaluation Result
Task: HINDENMMCHTEXT23en-hi	Evaluation Result
Task: HINDENMMCHHI23en-hi	Evaluation Result
Task: HINDENMMCHMM23en-hi	Evaluation Result
Task: HINDENMMEVTEXT23en-ml	Evaluation Result
Task: HINDENMMEVHI23en-ml	Evaluation Result
Task: HINDENMMEVMM23en-ml	Evaluation Result
Task: HINDENMMCHTEXT23en-ml	Evaluation Result
Task: HINDENMMCHHI23en-ml	Evaluation Result
Task: HINDENMMCHMM23en-ml	Evaluation Result
[Flickr30kEnt-JP (Multimodal task)]
Task: MMTen-ja	Evaluation Result
Task: MMTja-en	Evaluation Result
[Ambiguous MS COCO]**
Task: MSCOCOen-ja	Evaluation Result
Task: MSCOCOja-en	Evaluation Result
------------------------------
+ Indic tasks:
[MultiIndicMT (Indic Languages Multilingual Parallel Corpus 2023)]**
Task: INDIC23en-as	Evaluation Result
Task: INDIC23as-en	Evaluation Result
Task: INDIC23en-bn	Evaluation Result
Task: INDIC23bn-en	Evaluation Result
Task: INDIC23en-gu	Evaluation Result
Task: INDIC23gu-en	Evaluation Result
Task: INDIC23en-hi	Evaluation Result
Task: INDIC23hi-en	Evaluation Result
Task: INDIC23en-kn	Evaluation Result
Task: INDIC23kn-en	Evaluation Result
Task: INDIC23en-ml	Evaluation Result
Task: INDIC23ml-en	Evaluation Result
Task: INDIC23en-mr	Evaluation Result
Task: INDIC23mr-en	Evaluation Result
Task: INDIC23en-ne	Evaluation Result
Task: INDIC23ne-en	Evaluation Result
Task: INDIC23en-or	Evaluation Result
Task: INDIC23or-en	Evaluation Result
Task: INDIC23en-pa	Evaluation Result
Task: INDIC23pa-en	Evaluation Result
Task: INDIC23en-sd	Evaluation Result
Task: INDIC23sd-en	Evaluation Result
Task: INDIC23en-si	Evaluation Result
Task: INDIC23si-en	Evaluation Result
Task: INDIC23en-ta	Evaluation Result
Task: INDIC23ta-en	Evaluation Result
Task: INDIC23en-te	Evaluation Result
Task: INDIC23te-en	Evaluation Result
Task: INDIC23en-ur	Evaluation Result
Task: INDIC23ur-en	Evaluation Result
Task: INDIC23en-sat	Evaluation Result
Task: INDIC23sat-en	Evaluation Result
Task: INDIC23en-ksa	Evaluation Result
Task: INDIC23ksa-en	Evaluation Result
Task: INDIC23en-ksd	Evaluation Result
Task: INDIC23ksd-en	Evaluation Result
Task: INDIC23en-mai	Evaluation Result
Task: INDIC23mai-en	Evaluation Result
------------------------------
+ ALT+ tasks:
[+UCSY]
Task: ALT2en-my	Evaluation Result
Task: ALT2my-en	Evaluation Result
------------------------------
+ Patent task:
[JPC3 (JPO Patent Corpus3 (2022 -))]* **
Task: JPC22zh-ja	Evaluation Result
Task: JPC22ja-zh	Evaluation Result
Task: JPC22ko-ja	Evaluation Result
Task: JPC22ja-ko	Evaluation Result
Task: JPC22en-ja	Evaluation Result
Task: JPC22ja-en	Evaluation Result
Task: JPCN1zh-ja	Evaluation Result
Task: JPCN1ja-zh	Evaluation Result
Task: JPCN1ko-ja	Evaluation Result
Task: JPCN1ja-ko	Evaluation Result
Task: JPCN1en-ja	Evaluation Result
Task: JPCN1ja-en	Evaluation Result
Task: JPCN2zh-ja	Evaluation Result
Task: JPCN2ja-zh	Evaluation Result
~~Task: JPCN2ko-ja~~
~~Task: JPCN2ja-ko~~
Task: JPCN2en-ja	Evaluation Result
Task: JPCN2ja-en	Evaluation Result
Task: JPCN3zh-ja	Evaluation Result
Task: JPCN3ja-zh	Evaluation Result
Task: JPCN3ko-ja	Evaluation Result
Task: JPCN3ja-ko	Evaluation Result
Task: JPCN3en-ja	Evaluation Result
Task: JPCN3ja-en	Evaluation Result
Task: JPCN4zh-ja	Evaluation Result
Task: JPCN4ja-zh	Evaluation Result
Task: JPCN4ko-ja	Evaluation Result
Task: JPCN4ja-ko	Evaluation Result
Task: JPCN4en-ja	Evaluation Result
Task: JPCN4ja-en	Evaluation Result
------------------------------
+ News Commentary task:
[JaRuNC (News Commentary task)]
Task: NCPDru-ja	Evaluation Result
Task: NCPDja-ru	Evaluation Result
------------------------------
+ Restricted Translation task:
Task: ASPECRTen-ja	Evaluation Result
Task: ASPECRTja-en	Evaluation Result
Task: ASPECRTja-zh	Evaluation Result
Task: ASPECRTzh-ja	Evaluation Result
------------------------------
+ Speech Translation Task:
[Low-Resource Khmer --> English/French Speech Translation Task]**
Task: ECCCkm-en	Evaluation Result
Task: ECCCkm-fr	Evaluation Result

* JPCN1{zh-ja,ja-zh,ko-ja,ja-ko,en-ja,ja-en} are the same tasks as JPC{zh-ja,ja-zh,ko-ja,ja-ko,en-ja,ja-en}, respectively.

** AMFM is not calculated for JPC{N,N2,N3,N4}, MMT, ALT20, BSD, MSCOCO, INDIC22, ECCC, MM{EV,CH}{TEXT,HI,MM}22, SOFTWARE22 and SWSTR22 tasks.

*** The BLEU values for SWSTR22 tasks are XML-BLUE. Also, RIBES for these tasks are not calculated. Please refer to this page for more information.

EVALUATION RESULTS (2014 - 2022)

[MultiIndicMT (Indic Languages Multilingual Parallel Corpus 2021)]**
Task: INDIC21en-bn	Evaluation Result
Task: INDIC21bn-en	Evaluation Result
Task: INDIC21en-gu	Evaluation Result
Task: INDIC21gu-en	Evaluation Result
Task: INDIC21en-hi	Evaluation Result
Task: INDIC21hi-en	Evaluation Result
Task: INDIC21en-kn	Evaluation Result
Task: INDIC21kn-en	Evaluation Result
Task: INDIC21en-ml	Evaluation Result
Task: INDIC21ml-en	Evaluation Result
Task: INDIC21en-mr	Evaluation Result
Task: INDIC21mr-en	Evaluation Result
Task: INDIC21en-or	Evaluation Result
Task: INDIC21or-en	Evaluation Result
Task: INDIC21en-pa	Evaluation Result
Task: INDIC21pa-en	Evaluation Result
Task: INDIC21en-ta	Evaluation Result
Task: INDIC21ta-en	Evaluation Result
Task: INDIC21en-te	Evaluation Result
Task: INDIC21te-en	Evaluation Result
[MultiIndicMT (Indic Languages Multilingual Parallel Corpus 2020)]
Task: INDIC20en-bn	Evaluation Result
Task: INDIC20bn-en	Evaluation Result
Task: INDIC20en-hi	Evaluation Result
Task: INDIC20hi-en	Evaluation Result
Task: INDIC20en-gu	Evaluation Result
Task: INDIC20gu-en	Evaluation Result
Task: INDIC20en-ml	Evaluation Result
Task: INDIC20ml-en	Evaluation Result
Task: INDIC20en-mr	Evaluation Result
Task: INDIC20mr-en	Evaluation Result
Task: INDIC20en-ta	Evaluation Result
Task: INDIC20ta-en	Evaluation Result
Task: INDIC20en-te	Evaluation Result
Task: INDIC20te-en	Evaluation Result
[IITB Corpus (2020)]
Task: HINDENen-hi	Evaluation Result
Task: HINDENhi-en	Evaluation Result
Task: HINDENhi-ja	Evaluation Result
Task: HINDENja-hi	Evaluation Result
Task: UFALta-en	Evaluation Result
Task: UFALen-ta	Evaluation Result
Task: ODIAENen-od	Evaluation Result
Task: ODIAENod-en	Evaluation Result
[Timely Disclosure tasks: Timely Disclosure Documents Corpus (2020)]
Task: TDDCITMen-ja	Evaluation Result
Task: TDDCITMja-en	Evaluation Result
Task: TDDCTXTen-ja	Evaluation Result
Task: TDDCTXTja-en	Evaluation Result
[Asian Language Treebank (ALT) Project 2019]
Task: ALT2en-km	Evaluation Result
Task: ALT2km-en	Evaluation Result
Task: ALT2en-my	Evaluation Result
Task: ALT2my-en	Evaluation Result
[JPO Patent Corpus2 (-2021)]
Task: JPCNzh-ja	Evaluation Result
Task: JPCNja-zh	Evaluation Result
Task: JPCNko-ja	Evaluation Result
Task: JPCNja-ko	Evaluation Result
Task: JPCNen-ja	Evaluation Result
Task: JPCNja-en	Evaluation Result
Task: JPCN2ko-ja	Evaluation Result
Task: JPCN2ja-ko	Evaluation Result
Task: JPCEPzh-ja	Evaluation Result
[JPO Patent Corpus (-2017)]
Task: JPCzh-ja*	Evaluation Result
Task: JPCja-zh*	Evaluation Result
Task: JPCko-ja*	Evaluation Result
Task: JPCja-ko*	Evaluation Result
Task: JPCen-ja*	Evaluation Result
Task: JPCja-en*	Evaluation Result
[BPPT Corpus (2016)]
Task: BPPTen-id	Evaluation Result
Task: BPPTid-en	Evaluation Result
[RECIPE Corpus]
Task: RECIPEALLen-ja	Evaluation Result
Task: RECIPEALLja-en	Evaluation Result
Task: RECIPETTLen-ja	Evaluation Result (for only title)
Task: RECIPETTLja-en	Evaluation Result (for only title)
Task: RECIPESTEen-ja	Evaluation Result (for only steps)
Task: RECIPESTEja-en	Evaluation Result (for only steps)
Task: RECIPEINGen-ja	Evaluation Result (for only ingredients)
Task: RECIPEINGja-en	Evaluation Result (for only ingredients)
[Indic Languages Multilingual Parallel Corpus]
Task: INDICen-bn	Evaluation Result
Task: INDICbn-en	Evaluation Result
Task: INDICen-hi	Evaluation Result
Task: INDIChi-en	Evaluation Result
Task: INDICen-ml	Evaluation Result
Task: INDICml-en	Evaluation Result
Task: INDICen-ta	Evaluation Result
Task: INDICta-en	Evaluation Result
Task: INDICen-te	Evaluation Result
Task: INDICte-en	Evaluation Result
Task: INDICen-ur	Evaluation Result
Task: INDICur-en	Evaluation Result
Task: INDICen-si	Evaluation Result
Task: INDICsi-en	Evaluation Result
Task: HINDENMMEVTEXTen-hi	Evaluation Result
Task: HINDENMMEVHIen-hi	Evaluation Result
Task: HINDENMMEVMMen-hi	Evaluation Result
Task: HINDENMMCHTEXTen-hi	Evaluation Result
Task: HINDENMMCHHIen-hi	Evaluation Result
Task: HINDENMMCHMMen-hi	Evaluation Result
[Asian Language Treebank (ALT) Project]
Task: ALTen-my	Evaluation Result
Task: ALTmy-en	Evaluation Result

AUTOMATIC EVALUATION PROCEDURES

Tools

Tools used by the automatic evaluation procedures

Procedures

For Japanese translation results (zh-ja, en-ja, JPC{22,N1,N2,N3,N4}zh-ja, JPC{22,N1,N3,N4}ko-ja, JPC{22,N1,N2,N3,N4}en-ja, ,HINDENhi-ja, JIJIen-ja, TDDCITMen-ja, TDDCTXTen-ja, NCPDru-ja)

For Chinese translation results (ja-zh, JPC{N,N1,N2,N3}ja-zh)

For Korean translation results (JPC{N,N1,N2,N3}ja-ko)

For Indic languages translation results (HINDEN{en,ja}-hi, UFALen-ta)

For English translation results (ja-en, JPC{N,N1,N2,N3}ja-en, HINDENhi-en, JIJIja-en, ALT2my-en, ALT2km-en, TDDCITMja-en, TDDCTXTja-en, UFALta-en)

For Myanmar translation results (ALT2en-my)

For Khmer translation results (ALT2en-km)

For Russian translation results (NCPDja-ru)

EVALUATION RESULTS USAGE POLICY

When you use the WAT evaluation results for any purpose such as:
- writing technical papers,
- making presentations about your system,
- advertising your MT system to the customers,
you can use the information about translation directions, scores (including both automatic and human evaluations) and ranks of your system among others. You can also use the scores of the other systems, but you MUST anonymize the other system's names. In addition, you can show the links (URLs) to the WAT evaluation result pages.

TECHNICAL COLLABORATOR

Luis Fernando D'Haro, Universidad Politécnica de Madrid, Spain
Rafael E. Banchs, Nanyang Technological University, Singapore
Haizhou Li, National University of Singapore, Singapore

NICT (National Institute of Information and Communications Technology)
Kyoto University
Last Modified: 2018-08-13

WAT The Workshop on Asian Translation Evaluation Results