WAT 2016

The 3rd Workshop on Asian Translation

December 12, 2016
Osaka, Japan

[SPONSOR] | [TIMETABLE] | [INVITED TALK] | [IMPORTANT DATES] | [RESEARCH PAPER] | [TASK] | [EVALUATION] | [BASELINE SYSTEM] | [PAPER SUBMISSION INFORMATION] | [TRANSLATION RESULTS SUBMISSION INFORMATION] | [REGISTRATION] | [PAPERS] | [CONTACT] | [PREVIOUS WORKSHOPS]

papers with supplementary materials has been uploaded (2017/01/30)
revision of the proceedings has been uploaded (2016/12/21)
Please answer the questionnaire after the workshop
proceedings has been uploaded (2016/12/11)

INTRODUCTION

The Workshop on Asian Translation (WAT) is a new open evaluation campaign focusing on Asian languages. We would like to invite a broad range of participants and conduct various forms of machine translation experiments and evaluation. Collecting and sharing our knowledge will allow us to understand the essence of machine translation and the problems to be solved. We are working toward the practical use of machine translation among all Asian countries.

For the 3rd WAT, we chose scientific papers, patents, news, and TED talks as the targeted domain, and selected the languages Japanese, Chinese, Korean, Hindi, Indonesian, and English. In addtion to the shared tasks, the 3rd WAT calls for research papers on topics related to machine translation, especially for Asian languages.

What makes WAT unique:

• Open innovation platform
  The test data is fixed and open, so you can repeat evaluations on the same data and confirm changes in translation accuracy over time. WAT has no deadline for the automatic translation quality evaluation (continuous evaluation), so you can submit translation results at any time.
• Domain and language pairs
  WAT is the world's first workshop that uses scientific papers as a domain and Japanese-Chinese as a language pair. Many Asian languages, such as Korean, Hindi, and Indonesian are involved in WAT. In the future, more Asian languages will be covered, such as Vietnamese, Thai, Myanmar and so on.
• Evaluation method
  Evaluation will be done by both automatic and human evaluation. For human evaluation, WAT will use crowdsourcing, which is low cost and allows multiple evaluations.

SPONSOR

SunFlare Co., Ltd.

TOIN Corporation

Baobab, Inc.

Asia-Pacific Association for Machine Translation (AAMT)

PostEdit.Tokyo Co., Ltd.

TIMETABLE

INVITED TALK

Dr. Hideto Kazawa, Senior Engineering Manager, Google [Short bio.] Title: Google's Neural Machine Translation System: Training and Serving Very Large Neural MT Models (Click here to more information. ) Time: 14:00 - 14:45

IMPORTANT DATES

RESEARCH PAPER

• Word-/phrase-/syntax-/semantics-/rule-based, neural, and hybrids machine translation
  
• Asian language processing
  
• Incorporating linguistic information into machine translation
  
• Decoding algorithms
  
• System combination
  
• Error analysis
  
• Manual and automatic machine translation evaluation
  
• Machine translation applications
  
• Quality estimation
  
• Domain adaptation
  
• Machine translation for low resource languages
  
• Language resources
  

TASK

Subtasks:

• Scientific paper Subtasks:
  
  • English <--> Japanese
    
  • Chinese <--> Japanese
    
• Patent Subtasks:
  
  • Chinese <--> Japanese
    
  • Korean <--> Japanese
    
  • English <--> Japanese
    
• Newswire Subtasks:
  
  • Indonesian <--> English
    
  • Chinese <--> Engish
    
  • Japanese <--> Engish
    
• Mixed domain Subtask:
  
  • Hindi <--> English
    
  • Hindi <--> Japanese
    

Dataset:

• Scientific paper Subtasks:
  WAT uses ASPEC for the dataset including training, development, development test and test data. Participants of the scientific papers subtask must get a copy of ASPEC by themselves. ASPEC consists of approximately 3 million Japanese-English parallel sentences from paper abstracts (ASPEC-JE) and approximately 0.7 million Japanese-Chinese paper excerpts (ASPEC-JC)
• Patent Subtasks:
  WAT uses JPO Patent Corpus, which is constructed by Japan Patent Office (JPO). This corpus consists of 1 million English-Japanese parallel sentences, 1 million Chinese-Japanese parallel sentences, and 1 million Korean-Japanese parallel sentences from patent description with four categories. Participants of patents subtask are required to get it on WAT2016 site of JPO Patent Corpus.
• Newswire Subtask (Indonesian <--> English):
  WAT uses BPPT Corpus, which is constructed by Badan Pengkajian dan Penerapan Teknologi (BPPT). This corpus consists of 50,000 Indonesian-English parallel sentences from news description with five categories. Participants of patents subtask are required to get it on WAT2016 site of BPPT Corpus.
• Mixed domain Subtasks:
  
  • Hindi <--> English
    WAT uses IITB Corpus for the dataset for training, development, development test and test data. The training corpus is mixed domain and contains around 1 million lines of sentences and phrases. In order to access the corpus participants should sign the following agreement, scan and send it to the addresss mentioned in it. The training corpus is a mixed domain corpus. The development and test set are from the News domain and are exactly the same as the ones in WMT 2014.
  • Hindi <--> Japanese Pivot Language Task
    For the first time we are introducing a pivot language task. For this tasks participants can use the following corpora.
    1. A parallel corpus (created using openly available corpora) which is located at here.
    2. The Hindi-English (IITB) task corpus and the English-Japanese (ASPEC) task corpus for pivoting. For triangulation of the source-pivot and pivot-target phrase tables they can use the scripts provided by: MultiMT .
    The objective of this task is to compare the performance of a baseline system constructed only on a mixed domain parallel corpus with a system that uses additional mixed domain corpus by means of pivoting.

Baseline system:

Baseline systems site is now open.

EVALUATION

We will evaluate the translation performance of the results submitted through automatic evaluation and human evaluation.

Automatic evaluation:
We will prepare an automatic evaluation server. You will be able to evaluate the translation results at any time using this server.

• Metric:
  BLEU, RIBES, and AM-FM^*
  * A two-dimensional automatic evaluation metric for machine translation, designed to operate at the sentence level.
  * It is based on adequacy and fluency, to decouple semantic and syntactic components of the translation process to provide a balanced view on translation quality.
  
• Format:
  The submission format and the submission method are given at the submission site below.
• Notice:
  When submitting, participants have to agree that the submitted results are attributed to JST and NICT. The results will be used and distributed for research by JST and NICT.
  Thanks to the technical collaborators: Luis Fernando D'Haro, Rafael E. Banchs and Haizhou Li.
  

Human evaluation:
Human evaluation will be carried out with two kinds of method, which are Pairwise Crowdsourcing Evaluation and JPO Adequacy Evaluation.

• Pairwise Crowdsourcing Evaluation:
  Pairwise Crowdsourcing Evaluation will be carried out using crowdsourcing. Organizers will sample 400 sentences from the test data for the pairwise crowdsourcing evaluation. Participants can submit translation results for the human evaluation a maximum of twice until the pairwise crowdsourcing evaluation is due. (For automatic evaluation, there is no limitation on submitting times.)
  
  • Metric:
    Sentence-by-sentence pair-wise evaluation compared to the baseline system. The crowdsourcing workers will be asked to judge which translation is better than the other in view of adequacy and fluency. To guarantee the quality of the evaluation, each sentence is evaluated by 5 different workers and the final decision is made by the voting of the judgements.
  • Format:
    The submission format is the same as that of automatic evaluation. Participants can select their translation results from the ones submitted for automatic evaluation.
  • Ranking:
    All systems will be ranked by the percentage of translations judged to improve upon the baseline system.

• JPO Adequacy Evaluation:
  We will also evaluate with the criteria of Content Transmission Level Evaluation which JPO defined (pages 5 to 8 in the PDF file (Japanese Page)). We will sample 200 sentences from the pairwise crowdsourcing evaluation data for the JPO adequacy evaluation. The JPO adequacy evaluation will be conducted only for translation results of 3 top-scored teams on the Pairwise Crowdsourcing Evaluation in each subtask's language pair.

Submission:

Submission site is now open.
(User Name and Password is necessary to access.)

Evaluation results:

Evaluation results site is now open.

PAPER SUBMISSION INFORMATION

Participants who submit results for human evaluation should submit description papers of their translation systems and evaluation results. All submissions and feedback are handled electronically as below.

• Format:
  PDF file format, single columns, no smaller than nine (9) point type font throughout the paper.
• Language:
  Papers should be written in English.
• Maximum length:
  The maximum length is 10 pages including Appendix, plus two extra pages for references. The number of pages for references may exceed 2 pages, on the condition that the number of total pages is less than or equal to 12.
• Template:
  Participants must use the COLING 2016 LaTeX style files or Microsoft Word style files that are available on the COLING 2016 conference web site.
• Submission:
  Submission site is now open.

TRANSLATION RESULT SUBMISSION INFORMATION

The applicating site for task participants of WAT2016 is now open.

REGISTRATION

The registration will be handled by COLING 2016

ORGANIZERS

• Toshiaki Nakazawa, Japan Science and Technology Agency (JST), Japan
  
• Hideya Mino, National Institute of Information and Communications Technology (NICT), Japan
  
• Chenchen Ding, National Institute of Information and Communications Technology (NICT), Japan
  
• Isao Goto, Japan Broadcasting Corporation (NHK), Japan
  
• Graham Neubig, Nara Institute of Science and Technology (NAIST), Japan
  
• Sadao Kurohashi, Kyoto University, Japan
  
• Ir. Hammam Riza, Agency for the Assessment and Application of Technology (BPPT), Indonesia
  
• Pushpak Bhattacharyya, Indian Institute of Technology Bombay (IIT), India
  

PROGRAM COMMITTEE MEMBERS

• Rafael E. Banchs, Institute for Infocomm Research, Singapore
• Hailong Cao, Harbin Institute of Technology, China
• Michael Carl, Copenhagen Business School, Denmark
• Marine Carpuat, University of Maryland, USA
• Chenhui Chu, JST, Japan
• Fabien Cromières, JST, Japan
• Hideto Kazawa, Google, Japan
• Anoop Kunchookuttan, IIT Bombay, India
• Jong-Hyeok Lee, Pohang University of Science and Technology, Korea
• Gurpreet Singh Lehal, Punjabi University, Patiala, India
• Haizhou Li, Institute for Infocomm Research, Singapore
• Qun Liu, Dublin City University, Ireland
• Liling Tan, Universität des Saarlandes, Germany
• Masao Utiyama, NICT, Japan
• Andy Way, Dublin City University, Ireland
• Dekai Wu, HKUST, Hong Kong
• Deyi Xiong, Soochow University, China
• Dongdong Zhang, Microsoft Research Asia, China
• Jiajun Zhang, Chinese Academy of Sciences, China

TECHNICAL COLLABORATOR

• Luis Fernando D'Haro, Institute for Infocomm Research, Singapore
• Rafael E. Banchs, Institute for Infocomm Research, Singapore
• Haizhou Li, Institute for Infocomm Research, Singapore

CONTACT

For questions, comments, etc. please email to "wat -at- nlp -dot- ist -dot- i -dot- kyoto -hyphen- u -dot- ac -dot- jp".

RELATED LINK

COLING 2016
Japan Patent Office
JPO Patent Corpus
The Association for Natural Language Processing (Japanese Page)
Asian Scientific Paper Excerpt Corpus (ASPEC)
Japan Science and Technology Agency (JST)
National Institute of Information and Communications Technology (NICT)

PREVIOUS WORKSHOPS

• WAT 2015
  The 2nd Workshop on Asian Translation (WAT 2015) was held in October 2015 in Kyoto, Japan.
• WAT 2014
  The 1st Workshop on Asian Translation (WAT 2014) was held in October 2014 in Tokyo, Japan.

CHANGE LOG

2016-07-14: data preparation for patent CJ updated, PB SMT baseline for JK updated
2016-07-12: paper submission site open
2016-06-29: links for baseline system and evaluation results are added
2016-06-28: tasks modified, data preparation / PB SMT baseline for IE / EI updated, data preparation for patent JE / EJ updated
2016-06-14: application site open
2016-06-11: schedule fixed
2016-05-25: site open

JST (Japan Science and Technology Agency)
NICT (National Institute of Information and Communications Technology)
Kyoto University
Last Modified: 2016-07-14