WAT 2020 Newswire Tasks Description

 

1.    Subtasks

1.1  There are two language directions:

l  Japanese to English

l  English to Japanese

1.2  There are two types of test sets:

l  Test set I: A pair of test and reference sentences. The references were automatically extracted from English newswire sentences and not manually checked. There are no context data.

l  Test set II: New test set added at WAT 2020. A pair of test and reference sentences and context data that are articles including test sentences. The references were automatically extracted from English newswire sentences and manually selected. Therefore, the quality of the references of test set II is better than that of test set I.

Participants submit the translation results of one or more of the test data.

 

2.    Official Data

2.1  Files

filename

Contents

train.txt

0.2 million sentence pairs

devtest.txt

2,000 sentence pairs

dev.txt

2,000 sentence pairs

test.txt

2,000 sentence pairs

devc.tsv

479 sentence pairs

testc.tsv

1851 sentence pairs

context-devc.ja.tsv

132 articles

context-devc.en.tsv

132 articles

context-testc.ja.tsv

546 articles

context-testc.en.tsv

546 articles

 

3.    Definition of Data Use 

 

Use

Contents

Japanese to English

Training

train.txt, devtest.txt, dev.txt, devc.tsv, context-devc.ja.tsv, and context-dev.en.tsv

Test set I

To be translated

Japanese sentences in test.txt

Reference

English sentences in test.txt

Test set II

To be translated

Japanese sentences in testc.tsv

Context

context-testc.ja.tsv

Reference

English sentences in testc.tsv

English to Japanese

Training

train.txt, devtest.txt, dev.txt, devc.tsv, context-devc.ja.tsv, and context-dev.en.tsv

Test set I

To be translated

English sentences in test.txt

Reference

Japanese sentences in test.txt

Test set II

To be translated

English sentences in testc.tsv

Context

context-testc.en.tsv

Reference

Japanese sentences in testc.tsv

 

4.    Data Format

Files with the extension ".txt" are text files and their format is "<Japanese text> ||| <English text>".

Files with the extension ".tsv" are tab separated values and their columns are Sentence ID, Japanese text and corresponding English text for {devc,testc}.tsv and Article ID, Sentence ID, Japanese text and corresponding English text for context-{devc,testc}.{en,ja}.tsv.

One line may contain more than one sentence. Sentences are separated by a " || ".

Some of personal names were anonymized by replacing them with 〇〇〇〇.

 

5.    Policy for External Resource Usage and Declaration

Participants will be allowed to use external data other than the official data for their MT systems, if they make a declaration of the resources they used for each submission.

When participants use external resources to train their MT systems, they may not use JIJI news articles published between January, 2018 to March, 2018 because test set II were selected from JIJI news articles published in this period.

Participants may also not use sentences that are included in test set I when they use JIJI news published before June, 2017 because the test set I were selected from JIJI news articles published before June, 2017 (the specific period is unknown).

 

CHANGE LOG

2020-4-11: initial version