NICT_LOGO.JPG KYOTO-U_LOGO.JPG

WAT

The Workshop on Asian Translation
Evaluation Results

[EVALUATION RESULTS TOP] | [BLEU] | [RIBES] | [AMFM] | [HUMAN (WAT2020)] | [HUMAN (WAT2019)] | [HUMAN (WAT2018)] | [HUMAN (WAT2017)] | [HUMAN (WAT2016)] | [HUMAN (WAT2015)] | [HUMAN (WAT2014)] | [EVALUATION RESULTS USAGE POLICY]

BLEU


# Team Task Date/Time DataID BLEU
Method
Other
Resources
System
Description
juman kytea mecab moses-
tokenizer
stanford-
segmenter-
ctb
stanford-
segmenter-
pku
indic-
tokenizer
unuse myseg kmseg
1NTTen-ja2019/07/28 15:44:12323645.8347.6346.57-------NMTNoASPEC first 1.5M + Synthetic 1.5M, 6 ensemble
2srcben-ja2019/07/27 15:50:19321245.7147.5546.29-------NMTNoTransformer(Big) with relative position, sentence-wise smooth, deep transformer, back translation, ensemble of 10 models, rerank.
3srcben-ja2019/07/27 15:28:30320645.3047.1245.93-------NMTNoTransformer(Big) with relative position, sentence-wise smooth, deep transformer, back translation, ensemble of 7 models, re-rank.
4NTTen-ja2019/07/28 15:55:18323945.3047.1346.04-------NMTYesParaCrawl + (ASPEC first 1.5M + Synthetic 1.5M) * 2 oversampling, fine-tune ASPEC, SINGLE MODEL
5NICT-2en-ja2019/07/27 10:57:26318244.6146.5945.66-------NMTNoTransformer, ensemble of 4 models w/ long warm-up and self-training
6KNU_Hyundaien-ja2019/07/27 09:19:31317244.0845.8844.78-------NMTNoTransformer Base, relative position, BT, r2l reranking, ensemble of 3 models
7NICT-2en-ja2019/07/27 10:56:02318143.7945.8244.83-------NMTNoTransformer, sigle model w/ long warm-up and self-training
8AISTAIen-ja2019/08/30 20:23:24335743.7645.7044.52-------NMTNoTransformer, 1.5M sentences, relative position, ensemble of 3 models, by OpenNMT-py.
9srcben-ja2019/07/25 11:43:06291843.6045.4944.29-------NMTNoTransformer (Big) with relative position, sentence-wise smooth
10srcben-ja2018/09/16 15:52:17248043.4344.9444.05----- 0.00 0.00NMTNoTransformer with relative position, ensemble of 4 models, rerank.
11AISTAIen-ja2019/09/05 08:18:45337342.9244.9043.67-------NMTNoTransformer (big), 1.5M sentences, train_steps=300000, Averaged the last 20 ckpts, by Tensor2Tensor.
12NICT-5en-ja2018/09/03 16:49:57221942.8744.4243.49----- 0.00 0.00NMTNoBig Bidirectional Transformer. 1.5M sentences only.
13AISTAIen-ja2019/07/29 07:33:56325142.6444.1743.34-------NMTNoTransformer (big). 1.5M sentences, train_steps=131000 only. Averaged the last 10 ckpts.
14srcben-ja2018/09/16 15:26:37247942.4944.1143.20----- 0.00 0.00NMTNoTransformer with relative position, ensemble of 3 models.
15NICT-5en-ja2018/08/22 18:43:00204841.9143.5042.60----- 0.00 0.00NMTNoTensor2Tensor's transformer implementation. Used recurrently stacked layers model logic. Layer recurrence logic: 1-2-3-1-2-3-1-2-3-1-2-3. Codename: Megarecurrence.
16Kyoto-Uen-ja2017/09/04 22:31:18175941.5343.2642.13---- 0.00 0.00 0.00SMTNoSyscomb of AIAYN and KNMT
17ORGANIZERen-ja2017/08/02 01:04:49173740.7942.5541.50---- 0.00 0.00 0.00NMTNoGoogle's "Attention Is All You Need"
18NTTen-ja2017/08/01 15:42:13172940.3242.8040.95---- 0.00 0.00 0.00NMTNoEnsemble 8 Models: joint BPE 16k, BiLSTM Encoder 512*2*2, LtoR LSTM Decoder 512*2, Beam Search 20 w/ length-based reranking, add 1M pseudo training data (generated from provided training data)
19NICT-2en-ja2017/07/26 14:03:00147940.1742.2541.17---- 0.00 0.00 0.00NMTNoNMT 6 Ensembles * Bi-directional Reranking
20NICT-5en-ja2018/08/22 18:48:51205040.0241.7440.77----- 0.00 0.00NMTNoRSNMT 6 layer
21NTTen-ja2017/08/01 07:12:25168439.8042.2740.47---- 0.00 0.00 0.00NMTNoEnsemble 8 Models: joint BPE 16k, BiLSTM Encoder 512*2*2, LtoR LSTM Decoder 512*2, Beam Search 20 w/ length-based reranking
22NTTen-ja2017/08/01 02:22:56167338.8741.4239.71---- 0.00 0.00 0.00NMTNoSingle Model: joint BPE 16k, BiLSTM Encoder 512*2*2, LtoR LSTM Decoder 512*2, Beam Search 20 w/ length-based reranking, add 1M pseudo training data (generated from provided training data)
23Kyoto-Uen-ja2017/08/01 15:49:51173138.7240.6539.37---- 0.00 0.00 0.00NMTNoEnsemble of 4 BPE Averaged
24UT-KAYen-ja2017/06/27 16:58:42136938.3241.1239.32---- 0.00 0.00 0.00NMTNoHashimoto and Tsuruoka (2017), https://arxiv.org/abs/1702.02265
25NAIST-NICTen-ja2017/07/27 21:50:15150738.2540.2939.05---- 0.00 0.00 0.00NMTNoSingle Model: SPM16k/16k, BiLSTM Encoder 512*2*2, UniLSTM Decoder 512*2, Adjusted Search
26Osaka-Uen-ja2018/09/15 23:02:05243938.0140.0039.10----- 0.00 0.00NMTYesrewarding model
27EHRen-ja2018/09/08 12:42:03224537.9740.0038.66----- 0.00 0.00NMTNoSMT reranked NMT
28NTTen-ja2017/07/30 16:16:25160837.9040.4838.61---- 0.00 0.00 0.00NMTNoSingle Model: joint BPE 16k, BiLSTM Encoder 512*2*2, LtoR LSTM Decoder 512*2, Beam Search 20 w/ length-based reranking
29NICT-2en-ja2017/07/26 13:52:35147536.8538.9437.87---- 0.00 0.00 0.00NMTNoNMT Single Model: BPE50k, Bi-LSTM(500*2) Encoder, LSTM(1000) Left-to-Right Decoder
30NAIST-NICTen-ja2017/07/27 21:49:04150636.4738.5437.30---- 0.00 0.00 0.00NMTNoSingle Model: SPM16k/16k, BiLSTM Encoder 512*2*2, UniLSTM Decoder 512*2, One-best Search
31ORGANIZERen-ja2018/08/14 10:58:49190036.3738.4837.15----- 0.00 0.00NMTNoNMT with Attention
32UT-IISen-ja2017/08/01 12:03:22171036.2638.9337.06---- 0.00 0.00 0.00NMTNoNMT 8 Ensembles with beam search, Sentence Piece, Embedding layer initialization
33Kyoto-Uen-ja2016/08/18 03:45:51117236.1938.2036.78---- 0.00 0.00 0.00NMTNoBPE tgt/src: 52k 2-layer lstm self-ensemble of 3
34NAISTen-ja2015/08/25 12:39:2376135.8338.1736.61--- 0.00 0.00 0.00 0.00SMTNoTravatar System with NeuralMT Reranking
35TMUen-ja2017/08/04 11:06:51174735.7338.3437.00---- 0.00 0.00 0.00NMTNobeam_size: 10, ensemble of different dropout rates.
36Senseen-ja2018/08/24 11:38:23209235.2937.2136.02----- 0.00 0.00NMTNo
37TMUen-ja2018/09/16 12:44:23246935.0837.6936.14----- 0.00 0.00NMTNoEnsemble of 6 Baseline-NMT
38NAISTen-ja2014/07/31 11:38:3711835.0337.1635.81--- 0.00 0.00 0.00 0.00SMTNoTravatar-based Forest-to-String SMT System
39NAISTen-ja2014/08/01 17:37:2312634.8437.1535.67--- 0.00 0.00 0.00 0.00SMTNoTravatar-based Forest-to-String SMT System (Tuned BLEU+RIBES)
40NICT-2en-ja2016/08/05 17:47:05109734.6736.8635.37---- 0.00 0.00 0.00SMTYesPhrase-based SMT with Preordering + Domain Adaptation (JPC and ASPEC) + Google 5-gram LM
41naveren-ja2015/08/31 14:18:1883734.6036.1435.30--- 0.00 0.00 0.00 0.00SMTNoSMT t2s + Spell correction + NMT reranking
42TMUen-ja2017/08/02 10:20:14174134.5836.9535.63---- 0.00 0.00 0.00NMTNothe ensemble system of different dropout rate.
43NAISTen-ja2015/08/25 12:47:4976334.3836.5835.16--- 0.00 0.00 0.00 0.00SMTNoTravatar System Baseline
44TMUen-ja2018/09/16 17:25:36248434.3837.0635.44----- 0.00 0.00NMTNoEnsemble of 6 NMT ( 2 Baseline + 2 Reconstructor + 2 GAN )
45TMUen-ja2017/08/01 11:56:31170934.0536.6935.32---- 0.00 0.00 0.00NMTNobaseline system with beam20
46TMUen-ja2018/09/16 12:10:49246734.0236.6935.17----- 0.00 0.00NMTNoBaseline-NMT ( Single )
47TMUen-ja2018/09/16 12:11:58246834.0036.6535.09----- 0.00 0.00NMTNoGAN-NMT ( Single )
48TMUen-ja2018/09/16 16:56:49248233.8636.5834.89----- 0.00 0.00NMTNoReconstructor-NMT ( Single )
49UT-AKYen-ja2016/08/20 12:34:43122833.5736.9534.65---- 0.00 0.00 0.00NMTNotree-to-seq NMT model (word-based decoder)
50Wen-ja2015/08/26 16:00:4678633.2336.2134.05--- 0.00 0.00 0.00 0.00SMTNoNMT, LSTM Search, 5 ensembles, beam size 20, UNK replacing, System Combination with NMT score (Pick top-1k results from NMT)
51naveren-ja2015/08/31 14:12:0383633.1435.7533.93--- 0.00 0.00 0.00 0.00SMTNoNMT only
52Kyoto-Uen-ja2015/08/30 19:59:1883233.0635.5733.99--- 0.00 0.00 0.00 0.00EBMTNoKyotoEBMT system with bilingual RNNLM reranking
53naveren-ja2015/08/25 16:20:3077032.7634.5333.47--- 0.00 0.00 0.00 0.00SMTNoSMT t2s + Spell correction
54Wen-ja2014/08/26 16:17:1520232.6935.0433.40--- 0.00 0.00 0.00 0.00SMTNoWeblio Pre-reordering SMT System (with forest inputs)
55TMUen-ja2017/08/01 11:42:03170432.6535.0533.72---- 0.00 0.00 0.00NMTNoour baseline system in 2017
56naveren-ja2015/08/04 16:48:3258132.6334.4633.34--- 0.00 0.00 0.00 0.00SMTNoSMT t2s
57Wen-ja2014/08/16 00:57:1613232.5334.8733.26--- 0.00 0.00 0.00 0.00SMTNoWeblio Pre-reordering SMT System Baseline
58TOSHIBAen-ja2015/07/28 16:24:3752432.0634.1732.76--- 0.00 0.00 0.00 0.00SMT and RBMTYesRBMT with SPE(Statistical Post Editing) system
59TOKYOMTen-ja2016/08/19 23:18:46121732.0334.7732.98---- 0.00 0.00 0.00NMTNoCombination of NMT and T2S
60TOSHIBAen-ja2015/07/30 11:22:1154031.8233.9432.54--- 0.00 0.00 0.00 0.00SMT and RBMTYesSystem combination SMT and RBMT(SPE) with RNNLM language model
61EHRen-ja2016/08/15 11:21:51114031.3233.5832.28---- 0.00 0.00 0.00SMTNoPBSMT with preordering (DL=6)
62bjtu_nlpen-ja2016/08/16 11:21:07114331.1833.4731.80---- 0.00 0.00 0.00NMTNoRNN Encoder-Decoder with attention mechanism, single model
63Kyoto-Uen-ja2014/09/01 21:06:3226731.0933.5531.73--- 0.00 0.00 0.00 0.00EBMTNoOur new baseline system after several modifications + 20-best parses, KN7, RNNLM reranking
64ORGANIZERen-ja2014/07/11 20:03:211231.0533.4432.10--- 0.00 0.00 0.00 0.00SMTNoTree-to-String SMT (2014)
65ORGANIZERen-ja2015/09/10 13:29:3587531.0533.4432.10--- 0.00 0.00 0.00 0.00SMTNoTree-to-String SMT (2015)
66Kyoto-Uen-ja2016/08/19 10:18:09120131.0333.4031.88---- 0.00 0.00 0.00EBMTNoKyotoEBMT 2016 w/o reranking
67EHRen-ja2015/09/09 12:21:3487330.8833.2031.74--- 0.00 0.00 0.00 0.00SMT and RBMTYesSystem combination of RBMT with user dictionary plus SPE and phrase based SMT with preordering. Candidate selection by language model score.
68Wen-ja2015/08/28 14:30:5681330.7234.1931.57--- 0.00 0.00 0.00 0.00OtherNoNMT, LSTM Search, Beam Size 20, Ensemble of 2 models, UNK replacing
69Kyoto-Uen-ja2015/08/27 22:56:1380530.6933.2531.71--- 0.00 0.00 0.00 0.00EBMTNoKyotoEBMT system without reranking
70SAS_MTen-ja2014/09/01 10:39:2726430.4733.0031.47--- 0.00 0.00 0.00 0.00SMTNoSyntactic reordering Hierarchical SMT (using part of data)
71EHRen-ja2015/09/09 12:22:5887430.4632.5631.28--- 0.00 0.00 0.00 0.00SMT and RBMTYesRBMT with user dictionary plus SPE.
72Kyoto-Uen-ja2014/08/25 14:16:1518630.2532.7830.84--- 0.00 0.00 0.00 0.00EBMTNoUsing n-best parses and RNNLM.
73TOKYOMTen-ja2016/08/12 11:21:43113130.2133.3831.24---- 0.00 0.00 0.00NMTNochar 1 , ens 2 , version 1
74ORGANIZERen-ja2014/09/16 13:36:3536730.1932.5630.94--- 0.00 0.00 0.00 0.00SMTNoHierarchical Phrase-based SMT (2014)
75UT-AKYen-ja2016/08/20 09:25:15122430.1433.2031.09---- 0.00 0.00 0.00NMTNotree-to-seq NMT model (character-based decoder)
76EHRen-ja2015/08/22 12:28:1574229.7832.3630.71--- 0.00 0.00 0.00 0.00SMTNoPhrase based SMT with preordering.
77Kyoto-Uen-ja2014/08/31 10:33:2325329.7632.4630.46--- 0.00 0.00 0.00 0.00EBMTNoOur new baseline system after several modifications.
78Kyoto-Uen-ja2014/08/19 10:16:0213428.9331.6129.59--- 0.00 0.00 0.00 0.00EBMTNoOur baseline system using 3M parallel sentences.
79Senseen-ja2014/08/25 01:07:2718427.9230.1828.66--- 0.00 0.00 0.00 0.00SMTNoBaseline SMT
80Senseen-ja2015/08/28 19:22:2482127.9130.0328.78--- 0.00 0.00 0.00 0.00SMTNoBaseline-2015 (train1 only)
81ORGANIZERen-ja2014/07/11 19:49:03527.4829.8028.27--- 0.00 0.00 0.00 0.00SMTNoPhrase-based SMT
82ORGANIZERen-ja2016/11/16 10:50:13133426.1928.2226.68---- 0.00 0.00 0.00NMTYesOnline A (2016/11/14)
83Senseen-ja2015/07/28 22:26:5553125.2227.2225.90--- 0.00 0.00 0.00 0.00SMTNoBaseline-2015
84Senseen-ja2015/08/18 22:04:0971524.4326.5825.36--- 0.00 0.00 0.00 0.00SMTYesPassive JSTx3
85Senseen-ja2015/08/18 21:52:0170024.1326.2424.96--- 0.00 0.00 0.00 0.00SMTNoBaseline-dictmt
86Osaka-Uen-ja2018/09/16 13:06:45247023.2425.5024.26----- 0.00 0.00SMTNopreordering with neural network
87JAPIOen-ja2016/08/17 12:50:50116520.5222.5621.05---- 0.00 0.00 0.00SMTYesPhrase-based SMT with Preordering + JAPIO corpus + rule-based posteditor
88ORGANIZERen-ja2014/07/18 11:02:253419.6621.6320.17--- 0.00 0.00 0.00 0.00OtherYesOnline A (2014)
89ORGANIZERen-ja2016/07/26 11:31:47104118.2819.8118.51---- 0.00 0.00 0.00OtherYesOnline A (2016)
90ORGANIZERen-ja2015/08/25 18:54:2977418.2219.7718.46--- 0.00 0.00 0.00 0.00OtherYesOnline A (2015)
91ORGANIZERen-ja2015/09/10 19:02:3888917.8019.5218.11--- 0.00 0.00 0.00 0.00OtherYesOnline B (2015)
92ORGANIZERen-ja2014/07/22 13:30:139117.0418.6717.36--- 0.00 0.00 0.00 0.00OtherYesOnline B (2014)
93ORGANIZERen-ja2014/07/21 11:38:126613.1814.8513.48--- 0.00 0.00 0.00 0.00OtherYesRBMT B (2014)
94ORGANIZERen-ja2015/09/10 14:26:2888313.1814.8513.48--- 0.00 0.00 0.00 0.00OtherYesRBMT B (2015)
95ORGANIZERen-ja2014/07/21 11:40:506812.8614.4313.16--- 0.00 0.00 0.00 0.00OtherYesRBMT A
96ORGANIZERen-ja2014/07/23 14:50:449512.1913.3212.14--- 0.00 0.00 0.00 0.00OtherYesRBMT C

Notice:

Back to top

RIBES


# Team Task Date/Time DataID RIBES
Method
Other
Resources
System
Description
juman kytea mecab moses-
tokenizer
stanford-
segmenter-
ctb
stanford-
segmenter-
pku
indic-
tokenizer
unuse myseg kmseg
1NTTen-ja2019/07/28 15:44:1232360.8619940.8656400.868175-------NMTNoASPEC first 1.5M + Synthetic 1.5M, 6 ensemble
2NTTen-ja2019/07/28 15:55:1832390.8596730.8635320.866183-------NMTYesParaCrawl + (ASPEC first 1.5M + Synthetic 1.5M) * 2 oversampling, fine-tune ASPEC, SINGLE MODEL
3srcben-ja2019/07/27 15:50:1932120.8575060.8606420.862819-------NMTNoTransformer(Big) with relative position, sentence-wise smooth, deep transformer, back translation, ensemble of 10 models, rerank.
4KNU_Hyundaien-ja2019/07/27 09:19:3131720.8570600.8598850.863400-------NMTNoTransformer Base, relative position, BT, r2l reranking, ensemble of 3 models
5srcben-ja2019/07/27 15:28:3032060.8562450.8590250.863177-------NMTNoTransformer(Big) with relative position, sentence-wise smooth, deep transformer, back translation, ensemble of 7 models, re-rank.
6AISTAIen-ja2019/08/30 20:23:2433570.8560820.8591660.862434-------NMTNoTransformer, 1.5M sentences, relative position, ensemble of 3 models, by OpenNMT-py.
7NICT-2en-ja2019/07/27 10:57:2631820.8529700.8567550.859059-------NMTNoTransformer, ensemble of 4 models w/ long warm-up and self-training
8NICT-2en-ja2019/07/27 10:56:0231810.8504580.8539090.857275-------NMTNoTransformer, sigle model w/ long warm-up and self-training
9srcben-ja2018/09/16 15:26:3724790.8503180.8522090.857017-----0.0000000.000000NMTNoTransformer with relative position, ensemble of 3 models.
10AISTAIen-ja2019/07/29 07:33:5632510.8491290.8514000.856177-------NMTNoTransformer (big). 1.5M sentences, train_steps=131000 only. Averaged the last 10 ckpts.
11srcben-ja2019/07/25 11:43:0629180.8489290.8529480.855857-------NMTNoTransformer (Big) with relative position, sentence-wise smooth
12AISTAIen-ja2019/09/05 08:18:4533730.8478630.8516730.854716-------NMTNoTransformer (big), 1.5M sentences, train_steps=300000, Averaged the last 20 ckpts, by Tensor2Tensor.
13srcben-ja2018/09/16 15:52:1724800.8476780.8501800.855073-----0.0000000.000000NMTNoTransformer with relative position, ensemble of 4 models, rerank.
14NICT-5en-ja2018/09/03 16:49:5722190.8471340.8493990.853634-----0.0000000.000000NMTNoBig Bidirectional Transformer. 1.5M sentences only.
15ORGANIZERen-ja2017/08/02 01:04:4917370.8448960.8475590.851471----0.0000000.0000000.000000NMTNoGoogle's "Attention Is All You Need"
16NICT-2en-ja2017/07/26 14:03:0014790.8422060.8481700.849929----0.0000000.0000000.000000NMTNoNMT 6 Ensembles * Bi-directional Reranking
17NICT-5en-ja2018/08/22 18:48:5120500.8410110.8443600.848283-----0.0000000.000000NMTNoRSNMT 6 layer
18Kyoto-Uen-ja2017/09/04 22:31:1817590.8410000.8437250.847819----0.0000000.0000000.000000SMTNoSyscomb of AIAYN and KNMT
19NICT-5en-ja2018/08/22 18:43:0020480.8407760.8450420.849326-----0.0000000.000000NMTNoTensor2Tensor's transformer implementation. Used recurrently stacked layers model logic. Layer recurrence logic: 1-2-3-1-2-3-1-2-3-1-2-3. Codename: Megarecurrence.
20NTTen-ja2017/08/01 15:42:1317290.8385940.8417690.846486----0.0000000.0000000.000000NMTNoEnsemble 8 Models: joint BPE 16k, BiLSTM Encoder 512*2*2, LtoR LSTM Decoder 512*2, Beam Search 20 w/ length-based reranking, add 1M pseudo training data (generated from provided training data)
21NTTen-ja2017/08/01 07:12:2516840.8358060.8399810.844326----0.0000000.0000000.000000NMTNoEnsemble 8 Models: joint BPE 16k, BiLSTM Encoder 512*2*2, LtoR LSTM Decoder 512*2, Beam Search 20 w/ length-based reranking
22NAIST-NICTen-ja2017/07/27 21:50:1515070.8344920.8393210.842337----0.0000000.0000000.000000NMTNoSingle Model: SPM16k/16k, BiLSTM Encoder 512*2*2, UniLSTM Decoder 512*2, Adjusted Search
23NTTen-ja2017/08/01 02:22:5616730.8335410.8382450.841739----0.0000000.0000000.000000NMTNoSingle Model: joint BPE 16k, BiLSTM Encoder 512*2*2, LtoR LSTM Decoder 512*2, Beam Search 20 w/ length-based reranking, add 1M pseudo training data (generated from provided training data)
24Kyoto-Uen-ja2017/08/01 15:49:5117310.8324720.8358700.839646----0.0000000.0000000.000000NMTNoEnsemble of 4 BPE Averaged
25UT-KAYen-ja2017/06/27 16:58:4213690.8296000.8339200.838260----0.0000000.0000000.000000NMTNoHashimoto and Tsuruoka (2017), https://arxiv.org/abs/1702.02265
26EHRen-ja2018/09/08 12:42:0322450.8287460.8333330.837806-----0.0000000.000000NMTNoSMT reranked NMT
27NTTen-ja2017/07/30 16:16:2516080.8281930.8320060.836381----0.0000000.0000000.000000NMTNoSingle Model: joint BPE 16k, BiLSTM Encoder 512*2*2, LtoR LSTM Decoder 512*2, Beam Search 20 w/ length-based reranking
28UT-IISen-ja2017/08/01 12:03:2217100.8278910.8320540.836394----0.0000000.0000000.000000NMTNoNMT 8 Ensembles with beam search, Sentence Piece, Embedding layer initialization
29NICT-2en-ja2017/07/26 13:52:3514750.8267910.8344480.835255----0.0000000.0000000.000000NMTNoNMT Single Model: BPE50k, Bi-LSTM(500*2) Encoder, LSTM(1000) Left-to-Right Decoder
30Osaka-Uen-ja2018/09/15 23:02:0524390.8250610.8293280.833200-----0.0000000.000000NMTYesrewarding model
31ORGANIZERen-ja2018/08/14 10:58:4919000.8249850.8311830.833207-----0.0000000.000000NMTNoNMT with Attention
32TMUen-ja2017/08/04 11:06:5117470.8248010.8295800.832569----0.0000000.0000000.000000NMTNobeam_size: 10, ensemble of different dropout rates.
33TMUen-ja2018/09/16 12:44:2324690.8236530.8291560.831219-----0.0000000.000000NMTNoEnsemble of 6 Baseline-NMT
34NAIST-NICTen-ja2017/07/27 21:49:0415060.8219890.8272250.830116----0.0000000.0000000.000000NMTNoSingle Model: SPM16k/16k, BiLSTM Encoder 512*2*2, UniLSTM Decoder 512*2, One-best Search
35TMUen-ja2018/09/16 17:25:3624840.8209610.8267790.829178-----0.0000000.000000NMTNoEnsemble of 6 NMT ( 2 Baseline + 2 Reconstructor + 2 GAN )
36Kyoto-Uen-ja2016/08/18 03:45:5111720.8198360.8238780.828956----0.0000000.0000000.000000NMTNoBPE tgt/src: 52k 2-layer lstm self-ensemble of 3
37TMUen-ja2018/09/16 12:11:5824680.8191480.8253670.827650-----0.0000000.000000NMTNoGAN-NMT ( Single )
38TMUen-ja2018/09/16 12:10:4924670.8179480.8242390.827386-----0.0000000.000000NMTNoBaseline-NMT ( Single )
39TMUen-ja2017/08/02 10:20:1417410.8172800.8243320.825683----0.0000000.0000000.000000NMTNothe ensemble system of different dropout rate.
40TMUen-ja2018/09/16 16:56:4924820.8171530.8240660.826013-----0.0000000.000000NMTNoReconstructor-NMT ( Single )
41UT-AKYen-ja2016/08/20 12:34:4312280.8169840.8244560.827647----0.0000000.0000000.000000NMTNotree-to-seq NMT model (word-based decoder)
42TMUen-ja2017/08/01 11:56:3117090.8129260.8184430.821563----0.0000000.0000000.000000NMTNobaseline system with beam20
43NAISTen-ja2015/08/25 12:39:237610.8114790.8138270.820337---0.0000000.0000000.0000000.000000SMTNoTravatar System with NeuralMT Reranking
44TOKYOMTen-ja2016/08/12 11:21:4311310.8096910.8172580.819951----0.0000000.0000000.000000NMTNochar 1 , ens 2 , version 1
45TOKYOMTen-ja2016/08/19 23:18:4612170.8081890.8144520.818130----0.0000000.0000000.000000NMTNoCombination of NMT and T2S
46naveren-ja2015/08/31 14:12:038360.8072800.8114870.817343---0.0000000.0000000.0000000.000000SMTNoNMT only
47UT-AKYen-ja2016/08/20 09:25:1512240.8060250.8144900.815836----0.0000000.0000000.000000NMTNotree-to-seq NMT model (character-based decoder)
48Wen-ja2015/08/26 16:00:467860.8047220.8090650.814337---0.0000000.0000000.0000000.000000SMTNoNMT, LSTM Search, 5 ensembles, beam size 20, UNK replacing, System Combination with NMT score (Pick top-1k results from NMT)
49TMUen-ja2017/08/01 11:42:0317040.8022620.8096490.811057----0.0000000.0000000.000000NMTNoour baseline system in 2017
50NAISTen-ja2014/08/01 17:37:231260.8017420.8070100.811081---0.0000000.0000000.0000000.000000SMTNoTravatar-based Forest-to-String SMT System (Tuned BLEU+RIBES)
51naveren-ja2015/08/31 14:18:188370.7999660.8031540.808787---0.0000000.0000000.0000000.000000SMTNoSMT t2s + Spell correction + NMT reranking
52Wen-ja2015/08/28 14:30:568130.7968630.8026660.807186---0.0000000.0000000.0000000.000000OtherNoNMT, LSTM Search, Beam Size 20, Ensemble of 2 models, UNK replacing
53Senseen-ja2018/08/24 11:38:2320920.7962140.8007270.803453-----0.0000000.000000NMTNo
54NAISTen-ja2014/07/31 11:38:371180.7960790.8015200.806581---0.0000000.0000000.0000000.000000SMTNoTravatar-based Forest-to-String SMT System
55NAISTen-ja2015/08/25 12:47:497630.7924470.7964890.802228---0.0000000.0000000.0000000.000000SMTNoTravatar System Baseline
56Kyoto-Uen-ja2015/08/30 19:59:188320.7895140.7971820.799979---0.0000000.0000000.0000000.000000EBMTNoKyotoEBMT system with bilingual RNNLM reranking
57Wen-ja2014/08/26 16:17:152020.7850150.7900660.795027---0.0000000.0000000.0000000.000000SMTNoWeblio Pre-reordering SMT System (with forest inputs)
58NICT-2en-ja2016/08/05 17:47:0510970.7843350.7909930.793409----0.0000000.0000000.000000SMTYesPhrase-based SMT with Preordering + Domain Adaptation (JPC and ASPEC) + Google 5-gram LM
59naveren-ja2015/08/25 16:20:307700.7841740.7888110.794452---0.0000000.0000000.0000000.000000SMTNoSMT t2s + Spell correction
60naveren-ja2015/08/04 16:48:325810.7833100.7881650.793723---0.0000000.0000000.0000000.000000SMTNoSMT t2s
61Wen-ja2014/08/16 00:57:161320.7820660.7869020.792616---0.0000000.0000000.0000000.000000SMTNoWeblio Pre-reordering SMT System Baseline
62bjtu_nlpen-ja2016/08/16 11:21:0711430.7805100.7874970.791088----0.0000000.0000000.000000NMTNoRNN Encoder-Decoder with attention mechanism, single model
63ORGANIZERen-ja2016/11/16 10:50:1313340.7767870.7802170.782674----0.0000000.0000000.000000NMTYesOnline A (2016/11/14)
64Kyoto-Uen-ja2016/08/19 10:18:0912010.7712400.7797740.781960----0.0000000.0000000.000000EBMTNoKyotoEBMT 2016 w/o reranking
65TOSHIBAen-ja2015/07/28 16:24:375240.7709890.7785700.780467---0.0000000.0000000.0000000.000000SMT and RBMTYesRBMT with SPE(Statistical Post Editing) system
66TOSHIBAen-ja2015/07/30 11:22:115400.7698650.7768410.779574---0.0000000.0000000.0000000.000000SMT and RBMTYesSystem combination SMT and RBMT(SPE) with RNNLM language model
67EHRen-ja2015/09/09 12:22:588740.7685630.7765260.778113---0.0000000.0000000.0000000.000000SMT and RBMTYesRBMT with user dictionary plus SPE.
68Kyoto-Uen-ja2015/08/27 22:56:138050.7677780.7766720.778358---0.0000000.0000000.0000000.000000EBMTNoKyotoEBMT system without reranking
69EHRen-ja2015/09/09 12:21:348730.7657000.7748190.778269---0.0000000.0000000.0000000.000000SMT and RBMTYesSystem combination of RBMT with user dictionary plus SPE and phrase based SMT with preordering. Candidate selection by language model score.
70EHRen-ja2016/08/15 11:21:5111400.7599140.7714270.775023----0.0000000.0000000.000000SMTNoPBSMT with preordering (DL=6)
71Kyoto-Uen-ja2014/09/01 21:06:322670.7596080.7709080.771545---0.0000000.0000000.0000000.000000EBMTNoOur new baseline system after several modifications + 20-best parses, KN7, RNNLM reranking
72SAS_MTen-ja2014/09/01 10:39:272640.7594150.7709480.771605---0.0000000.0000000.0000000.000000SMTNoSyntactic reordering Hierarchical SMT (using part of data)
73Kyoto-Uen-ja2014/08/25 14:16:151860.7556290.7652510.766495---0.0000000.0000000.0000000.000000EBMTNoUsing n-best parses and RNNLM.
74EHRen-ja2015/08/22 12:28:157420.7535760.7660440.768105---0.0000000.0000000.0000000.000000SMTNoPhrase based SMT with preordering.
75Kyoto-Uen-ja2014/08/31 10:33:232530.7520580.7640490.766435---0.0000000.0000000.0000000.000000EBMTNoOur new baseline system after several modifications.
76ORGANIZERen-ja2014/07/11 20:03:21120.7488830.7580310.760516---0.0000000.0000000.0000000.000000SMTNoTree-to-String SMT (2014)
77ORGANIZERen-ja2015/09/10 13:29:358750.7488830.7580310.760516---0.0000000.0000000.0000000.000000SMTNoTree-to-String SMT (2015)
78Kyoto-Uen-ja2014/08/19 10:16:021340.7439690.7557440.756545---0.0000000.0000000.0000000.000000EBMTNoOur baseline system using 3M parallel sentences.
79ORGANIZERen-ja2014/09/16 13:36:353670.7347050.7469780.747722---0.0000000.0000000.0000000.000000SMTNoHierarchical Phrase-based SMT (2014)
80JAPIOen-ja2016/08/17 12:50:5011650.7234670.7285840.731474----0.0000000.0000000.000000SMTYesPhrase-based SMT with Preordering + JAPIO corpus + rule-based posteditor
81ORGANIZERen-ja2014/07/18 11:02:25340.7180190.7234860.725848---0.0000000.0000000.0000000.000000OtherYesOnline A (2014)
82Osaka-Uen-ja2018/09/16 13:06:4524700.7168890.7264690.729323-----0.0000000.000000SMTNopreordering with neural network
83ORGANIZERen-ja2016/07/26 11:31:4710410.7066390.7152220.718559----0.0000000.0000000.000000OtherYesOnline A (2016)
84ORGANIZERen-ja2015/08/25 18:54:297740.7058820.7139600.718150---0.0000000.0000000.0000000.000000OtherYesOnline A (2015)
85ORGANIZERen-ja2015/09/10 19:02:388890.6933590.7019660.703859---0.0000000.0000000.0000000.000000OtherYesOnline B (2015)
86Senseen-ja2014/08/25 01:07:271840.6904640.7005830.703049---0.0000000.0000000.0000000.000000SMTNoBaseline SMT
87Senseen-ja2015/08/28 19:22:248210.6893130.6978170.700994---0.0000000.0000000.0000000.000000SMTNoBaseline-2015 (train1 only)
88ORGANIZERen-ja2014/07/22 13:30:13910.6877970.6933900.698126---0.0000000.0000000.0000000.000000OtherYesOnline B (2014)
89ORGANIZERen-ja2014/07/11 19:49:0350.6837350.6919260.695390---0.0000000.0000000.0000000.000000SMTNoPhrase-based SMT
90ORGANIZERen-ja2014/07/21 11:38:12660.6719580.6807480.682683---0.0000000.0000000.0000000.000000OtherYesRBMT B (2014)
91ORGANIZERen-ja2015/09/10 14:26:288830.6719580.6807480.682683---0.0000000.0000000.0000000.000000OtherYesRBMT B (2015)
92ORGANIZERen-ja2014/07/21 11:40:50680.6701670.6764640.678934---0.0000000.0000000.0000000.000000OtherYesRBMT A
93ORGANIZERen-ja2014/07/23 14:50:44950.6683720.6726450.676018---0.0000000.0000000.0000000.000000OtherYesRBMT C
94Senseen-ja2015/07/28 22:26:555310.6462880.6538140.659505---0.0000000.0000000.0000000.000000SMTNoBaseline-2015
95Senseen-ja2015/08/18 21:52:017000.6373780.6427890.647831---0.0000000.0000000.0000000.000000SMTNoBaseline-dictmt
96Senseen-ja2015/08/18 22:04:097150.6359330.6415170.646682---0.0000000.0000000.0000000.000000SMTYesPassive JSTx3

Notice:

Back to top

AMFM


# Team Task Date/Time DataID AMFM
Method
Other
Resources
System
Description
juman kytea mecab moses-
tokenizer
stanford-
segmenter-
ctb
stanford-
segmenter-
pku
indic-
tokenizer
unuse myseg kmseg
1srcben-ja2018/09/16 15:26:3724790.7810000.7810000.781000-----0.0000000.000000NMTNoTransformer with relative position, ensemble of 3 models.
2srcben-ja2018/09/16 15:52:1724800.7798200.7798200.779820-----0.0000000.000000NMTNoTransformer with relative position, ensemble of 4 models, rerank.
3NICT-5en-ja2018/09/03 16:49:5722190.7795600.7795600.779560-----0.0000000.000000NMTNoBig Bidirectional Transformer. 1.5M sentences only.
4NTTen-ja2019/07/28 15:44:1232360.7749500.7749500.774950-------NMTNoASPEC first 1.5M + Synthetic 1.5M, 6 ensemble
5NTTen-ja2019/07/28 15:55:1832390.7725900.7725900.772590-------NMTYesParaCrawl + (ASPEC first 1.5M + Synthetic 1.5M) * 2 oversampling, fine-tune ASPEC, SINGLE MODEL
6NICT-5en-ja2018/08/22 18:43:0020480.7714000.7714000.771400-----0.0000000.000000NMTNoTensor2Tensor's transformer implementation. Used recurrently stacked layers model logic. Layer recurrence logic: 1-2-3-1-2-3-1-2-3-1-2-3. Codename: Megarecurrence.
7Kyoto-Uen-ja2017/09/04 22:31:1817590.7710200.7710200.771020----0.0000000.0000000.000000SMTNoSyscomb of AIAYN and KNMT
8NICT-5en-ja2018/08/22 18:48:5120500.7705600.7705600.770560-----0.0000000.000000NMTNoRSNMT 6 layer
9NAIST-NICTen-ja2017/07/27 21:50:1515070.7704800.7704800.770480----0.0000000.0000000.000000NMTNoSingle Model: SPM16k/16k, BiLSTM Encoder 512*2*2, UniLSTM Decoder 512*2, Adjusted Search
10srcben-ja2019/07/27 15:50:1932120.7704400.7704400.770440-------NMTNoTransformer(Big) with relative position, sentence-wise smooth, deep transformer, back translation, ensemble of 10 models, rerank.
11ORGANIZERen-ja2017/08/02 01:04:4917370.7686300.7686300.768630----0.0000000.0000000.000000NMTNoGoogle's "Attention Is All You Need"
12NICT-2en-ja2017/07/26 14:03:0014790.7655800.7655800.765580----0.0000000.0000000.000000NMTNoNMT 6 Ensembles * Bi-directional Reranking
13srcben-ja2019/07/27 15:28:3032060.7653300.7653300.765330-------NMTNoTransformer(Big) with relative position, sentence-wise smooth, deep transformer, back translation, ensemble of 7 models, re-rank.
14NTTen-ja2017/08/01 02:22:5616730.7637700.7637700.763770----0.0000000.0000000.000000NMTNoSingle Model: joint BPE 16k, BiLSTM Encoder 512*2*2, LtoR LSTM Decoder 512*2, Beam Search 20 w/ length-based reranking, add 1M pseudo training data (generated from provided training data)
15NAIST-NICTen-ja2017/07/27 21:49:0415060.7633100.7633100.763310----0.0000000.0000000.000000NMTNoSingle Model: SPM16k/16k, BiLSTM Encoder 512*2*2, UniLSTM Decoder 512*2, One-best Search
16Osaka-Uen-ja2018/09/15 23:02:0524390.7631400.7631400.763140-----0.0000000.000000NMTYesrewarding model
17NTTen-ja2017/08/01 15:42:1317290.7621700.7621700.762170----0.0000000.0000000.000000NMTNoEnsemble 8 Models: joint BPE 16k, BiLSTM Encoder 512*2*2, LtoR LSTM Decoder 512*2, Beam Search 20 w/ length-based reranking, add 1M pseudo training data (generated from provided training data)
18NICT-2en-ja2019/07/27 10:57:2631820.7613400.7613400.761340-------NMTNoTransformer, ensemble of 4 models w/ long warm-up and self-training
19AISTAIen-ja2019/08/30 20:23:2433570.7604100.7604100.760410-------NMTNoTransformer, 1.5M sentences, relative position, ensemble of 3 models, by OpenNMT-py.
20KNU_Hyundaien-ja2019/07/27 09:19:3131720.7600500.7600500.760050-------NMTNoTransformer Base, relative position, BT, r2l reranking, ensemble of 3 models
21AISTAIen-ja2019/09/05 08:18:4533730.7599400.7599400.759940-------NMTNoTransformer (big), 1.5M sentences, train_steps=300000, Averaged the last 20 ckpts, by Tensor2Tensor.
22ORGANIZERen-ja2018/08/14 10:58:4919000.7599100.7599100.759910-----0.0000000.000000NMTNoNMT with Attention
23NICT-2en-ja2017/07/26 13:52:3514750.7595700.7595700.759570----0.0000000.0000000.000000NMTNoNMT Single Model: BPE50k, Bi-LSTM(500*2) Encoder, LSTM(1000) Left-to-Right Decoder
24srcben-ja2019/07/25 11:43:0629180.7595100.7595100.759510-------NMTNoTransformer (Big) with relative position, sentence-wise smooth
25NICT-2en-ja2019/07/27 10:56:0231810.7592600.7592600.759260-------NMTNoTransformer, sigle model w/ long warm-up and self-training
26EHRen-ja2018/09/08 12:42:0322450.7587500.7587500.758750-----0.0000000.000000NMTNoSMT reranked NMT
27NAISTen-ja2014/08/01 17:37:231260.7587400.7587400.7587400.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoTravatar-based Forest-to-String SMT System (Tuned BLEU+RIBES)
28NTTen-ja2017/08/01 07:12:2516840.7577400.7577400.757740----0.0000000.0000000.000000NMTNoEnsemble 8 Models: joint BPE 16k, BiLSTM Encoder 512*2*2, LtoR LSTM Decoder 512*2, Beam Search 20 w/ length-based reranking
29AISTAIen-ja2019/07/29 07:33:5632510.7575900.7575900.757590-------NMTNoTransformer (big). 1.5M sentences, train_steps=131000 only. Averaged the last 10 ckpts.
30NTTen-ja2017/07/30 16:16:2516080.7564800.7564800.756480----0.0000000.0000000.000000NMTNoSingle Model: joint BPE 16k, BiLSTM Encoder 512*2*2, LtoR LSTM Decoder 512*2, Beam Search 20 w/ length-based reranking
31Kyoto-Uen-ja2017/08/01 15:49:5117310.7542200.7542200.754220----0.0000000.0000000.000000NMTNoEnsemble of 4 BPE Averaged
32UT-KAYen-ja2017/06/27 16:58:4213690.7530900.7530900.753090----0.0000000.0000000.000000NMTNoHashimoto and Tsuruoka (2017), https://arxiv.org/abs/1702.02265
33NICT-2en-ja2016/08/05 17:47:0510970.7530800.7530800.753080----0.0000000.0000000.000000SMTYesPhrase-based SMT with Preordering + Domain Adaptation (JPC and ASPEC) + Google 5-gram LM
34TMUen-ja2018/09/16 12:44:2324690.7530400.7530400.753040-----0.0000000.000000NMTNoEnsemble of 6 Baseline-NMT
35NAISTen-ja2014/07/31 11:38:371180.7528500.7528500.7528500.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoTravatar-based Forest-to-String SMT System
36TMUen-ja2018/09/16 17:25:3624840.7524200.7524200.752420-----0.0000000.000000NMTNoEnsemble of 6 NMT ( 2 Baseline + 2 Reconstructor + 2 GAN )
37NAISTen-ja2015/08/25 12:47:497630.7522600.7522600.7522600.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoTravatar System Baseline
38TMUen-ja2018/09/16 16:56:4924820.7521000.7521000.752100-----0.0000000.000000NMTNoReconstructor-NMT ( Single )
39TMUen-ja2017/08/02 10:20:1417410.7516600.7516600.751660----0.0000000.0000000.000000NMTNothe ensemble system of different dropout rate.
40TMUen-ja2018/09/16 12:11:5824680.7503500.7503500.750350-----0.0000000.000000NMTNoGAN-NMT ( Single )
41TMUen-ja2017/08/04 11:06:5117470.7494100.7494100.749410----0.0000000.0000000.000000NMTNobeam_size: 10, ensemble of different dropout rates.
42TMUen-ja2018/09/16 12:10:4924670.7491900.7491900.749190-----0.0000000.000000NMTNoBaseline-NMT ( Single )
43NAISTen-ja2015/08/25 12:39:237610.7488700.7488700.7488700.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoTravatar System with NeuralMT Reranking
44Wen-ja2014/08/26 16:17:152020.7476500.7476500.7476500.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoWeblio Pre-reordering SMT System (with forest inputs)
45Kyoto-Uen-ja2016/08/19 10:18:0912010.7475100.7475100.747510----0.0000000.0000000.000000EBMTNoKyotoEBMT 2016 w/o reranking
46UT-IISen-ja2017/08/01 12:03:2217100.7469100.7469100.746910----0.0000000.0000000.000000NMTNoNMT 8 Ensembles with beam search, Sentence Piece, Embedding layer initialization
47EHRen-ja2016/08/15 11:21:5111400.7467200.7467200.746720----0.0000000.0000000.000000SMTNoPBSMT with preordering (DL=6)
48Wen-ja2014/08/16 00:57:161320.7465700.7465700.7465700.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoWeblio Pre-reordering SMT System Baseline
49TMUen-ja2017/08/01 11:56:3117090.7448900.7448900.744890----0.0000000.0000000.000000NMTNobaseline system with beam20
50TOSHIBAen-ja2015/07/30 11:22:115400.7446600.7446600.7446600.0000000.0000000.0000000.0000000.0000000.0000000.000000SMT and RBMTYesSystem combination SMT and RBMT(SPE) with RNNLM language model
51naveren-ja2015/08/31 14:18:188370.7446300.7446300.7446300.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoSMT t2s + Spell correction + NMT reranking
52ORGANIZERen-ja2014/07/11 20:03:21120.7443700.7443700.7443700.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoTree-to-String SMT (2014)
53ORGANIZERen-ja2015/09/10 13:29:358750.7443700.7443700.7443700.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoTree-to-String SMT (2015)
54ORGANIZERen-ja2014/09/16 13:36:353670.7439000.7439000.7439000.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoHierarchical Phrase-based SMT (2014)
55Kyoto-Uen-ja2015/08/30 19:59:188320.7437100.7437100.7437100.0000000.0000000.0000000.0000000.0000000.0000000.000000EBMTNoKyotoEBMT system with bilingual RNNLM reranking
56TOSHIBAen-ja2015/07/28 16:24:375240.7421000.7421000.7421000.0000000.0000000.0000000.0000000.0000000.0000000.000000SMT and RBMTYesRBMT with SPE(Statistical Post Editing) system
57Kyoto-Uen-ja2014/08/31 10:33:232530.7417100.7417100.7417100.0000000.0000000.0000000.0000000.0000000.0000000.000000EBMTNoOur new baseline system after several modifications.
58Senseen-ja2018/08/24 11:38:2320920.7410900.7410900.741090-----0.0000000.000000NMTNo
59Kyoto-Uen-ja2015/08/27 22:56:138050.7410500.7410500.7410500.0000000.0000000.0000000.0000000.0000000.0000000.000000EBMTNoKyotoEBMT system without reranking
60TMUen-ja2017/08/01 11:42:0317040.7406200.7406200.740620----0.0000000.0000000.000000NMTNoour baseline system in 2017
61SAS_MTen-ja2014/09/01 10:39:272640.7405800.7405800.7405800.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoSyntactic reordering Hierarchical SMT (using part of data)
62naveren-ja2015/08/25 16:20:307700.7402100.7402100.7402100.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoSMT t2s + Spell correction
63Kyoto-Uen-ja2016/08/18 03:45:5111720.7387000.7387000.738700----0.0000000.0000000.000000NMTNoBPE tgt/src: 52k 2-layer lstm self-ensemble of 3
64Kyoto-Uen-ja2014/09/01 21:06:322670.7386800.7386800.7386800.0000000.0000000.0000000.0000000.0000000.0000000.000000EBMTNoOur new baseline system after several modifications + 20-best parses, KN7, RNNLM reranking
65Kyoto-Uen-ja2014/08/19 10:16:021340.7378900.7378900.7378900.0000000.0000000.0000000.0000000.0000000.0000000.000000EBMTNoOur baseline system using 3M parallel sentences.
66naveren-ja2015/08/04 16:48:325810.7375900.7375900.7375900.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoSMT t2s
67Senseen-ja2015/08/28 19:22:248210.7375600.7375600.7375600.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoBaseline-2015 (train1 only)
68ORGANIZERen-ja2014/07/11 19:49:0350.7363800.7363800.7363800.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoPhrase-based SMT
69Senseen-ja2014/08/25 01:07:271840.7333600.7333600.7333600.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoBaseline SMT
70Kyoto-Uen-ja2014/08/25 14:16:151860.7323800.7323800.7323800.0000000.0000000.0000000.0000000.0000000.0000000.000000EBMTNoUsing n-best parses and RNNLM.
71Senseen-ja2015/07/28 22:26:555310.7316200.7316200.7316200.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoBaseline-2015
72UT-AKYen-ja2016/08/20 12:34:4312280.7314400.7314400.731440----0.0000000.0000000.000000NMTNotree-to-seq NMT model (word-based decoder)
73Senseen-ja2015/08/18 22:04:097150.7314000.7314000.7314000.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTYesPassive JSTx3
74Senseen-ja2015/08/18 21:52:017000.7292000.7292000.7292000.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoBaseline-dictmt
75ORGANIZERen-ja2016/11/16 10:50:1313340.7270400.7270400.727040----0.0000000.0000000.000000NMTYesOnline A (2016/11/14)
76naveren-ja2015/08/31 14:12:038360.7258600.7258600.7258600.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoNMT only
77Wen-ja2015/08/26 16:00:467860.7253700.7253700.7253700.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoNMT, LSTM Search, 5 ensembles, beam size 20, UNK replacing, System Combination with NMT score (Pick top-1k results from NMT)
78TOKYOMTen-ja2016/08/19 23:18:4612170.7208100.7208100.720810----0.0000000.0000000.000000NMTNoCombination of NMT and T2S
79UT-AKYen-ja2016/08/20 09:25:1512240.7081400.7081400.708140----0.0000000.0000000.000000NMTNotree-to-seq NMT model (character-based decoder)
80Wen-ja2015/08/28 14:30:568130.7053400.7053400.7053400.0000000.0000000.0000000.0000000.0000000.0000000.000000OtherNoNMT, LSTM Search, Beam Size 20, Ensemble of 2 models, UNK replacing
81TOKYOMTen-ja2016/08/12 11:21:4311310.7052100.7052100.705210----0.0000000.0000000.000000NMTNochar 1 , ens 2 , version 1
82Osaka-Uen-ja2018/09/16 13:06:4524700.7050500.7050500.705050-----0.0000000.000000SMTNopreordering with neural network
83bjtu_nlpen-ja2016/08/16 11:21:0711430.7043400.7043400.704340----0.0000000.0000000.000000NMTNoRNN Encoder-Decoder with attention mechanism, single model
84ORGANIZERen-ja2014/07/18 11:02:25340.6954200.6954200.6954200.0000000.0000000.0000000.0000000.0000000.0000000.000000OtherYesOnline A (2014)
85ORGANIZERen-ja2015/08/25 18:54:297740.6772000.6772000.6772000.0000000.0000000.0000000.0000000.0000000.0000000.000000OtherYesOnline A (2015)
86ORGANIZERen-ja2016/07/26 11:31:4710410.6770200.6770200.677020----0.0000000.0000000.000000OtherYesOnline A (2016)
87JAPIOen-ja2016/08/17 12:50:5011650.6607900.6607900.660790----0.0000000.0000000.000000SMTYesPhrase-based SMT with Preordering + JAPIO corpus + rule-based posteditor
88ORGANIZERen-ja2015/09/10 19:02:388890.6461600.6461600.6461600.0000000.0000000.0000000.0000000.0000000.0000000.000000OtherYesOnline B (2015)
89ORGANIZERen-ja2014/07/22 13:30:13910.6430700.6430700.6430700.0000000.0000000.0000000.0000000.0000000.0000000.000000OtherYesOnline B (2014)
90EHRen-ja2015/09/09 12:21:348730.6304100.6304100.6304100.0000000.0000000.0000000.0000000.0000000.0000000.000000SMT and RBMTYesSystem combination of RBMT with user dictionary plus SPE and phrase based SMT with preordering. Candidate selection by language model score.
91ORGANIZERen-ja2014/07/21 11:40:50680.6269400.6269400.6269400.0000000.0000000.0000000.0000000.0000000.0000000.000000OtherYesRBMT A
92ORGANIZERen-ja2014/07/21 11:38:12660.6229300.6229300.6229300.0000000.0000000.0000000.0000000.0000000.0000000.000000OtherYesRBMT B (2014)
93ORGANIZERen-ja2015/09/10 14:26:288830.6229300.6229300.6229300.0000000.0000000.0000000.0000000.0000000.0000000.000000OtherYesRBMT B (2015)
94EHRen-ja2015/08/22 12:28:157420.6205000.6205000.6205000.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoPhrase based SMT with preordering.
95EHRen-ja2015/09/09 12:22:588740.6040900.6040900.6040900.0000000.0000000.0000000.0000000.0000000.0000000.000000SMT and RBMTYesRBMT with user dictionary plus SPE.
96ORGANIZERen-ja2014/07/23 14:50:44950.5943800.5943800.5943800.0000000.0000000.0000000.0000000.0000000.0000000.000000OtherYesRBMT C

Notice:

Back to top

HUMAN (WAT2020)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description

Notice:
Back to top

HUMAN (WAT2019)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description
1NTTen-ja2019/07/28 15:44:12323647.750NMTNoASPEC first 1.5M + Synthetic 1.5M, 6 ensemble
2NICT-2en-ja2019/07/27 10:57:26318242.250NMTNoTransformer, ensemble of 4 models w/ long warm-up and self-training
3srcben-ja2019/07/27 15:50:19321240.000NMTNoTransformer(Big) with relative position, sentence-wise smooth, deep transformer, back translation, ensemble of 10 models, rerank.
4AISTAIen-ja2019/07/29 07:33:56325136.750NMTNoTransformer (big). 1.5M sentences, train_steps=131000 only. Averaged the last 10 ckpts.
5KNU_Hyundaien-ja2019/07/27 09:19:31317236.000NMTNoTransformer Base, relative position, BT, r2l reranking, ensemble of 3 models

Notice:
Back to top

HUMAN (WAT2018)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description
1NICT-5en-ja2018/09/03 16:49:57221928.500NMTNoBig Bidirectional Transformer. 1.5M sentences only.
2srcben-ja2018/09/16 15:26:37247925.000NMTNoTransformer with relative position, ensemble of 3 models.
3NICT-5en-ja2018/08/22 18:43:00204820.250NMTNoTensor2Tensor's transformer implementation. Used recurrently stacked layers model logic. Layer recurrence logic: 1-2-3-1-2-3-1-2-3-1-2-3. Codename: Megarecurrence.
4Osaka-Uen-ja2018/09/15 23:02:0524394.500NMTYesrewarding model
5EHRen-ja2018/09/08 12:42:032245-0.500NMTNoSMT reranked NMT
6TMUen-ja2018/09/16 12:44:232469-12.000NMTNoEnsemble of 6 Baseline-NMT
7Osaka-Uen-ja2018/09/16 13:06:452470-82.250SMTNopreordering with neural network

Notice:
Back to top

HUMAN (WAT2017)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description
1NTTen-ja2017/08/01 15:42:13172975.750NMTNoEnsemble 8 Models: joint BPE 16k, BiLSTM Encoder 512*2*2, LtoR LSTM Decoder 512*2, Beam Search 20 w/ length-based reranking, add 1M pseudo training data (generated from provided training data)
2NICT-2en-ja2017/07/26 14:03:00147974.750NMTNoNMT 6 Ensembles * Bi-directional Reranking
3NTTen-ja2017/08/01 07:12:25168472.250NMTNoEnsemble 8 Models: joint BPE 16k, BiLSTM Encoder 512*2*2, LtoR LSTM Decoder 512*2, Beam Search 20 w/ length-based reranking
4NAIST-NICTen-ja2017/07/27 21:50:15150770.000NMTNoSingle Model: SPM16k/16k, BiLSTM Encoder 512*2*2, UniLSTM Decoder 512*2, Adjusted Search
5Kyoto-Uen-ja2017/08/01 15:49:51173169.750NMTNoEnsemble of 4 BPE Averaged
6ORGANIZERen-ja2017/08/02 01:04:49173769.750NMTNoGoogle's "Attention Is All You Need"
7UT-IISen-ja2017/08/01 12:03:22171068.000NMTNoNMT 8 Ensembles with beam search, Sentence Piece, Embedding layer initialization
8NAIST-NICTen-ja2017/07/27 21:49:04150663.500NMTNoSingle Model: SPM16k/16k, BiLSTM Encoder 512*2*2, UniLSTM Decoder 512*2, One-best Search
9NICT-2en-ja2017/07/26 13:52:35147562.000NMTNoNMT Single Model: BPE50k, Bi-LSTM(500*2) Encoder, LSTM(1000) Left-to-Right Decoder
10TMUen-ja2017/08/01 11:56:31170956.500NMTNobaseline system with beam20
11TMUen-ja2017/08/01 11:42:03170450.750NMTNoour baseline system in 2017

Notice:
Back to top

HUMAN (WAT2016)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description
1ORGANIZERen-ja2016/11/16 10:50:13133474.000NMTYesOnline A (2016/11/14)
2Kyoto-Uen-ja2016/08/18 03:45:51117255.250NMTNoBPE tgt/src: 52k 2-layer lstm self-ensemble of 3
3ORGANIZERen-ja2016/07/26 11:31:47104149.750OtherYesOnline A (2016)
4NICT-2en-ja2016/08/05 17:47:05109741.250SMTYesPhrase-based SMT with Preordering + Domain Adaptation (JPC and ASPEC) + Google 5-gram LM
5bjtu_nlpen-ja2016/08/16 11:21:07114339.500NMTNoRNN Encoder-Decoder with attention mechanism, single model
6EHRen-ja2016/08/15 11:21:51114039.000SMTNoPBSMT with preordering (DL=6)
7UT-AKYen-ja2016/08/20 12:34:43122836.250NMTNotree-to-seq NMT model (word-based decoder)
8TOKYOMTen-ja2016/08/19 23:18:46121730.500NMTNoCombination of NMT and T2S
9TOKYOMTen-ja2016/08/12 11:21:43113129.750NMTNochar 1 , ens 2 , version 1
10UT-AKYen-ja2016/08/20 09:25:15122421.750NMTNotree-to-seq NMT model (character-based decoder)
11JAPIOen-ja2016/08/17 12:50:5011654.250SMTYesPhrase-based SMT with Preordering + JAPIO corpus + rule-based posteditor

Notice:
Back to top

HUMAN (WAT2015)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description
1NAISTen-ja2015/08/25 12:39:2376162.250SMTNoTravatar System with NeuralMT Reranking
2Wen-ja2015/08/26 16:00:4678653.750SMTNoNMT, LSTM Search, 5 ensembles, beam size 20, UNK replacing, System Combination with NMT score (Pick top-1k results from NMT)
3naveren-ja2015/08/31 14:18:1883753.250SMTNoSMT t2s + Spell correction + NMT reranking
4Kyoto-Uen-ja2015/08/30 19:59:1883251.000EBMTNoKyotoEBMT system with bilingual RNNLM reranking
5NAISTen-ja2015/08/25 12:47:4976349.750SMTNoTravatar System Baseline
6naveren-ja2015/08/31 14:12:0383648.500SMTNoNMT only
7Wen-ja2015/08/28 14:30:5681343.500OtherNoNMT, LSTM Search, Beam Size 20, Ensemble of 2 models, UNK replacing
8Kyoto-Uen-ja2015/08/27 22:56:1380540.500EBMTNoKyotoEBMT system without reranking
9TOSHIBAen-ja2015/07/28 16:24:3752440.250SMT and RBMTYesRBMT with SPE(Statistical Post Editing) system
10ORGANIZERen-ja2015/08/25 18:54:2977434.250OtherYesOnline A (2015)
11EHRen-ja2015/08/22 12:28:1574232.500SMTNoPhrase based SMT with preordering.
12ORGANIZERen-ja2015/09/10 13:29:3587530.000SMTNoTree-to-String SMT (2015)
13ORGANIZERen-ja2015/09/10 14:26:288839.750OtherYesRBMT B (2015)
14Senseen-ja2015/08/18 22:04:09715-31.000SMTYesPassive JSTx3
15Senseen-ja2015/08/18 21:52:01700-36.250SMTNoBaseline-dictmt

Notice:
Back to top

HUMAN (WAT2014)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description
1NAISTen-ja2014/07/31 11:38:3711856.250SMTNoTravatar-based Forest-to-String SMT System
2NAISTen-ja2014/08/01 17:37:2312651.500SMTNoTravatar-based Forest-to-String SMT System (Tuned BLEU+RIBES)
3Wen-ja2014/08/16 00:57:1613243.250SMTNoWeblio Pre-reordering SMT System Baseline
4ORGANIZERen-ja2014/07/18 11:02:253442.500OtherYesOnline A (2014)
5Kyoto-Uen-ja2014/09/01 21:06:3226738.000EBMTNoOur new baseline system after several modifications + 20-best parses, KN7, RNNLM reranking
6Wen-ja2014/08/26 16:17:1520236.000SMTNoWeblio Pre-reordering SMT System (with forest inputs)
7ORGANIZERen-ja2014/07/11 20:03:211234.250SMTNoTree-to-String SMT (2014)
8Kyoto-Uen-ja2014/08/31 10:33:2325333.750EBMTNoOur new baseline system after several modifications.
9ORGANIZERen-ja2014/09/16 13:36:3536731.500SMTNoHierarchical Phrase-based SMT (2014)
10SAS_MTen-ja2014/09/01 10:39:2726427.500SMTNoSyntactic reordering Hierarchical SMT (using part of data)
11Senseen-ja2014/08/25 01:07:271843.750SMTNoBaseline SMT
12ORGANIZERen-ja2014/07/21 11:38:12660.750OtherYesRBMT B (2014)

Notice:
Back to top

EVALUATION RESULTS USAGE POLICY

When you use the WAT evaluation results for any purpose such as:
- writing technical papers,
- making presentations about your system,
- advertising your MT system to the customers,
you can use the information about translation directions, scores (including both automatic and human evaluations) and ranks of your system among others. You can also use the scores of the other systems, but you MUST anonymize the other system's names. In addition, you can show the links (URLs) to the WAT evaluation result pages.

NICT (National Institute of Information and Communications Technology)
Kyoto University
Last Modified: 2018-08-02