NICT_LOGO.JPG KYOTO-U_LOGO.JPG

WAT

The Workshop on Asian Translation
Evaluation Results

[EVALUATION RESULTS TOP] | [BLEU] | [RIBES] | [AMFM] | [HUMAN (WAT2022)] | [HUMAN (WAT2021)] | [HUMAN (WAT2020)] | [HUMAN (WAT2019)] | [HUMAN (WAT2018)] | [HUMAN (WAT2017)] | [HUMAN (WAT2016)] | [HUMAN (WAT2015)] | [HUMAN (WAT2014)] | [EVALUATION RESULTS USAGE POLICY]

BLEU


# Team Task Date/Time DataID BLEU
Method
Other
Resources
System
Description
juman kytea mecab moses-
tokenizer
stanford-
segmenter-
ctb
stanford-
segmenter-
pku
indic-
tokenizer
unuse myseg kmseg
1ORGANIZERen-ja2014/07/11 19:49:03527.4829.8028.27--- 0.00 0.00 0.00 0.00SMTNoPhrase-based SMT
2ORGANIZERen-ja2014/07/11 20:03:211231.0533.4432.10--- 0.00 0.00 0.00 0.00SMTNoTree-to-String SMT (2014)
3ORGANIZERen-ja2014/07/18 11:02:253419.6621.6320.17--- 0.00 0.00 0.00 0.00OtherYesOnline A (2014)
4ORGANIZERen-ja2014/07/21 11:38:126613.1814.8513.48--- 0.00 0.00 0.00 0.00OtherYesRBMT B (2014)
5ORGANIZERen-ja2014/07/21 11:40:506812.8614.4313.16--- 0.00 0.00 0.00 0.00OtherYesRBMT A
6ORGANIZERen-ja2014/07/22 13:30:139117.0418.6717.36--- 0.00 0.00 0.00 0.00OtherYesOnline B (2014)
7ORGANIZERen-ja2014/07/23 14:50:449512.1913.3212.14--- 0.00 0.00 0.00 0.00OtherYesRBMT C
8NAISTen-ja2014/07/31 11:38:3711835.0337.1635.81--- 0.00 0.00 0.00 0.00SMTNoTravatar-based Forest-to-String SMT System
9NAISTen-ja2014/08/01 17:37:2312634.8437.1535.67--- 0.00 0.00 0.00 0.00SMTNoTravatar-based Forest-to-String SMT System (Tuned BLEU+RIBES)
10Wen-ja2014/08/16 00:57:1613232.5334.8733.26--- 0.00 0.00 0.00 0.00SMTNoWeblio Pre-reordering SMT System Baseline
11Kyoto-Uen-ja2014/08/19 10:16:0213428.9331.6129.59--- 0.00 0.00 0.00 0.00EBMTNoOur baseline system using 3M parallel sentences.
12Senseen-ja2014/08/25 01:07:2718427.9230.1828.66--- 0.00 0.00 0.00 0.00SMTNoBaseline SMT
13Kyoto-Uen-ja2014/08/25 14:16:1518630.2532.7830.84--- 0.00 0.00 0.00 0.00EBMTNoUsing n-best parses and RNNLM.
14Wen-ja2014/08/26 16:17:1520232.6935.0433.40--- 0.00 0.00 0.00 0.00SMTNoWeblio Pre-reordering SMT System (with forest inputs)
15Kyoto-Uen-ja2014/08/31 10:33:2325329.7632.4630.46--- 0.00 0.00 0.00 0.00EBMTNoOur new baseline system after several modifications.
16SAS_MTen-ja2014/09/01 10:39:2726430.4733.0031.47--- 0.00 0.00 0.00 0.00SMTNoSyntactic reordering Hierarchical SMT (using part of data)
17Kyoto-Uen-ja2014/09/01 21:06:3226731.0933.5531.73--- 0.00 0.00 0.00 0.00EBMTNoOur new baseline system after several modifications + 20-best parses, KN7, RNNLM reranking
18ORGANIZERen-ja2014/09/16 13:36:3536730.1932.5630.94--- 0.00 0.00 0.00 0.00SMTNoHierarchical Phrase-based SMT (2014)
19TOSHIBAen-ja2015/07/28 16:24:3752432.0634.1732.76--- 0.00 0.00 0.00 0.00SMT and RBMTYesRBMT with SPE(Statistical Post Editing) system
20Senseen-ja2015/07/28 22:26:5553125.2227.2225.90--- 0.00 0.00 0.00 0.00SMTNoBaseline-2015
21TOSHIBAen-ja2015/07/30 11:22:1154031.8233.9432.54--- 0.00 0.00 0.00 0.00SMT and RBMTYesSystem combination SMT and RBMT(SPE) with RNNLM language model
22naveren-ja2015/08/04 16:48:3258132.6334.4633.34--- 0.00 0.00 0.00 0.00SMTNoSMT t2s
23Senseen-ja2015/08/18 21:52:0170024.1326.2424.96--- 0.00 0.00 0.00 0.00SMTNoBaseline-dictmt
24Senseen-ja2015/08/18 22:04:0971524.4326.5825.36--- 0.00 0.00 0.00 0.00SMTYesPassive JSTx3
25EHRen-ja2015/08/22 12:28:1574229.7832.3630.71--- 0.00 0.00 0.00 0.00SMTNoPhrase based SMT with preordering.
26NAISTen-ja2015/08/25 12:39:2376135.8338.1736.61--- 0.00 0.00 0.00 0.00SMTNoTravatar System with NeuralMT Reranking
27NAISTen-ja2015/08/25 12:47:4976334.3836.5835.16--- 0.00 0.00 0.00 0.00SMTNoTravatar System Baseline
28naveren-ja2015/08/25 16:20:3077032.7634.5333.47--- 0.00 0.00 0.00 0.00SMTNoSMT t2s + Spell correction
29ORGANIZERen-ja2015/08/25 18:54:2977418.2219.7718.46--- 0.00 0.00 0.00 0.00OtherYesOnline A (2015)
30Wen-ja2015/08/26 16:00:4678633.2336.2134.05--- 0.00 0.00 0.00 0.00SMTNoNMT, LSTM Search, 5 ensembles, beam size 20, UNK replacing, System Combination with NMT score (Pick top-1k results from NMT)
31Kyoto-Uen-ja2015/08/27 22:56:1380530.6933.2531.71--- 0.00 0.00 0.00 0.00EBMTNoKyotoEBMT system without reranking
32Wen-ja2015/08/28 14:30:5681330.7234.1931.57--- 0.00 0.00 0.00 0.00OtherNoNMT, LSTM Search, Beam Size 20, Ensemble of 2 models, UNK replacing
33Senseen-ja2015/08/28 19:22:2482127.9130.0328.78--- 0.00 0.00 0.00 0.00SMTNoBaseline-2015 (train1 only)
34Kyoto-Uen-ja2015/08/30 19:59:1883233.0635.5733.99--- 0.00 0.00 0.00 0.00EBMTNoKyotoEBMT system with bilingual RNNLM reranking
35naveren-ja2015/08/31 14:12:0383633.1435.7533.93--- 0.00 0.00 0.00 0.00SMTNoNMT only
36naveren-ja2015/08/31 14:18:1883734.6036.1435.30--- 0.00 0.00 0.00 0.00SMTNoSMT t2s + Spell correction + NMT reranking
37EHRen-ja2015/09/09 12:21:3487330.8833.2031.74--- 0.00 0.00 0.00 0.00SMT and RBMTYesSystem combination of RBMT with user dictionary plus SPE and phrase based SMT with preordering. Candidate selection by language model score.
38EHRen-ja2015/09/09 12:22:5887430.4632.5631.28--- 0.00 0.00 0.00 0.00SMT and RBMTYesRBMT with user dictionary plus SPE.
39ORGANIZERen-ja2015/09/10 13:29:3587531.0533.4432.10--- 0.00 0.00 0.00 0.00SMTNoTree-to-String SMT (2015)
40ORGANIZERen-ja2015/09/10 14:26:2888313.1814.8513.48--- 0.00 0.00 0.00 0.00OtherYesRBMT B (2015)
41ORGANIZERen-ja2015/09/10 19:02:3888917.8019.5218.11--- 0.00 0.00 0.00 0.00OtherYesOnline B (2015)
42ORGANIZERen-ja2016/07/26 11:31:47104118.2819.8118.51---- 0.00 0.00 0.00OtherYesOnline A (2016)
43NICT-2en-ja2016/08/05 17:47:05109734.6736.8635.37---- 0.00 0.00 0.00SMTYesPhrase-based SMT with Preordering + Domain Adaptation (JPC and ASPEC) + Google 5-gram LM
44TOKYOMTen-ja2016/08/12 11:21:43113130.2133.3831.24---- 0.00 0.00 0.00NMTNochar 1 , ens 2 , version 1
45EHRen-ja2016/08/15 11:21:51114031.3233.5832.28---- 0.00 0.00 0.00SMTNoPBSMT with preordering (DL=6)
46bjtu_nlpen-ja2016/08/16 11:21:07114331.1833.4731.80---- 0.00 0.00 0.00NMTNoRNN Encoder-Decoder with attention mechanism, single model
47JAPIOen-ja2016/08/17 12:50:50116520.5222.5621.05---- 0.00 0.00 0.00SMTYesPhrase-based SMT with Preordering + JAPIO corpus + rule-based posteditor
48Kyoto-Uen-ja2016/08/18 03:45:51117236.1938.2036.78---- 0.00 0.00 0.00NMTNoBPE tgt/src: 52k 2-layer lstm self-ensemble of 3
49Kyoto-Uen-ja2016/08/19 10:18:09120131.0333.4031.88---- 0.00 0.00 0.00EBMTNoKyotoEBMT 2016 w/o reranking
50TOKYOMTen-ja2016/08/19 23:18:46121732.0334.7732.98---- 0.00 0.00 0.00NMTNoCombination of NMT and T2S
51UT-AKYen-ja2016/08/20 09:25:15122430.1433.2031.09---- 0.00 0.00 0.00NMTNotree-to-seq NMT model (character-based decoder)
52UT-AKYen-ja2016/08/20 12:34:43122833.5736.9534.65---- 0.00 0.00 0.00NMTNotree-to-seq NMT model (word-based decoder)
53ORGANIZERen-ja2016/11/16 10:50:13133426.1928.2226.68---- 0.00 0.00 0.00NMTYesOnline A (2016/11/14)
54UT-KAYen-ja2017/06/27 16:58:42136938.3241.1239.32---- 0.00 0.00 0.00NMTNoHashimoto and Tsuruoka (2017), https://arxiv.org/abs/1702.02265
55NICT-2en-ja2017/07/26 13:52:35147536.8538.9437.87---- 0.00 0.00 0.00NMTNoNMT Single Model: BPE50k, Bi-LSTM(500*2) Encoder, LSTM(1000) Left-to-Right Decoder
56NICT-2en-ja2017/07/26 14:03:00147940.1742.2541.17---- 0.00 0.00 0.00NMTNoNMT 6 Ensembles * Bi-directional Reranking
57NAIST-NICTen-ja2017/07/27 21:49:04150636.4738.5437.30---- 0.00 0.00 0.00NMTNoSingle Model: SPM16k/16k, BiLSTM Encoder 512*2*2, UniLSTM Decoder 512*2, One-best Search
58NAIST-NICTen-ja2017/07/27 21:50:15150738.2540.2939.05---- 0.00 0.00 0.00NMTNoSingle Model: SPM16k/16k, BiLSTM Encoder 512*2*2, UniLSTM Decoder 512*2, Adjusted Search
59NTTen-ja2017/07/30 16:16:25160837.9040.4838.61---- 0.00 0.00 0.00NMTNoSingle Model: joint BPE 16k, BiLSTM Encoder 512*2*2, LtoR LSTM Decoder 512*2, Beam Search 20 w/ length-based reranking
60NTTen-ja2017/08/01 02:22:56167338.8741.4239.71---- 0.00 0.00 0.00NMTNoSingle Model: joint BPE 16k, BiLSTM Encoder 512*2*2, LtoR LSTM Decoder 512*2, Beam Search 20 w/ length-based reranking, add 1M pseudo training data (generated from provided training data)
61NTTen-ja2017/08/01 07:12:25168439.8042.2740.47---- 0.00 0.00 0.00NMTNoEnsemble 8 Models: joint BPE 16k, BiLSTM Encoder 512*2*2, LtoR LSTM Decoder 512*2, Beam Search 20 w/ length-based reranking
62TMUen-ja2017/08/01 11:42:03170432.6535.0533.72---- 0.00 0.00 0.00NMTNoour baseline system in 2017
63TMUen-ja2017/08/01 11:56:31170934.0536.6935.32---- 0.00 0.00 0.00NMTNobaseline system with beam20
64UT-IISen-ja2017/08/01 12:03:22171036.2638.9337.06---- 0.00 0.00 0.00NMTNoNMT 8 Ensembles with beam search, Sentence Piece, Embedding layer initialization
65NTTen-ja2017/08/01 15:42:13172940.3242.8040.95---- 0.00 0.00 0.00NMTNoEnsemble 8 Models: joint BPE 16k, BiLSTM Encoder 512*2*2, LtoR LSTM Decoder 512*2, Beam Search 20 w/ length-based reranking, add 1M pseudo training data (generated from provided training data)
66Kyoto-Uen-ja2017/08/01 15:49:51173138.7240.6539.37---- 0.00 0.00 0.00NMTNoEnsemble of 4 BPE Averaged
67ORGANIZERen-ja2017/08/02 01:04:49173740.7942.5541.50---- 0.00 0.00 0.00NMTNoGoogle's "Attention Is All You Need"
68TMUen-ja2017/08/02 10:20:14174134.5836.9535.63---- 0.00 0.00 0.00NMTNothe ensemble system of different dropout rate.
69TMUen-ja2017/08/04 11:06:51174735.7338.3437.00---- 0.00 0.00 0.00NMTNobeam_size: 10, ensemble of different dropout rates.
70Kyoto-Uen-ja2017/09/04 22:31:18175941.5343.2642.13---- 0.00 0.00 0.00SMTNoSyscomb of AIAYN and KNMT
71ORGANIZERen-ja2018/08/14 10:58:49190036.3738.4837.15----- 0.00 0.00NMTNoNMT with Attention
72NICT-5en-ja2018/08/22 18:43:00204841.9143.5042.60----- 0.00 0.00NMTNoTensor2Tensor's transformer implementation. Used recurrently stacked layers model logic. Layer recurrence logic: 1-2-3-1-2-3-1-2-3-1-2-3. Codename: Megarecurrence.
73NICT-5en-ja2018/08/22 18:48:51205040.0241.7440.77----- 0.00 0.00NMTNoRSNMT 6 layer
74Senseen-ja2018/08/24 11:38:23209235.2937.2136.02----- 0.00 0.00NMTNo
75NICT-5en-ja2018/09/03 16:49:57221942.8744.4243.49----- 0.00 0.00NMTNoBig Bidirectional Transformer. 1.5M sentences only.
76EHRen-ja2018/09/08 12:42:03224537.9740.0038.66----- 0.00 0.00NMTNoSMT reranked NMT
77Osaka-Uen-ja2018/09/15 23:02:05243938.0140.0039.10----- 0.00 0.00NMTYesrewarding model
78TMUen-ja2018/09/16 12:10:49246734.0236.6935.17----- 0.00 0.00NMTNoBaseline-NMT ( Single )
79TMUen-ja2018/09/16 12:11:58246834.0036.6535.09----- 0.00 0.00NMTNoGAN-NMT ( Single )
80TMUen-ja2018/09/16 12:44:23246935.0837.6936.14----- 0.00 0.00NMTNoEnsemble of 6 Baseline-NMT
81Osaka-Uen-ja2018/09/16 13:06:45247023.2425.5024.26----- 0.00 0.00SMTNopreordering with neural network
82srcben-ja2018/09/16 15:26:37247942.4944.1143.20----- 0.00 0.00NMTNoTransformer with relative position, ensemble of 3 models.
83srcben-ja2018/09/16 15:52:17248043.4344.9444.05----- 0.00 0.00NMTNoTransformer with relative position, ensemble of 4 models, rerank.
84TMUen-ja2018/09/16 16:56:49248233.8636.5834.89----- 0.00 0.00NMTNoReconstructor-NMT ( Single )
85TMUen-ja2018/09/16 17:25:36248434.3837.0635.44----- 0.00 0.00NMTNoEnsemble of 6 NMT ( 2 Baseline + 2 Reconstructor + 2 GAN )
86srcben-ja2019/07/25 11:43:06291843.6045.4944.29-------NMTNoTransformer (Big) with relative position, sentence-wise smooth
87KNU_Hyundaien-ja2019/07/27 09:19:31317244.0845.8844.78-------NMTNoTransformer Base, relative position, BT, r2l reranking, ensemble of 3 models
88NICT-2en-ja2019/07/27 10:56:02318143.7945.8244.83-------NMTNoTransformer, sigle model w/ long warm-up and self-training
89NICT-2en-ja2019/07/27 10:57:26318244.6146.5945.66-------NMTNoTransformer, ensemble of 4 models w/ long warm-up and self-training
90srcben-ja2019/07/27 15:28:30320645.3047.1245.93-------NMTNoTransformer(Big) with relative position, sentence-wise smooth, deep transformer, back translation, ensemble of 7 models, re-rank.
91srcben-ja2019/07/27 15:50:19321245.7147.5546.29-------NMTNoTransformer(Big) with relative position, sentence-wise smooth, deep transformer, back translation, ensemble of 10 models, rerank.
92NTTen-ja2019/07/28 15:44:12323645.8347.6346.57-------NMTNoASPEC first 1.5M + Synthetic 1.5M, 6 ensemble
93NTTen-ja2019/07/28 15:55:18323945.3047.1346.04-------NMTYesParaCrawl + (ASPEC first 1.5M + Synthetic 1.5M) * 2 oversampling, fine-tune ASPEC, SINGLE MODEL
94AISTAIen-ja2019/07/29 07:33:56325142.6444.1743.34-------NMTNoTransformer (big). 1.5M sentences, train_steps=131000 only. Averaged the last 10 ckpts.
95AISTAIen-ja2019/08/30 20:23:24335743.7645.7044.52-------NMTNoTransformer, 1.5M sentences, relative position, ensemble of 3 models, by OpenNMT-py.
96AISTAIen-ja2019/09/05 08:18:45337342.9244.9043.67-------NMTNoTransformer (big), 1.5M sentences, train_steps=300000, Averaged the last 20 ckpts, by Tensor2Tensor.

Notice:

Back to top

RIBES


# Team Task Date/Time DataID RIBES
Method
Other
Resources
System
Description
juman kytea mecab moses-
tokenizer
stanford-
segmenter-
ctb
stanford-
segmenter-
pku
indic-
tokenizer
unuse myseg kmseg
1ORGANIZERen-ja2014/07/11 19:49:0350.6837350.6919260.695390---0.0000000.0000000.0000000.000000SMTNoPhrase-based SMT
2ORGANIZERen-ja2014/07/11 20:03:21120.7488830.7580310.760516---0.0000000.0000000.0000000.000000SMTNoTree-to-String SMT (2014)
3ORGANIZERen-ja2014/07/18 11:02:25340.7180190.7234860.725848---0.0000000.0000000.0000000.000000OtherYesOnline A (2014)
4ORGANIZERen-ja2014/07/21 11:38:12660.6719580.6807480.682683---0.0000000.0000000.0000000.000000OtherYesRBMT B (2014)
5ORGANIZERen-ja2014/07/21 11:40:50680.6701670.6764640.678934---0.0000000.0000000.0000000.000000OtherYesRBMT A
6ORGANIZERen-ja2014/07/22 13:30:13910.6877970.6933900.698126---0.0000000.0000000.0000000.000000OtherYesOnline B (2014)
7ORGANIZERen-ja2014/07/23 14:50:44950.6683720.6726450.676018---0.0000000.0000000.0000000.000000OtherYesRBMT C
8NAISTen-ja2014/07/31 11:38:371180.7960790.8015200.806581---0.0000000.0000000.0000000.000000SMTNoTravatar-based Forest-to-String SMT System
9NAISTen-ja2014/08/01 17:37:231260.8017420.8070100.811081---0.0000000.0000000.0000000.000000SMTNoTravatar-based Forest-to-String SMT System (Tuned BLEU+RIBES)
10Wen-ja2014/08/16 00:57:161320.7820660.7869020.792616---0.0000000.0000000.0000000.000000SMTNoWeblio Pre-reordering SMT System Baseline
11Kyoto-Uen-ja2014/08/19 10:16:021340.7439690.7557440.756545---0.0000000.0000000.0000000.000000EBMTNoOur baseline system using 3M parallel sentences.
12Senseen-ja2014/08/25 01:07:271840.6904640.7005830.703049---0.0000000.0000000.0000000.000000SMTNoBaseline SMT
13Kyoto-Uen-ja2014/08/25 14:16:151860.7556290.7652510.766495---0.0000000.0000000.0000000.000000EBMTNoUsing n-best parses and RNNLM.
14Wen-ja2014/08/26 16:17:152020.7850150.7900660.795027---0.0000000.0000000.0000000.000000SMTNoWeblio Pre-reordering SMT System (with forest inputs)
15Kyoto-Uen-ja2014/08/31 10:33:232530.7520580.7640490.766435---0.0000000.0000000.0000000.000000EBMTNoOur new baseline system after several modifications.
16SAS_MTen-ja2014/09/01 10:39:272640.7594150.7709480.771605---0.0000000.0000000.0000000.000000SMTNoSyntactic reordering Hierarchical SMT (using part of data)
17Kyoto-Uen-ja2014/09/01 21:06:322670.7596080.7709080.771545---0.0000000.0000000.0000000.000000EBMTNoOur new baseline system after several modifications + 20-best parses, KN7, RNNLM reranking
18ORGANIZERen-ja2014/09/16 13:36:353670.7347050.7469780.747722---0.0000000.0000000.0000000.000000SMTNoHierarchical Phrase-based SMT (2014)
19TOSHIBAen-ja2015/07/28 16:24:375240.7709890.7785700.780467---0.0000000.0000000.0000000.000000SMT and RBMTYesRBMT with SPE(Statistical Post Editing) system
20Senseen-ja2015/07/28 22:26:555310.6462880.6538140.659505---0.0000000.0000000.0000000.000000SMTNoBaseline-2015
21TOSHIBAen-ja2015/07/30 11:22:115400.7698650.7768410.779574---0.0000000.0000000.0000000.000000SMT and RBMTYesSystem combination SMT and RBMT(SPE) with RNNLM language model
22naveren-ja2015/08/04 16:48:325810.7833100.7881650.793723---0.0000000.0000000.0000000.000000SMTNoSMT t2s
23Senseen-ja2015/08/18 21:52:017000.6373780.6427890.647831---0.0000000.0000000.0000000.000000SMTNoBaseline-dictmt
24Senseen-ja2015/08/18 22:04:097150.6359330.6415170.646682---0.0000000.0000000.0000000.000000SMTYesPassive JSTx3
25EHRen-ja2015/08/22 12:28:157420.7535760.7660440.768105---0.0000000.0000000.0000000.000000SMTNoPhrase based SMT with preordering.
26NAISTen-ja2015/08/25 12:39:237610.8114790.8138270.820337---0.0000000.0000000.0000000.000000SMTNoTravatar System with NeuralMT Reranking
27NAISTen-ja2015/08/25 12:47:497630.7924470.7964890.802228---0.0000000.0000000.0000000.000000SMTNoTravatar System Baseline
28naveren-ja2015/08/25 16:20:307700.7841740.7888110.794452---0.0000000.0000000.0000000.000000SMTNoSMT t2s + Spell correction
29ORGANIZERen-ja2015/08/25 18:54:297740.7058820.7139600.718150---0.0000000.0000000.0000000.000000OtherYesOnline A (2015)
30Wen-ja2015/08/26 16:00:467860.8047220.8090650.814337---0.0000000.0000000.0000000.000000SMTNoNMT, LSTM Search, 5 ensembles, beam size 20, UNK replacing, System Combination with NMT score (Pick top-1k results from NMT)
31Kyoto-Uen-ja2015/08/27 22:56:138050.7677780.7766720.778358---0.0000000.0000000.0000000.000000EBMTNoKyotoEBMT system without reranking
32Wen-ja2015/08/28 14:30:568130.7968630.8026660.807186---0.0000000.0000000.0000000.000000OtherNoNMT, LSTM Search, Beam Size 20, Ensemble of 2 models, UNK replacing
33Senseen-ja2015/08/28 19:22:248210.6893130.6978170.700994---0.0000000.0000000.0000000.000000SMTNoBaseline-2015 (train1 only)
34Kyoto-Uen-ja2015/08/30 19:59:188320.7895140.7971820.799979---0.0000000.0000000.0000000.000000EBMTNoKyotoEBMT system with bilingual RNNLM reranking
35naveren-ja2015/08/31 14:12:038360.8072800.8114870.817343---0.0000000.0000000.0000000.000000SMTNoNMT only
36naveren-ja2015/08/31 14:18:188370.7999660.8031540.808787---0.0000000.0000000.0000000.000000SMTNoSMT t2s + Spell correction + NMT reranking
37EHRen-ja2015/09/09 12:21:348730.7657000.7748190.778269---0.0000000.0000000.0000000.000000SMT and RBMTYesSystem combination of RBMT with user dictionary plus SPE and phrase based SMT with preordering. Candidate selection by language model score.
38EHRen-ja2015/09/09 12:22:588740.7685630.7765260.778113---0.0000000.0000000.0000000.000000SMT and RBMTYesRBMT with user dictionary plus SPE.
39ORGANIZERen-ja2015/09/10 13:29:358750.7488830.7580310.760516---0.0000000.0000000.0000000.000000SMTNoTree-to-String SMT (2015)
40ORGANIZERen-ja2015/09/10 14:26:288830.6719580.6807480.682683---0.0000000.0000000.0000000.000000OtherYesRBMT B (2015)
41ORGANIZERen-ja2015/09/10 19:02:388890.6933590.7019660.703859---0.0000000.0000000.0000000.000000OtherYesOnline B (2015)
42ORGANIZERen-ja2016/07/26 11:31:4710410.7066390.7152220.718559----0.0000000.0000000.000000OtherYesOnline A (2016)
43NICT-2en-ja2016/08/05 17:47:0510970.7843350.7909930.793409----0.0000000.0000000.000000SMTYesPhrase-based SMT with Preordering + Domain Adaptation (JPC and ASPEC) + Google 5-gram LM
44TOKYOMTen-ja2016/08/12 11:21:4311310.8096910.8172580.819951----0.0000000.0000000.000000NMTNochar 1 , ens 2 , version 1
45EHRen-ja2016/08/15 11:21:5111400.7599140.7714270.775023----0.0000000.0000000.000000SMTNoPBSMT with preordering (DL=6)
46bjtu_nlpen-ja2016/08/16 11:21:0711430.7805100.7874970.791088----0.0000000.0000000.000000NMTNoRNN Encoder-Decoder with attention mechanism, single model
47JAPIOen-ja2016/08/17 12:50:5011650.7234670.7285840.731474----0.0000000.0000000.000000SMTYesPhrase-based SMT with Preordering + JAPIO corpus + rule-based posteditor
48Kyoto-Uen-ja2016/08/18 03:45:5111720.8198360.8238780.828956----0.0000000.0000000.000000NMTNoBPE tgt/src: 52k 2-layer lstm self-ensemble of 3
49Kyoto-Uen-ja2016/08/19 10:18:0912010.7712400.7797740.781960----0.0000000.0000000.000000EBMTNoKyotoEBMT 2016 w/o reranking
50TOKYOMTen-ja2016/08/19 23:18:4612170.8081890.8144520.818130----0.0000000.0000000.000000NMTNoCombination of NMT and T2S
51UT-AKYen-ja2016/08/20 09:25:1512240.8060250.8144900.815836----0.0000000.0000000.000000NMTNotree-to-seq NMT model (character-based decoder)
52UT-AKYen-ja2016/08/20 12:34:4312280.8169840.8244560.827647----0.0000000.0000000.000000NMTNotree-to-seq NMT model (word-based decoder)
53ORGANIZERen-ja2016/11/16 10:50:1313340.7767870.7802170.782674----0.0000000.0000000.000000NMTYesOnline A (2016/11/14)
54UT-KAYen-ja2017/06/27 16:58:4213690.8296000.8339200.838260----0.0000000.0000000.000000NMTNoHashimoto and Tsuruoka (2017), https://arxiv.org/abs/1702.02265
55NICT-2en-ja2017/07/26 13:52:3514750.8267910.8344480.835255----0.0000000.0000000.000000NMTNoNMT Single Model: BPE50k, Bi-LSTM(500*2) Encoder, LSTM(1000) Left-to-Right Decoder
56NICT-2en-ja2017/07/26 14:03:0014790.8422060.8481700.849929----0.0000000.0000000.000000NMTNoNMT 6 Ensembles * Bi-directional Reranking
57NAIST-NICTen-ja2017/07/27 21:49:0415060.8219890.8272250.830116----0.0000000.0000000.000000NMTNoSingle Model: SPM16k/16k, BiLSTM Encoder 512*2*2, UniLSTM Decoder 512*2, One-best Search
58NAIST-NICTen-ja2017/07/27 21:50:1515070.8344920.8393210.842337----0.0000000.0000000.000000NMTNoSingle Model: SPM16k/16k, BiLSTM Encoder 512*2*2, UniLSTM Decoder 512*2, Adjusted Search
59NTTen-ja2017/07/30 16:16:2516080.8281930.8320060.836381----0.0000000.0000000.000000NMTNoSingle Model: joint BPE 16k, BiLSTM Encoder 512*2*2, LtoR LSTM Decoder 512*2, Beam Search 20 w/ length-based reranking
60NTTen-ja2017/08/01 02:22:5616730.8335410.8382450.841739----0.0000000.0000000.000000NMTNoSingle Model: joint BPE 16k, BiLSTM Encoder 512*2*2, LtoR LSTM Decoder 512*2, Beam Search 20 w/ length-based reranking, add 1M pseudo training data (generated from provided training data)
61NTTen-ja2017/08/01 07:12:2516840.8358060.8399810.844326----0.0000000.0000000.000000NMTNoEnsemble 8 Models: joint BPE 16k, BiLSTM Encoder 512*2*2, LtoR LSTM Decoder 512*2, Beam Search 20 w/ length-based reranking
62TMUen-ja2017/08/01 11:42:0317040.8022620.8096490.811057----0.0000000.0000000.000000NMTNoour baseline system in 2017
63TMUen-ja2017/08/01 11:56:3117090.8129260.8184430.821563----0.0000000.0000000.000000NMTNobaseline system with beam20
64UT-IISen-ja2017/08/01 12:03:2217100.8278910.8320540.836394----0.0000000.0000000.000000NMTNoNMT 8 Ensembles with beam search, Sentence Piece, Embedding layer initialization
65NTTen-ja2017/08/01 15:42:1317290.8385940.8417690.846486----0.0000000.0000000.000000NMTNoEnsemble 8 Models: joint BPE 16k, BiLSTM Encoder 512*2*2, LtoR LSTM Decoder 512*2, Beam Search 20 w/ length-based reranking, add 1M pseudo training data (generated from provided training data)
66Kyoto-Uen-ja2017/08/01 15:49:5117310.8324720.8358700.839646----0.0000000.0000000.000000NMTNoEnsemble of 4 BPE Averaged
67ORGANIZERen-ja2017/08/02 01:04:4917370.8448960.8475590.851471----0.0000000.0000000.000000NMTNoGoogle's "Attention Is All You Need"
68TMUen-ja2017/08/02 10:20:1417410.8172800.8243320.825683----0.0000000.0000000.000000NMTNothe ensemble system of different dropout rate.
69TMUen-ja2017/08/04 11:06:5117470.8248010.8295800.832569----0.0000000.0000000.000000NMTNobeam_size: 10, ensemble of different dropout rates.
70Kyoto-Uen-ja2017/09/04 22:31:1817590.8410000.8437250.847819----0.0000000.0000000.000000SMTNoSyscomb of AIAYN and KNMT
71ORGANIZERen-ja2018/08/14 10:58:4919000.8249850.8311830.833207-----0.0000000.000000NMTNoNMT with Attention
72NICT-5en-ja2018/08/22 18:43:0020480.8407760.8450420.849326-----0.0000000.000000NMTNoTensor2Tensor's transformer implementation. Used recurrently stacked layers model logic. Layer recurrence logic: 1-2-3-1-2-3-1-2-3-1-2-3. Codename: Megarecurrence.
73NICT-5en-ja2018/08/22 18:48:5120500.8410110.8443600.848283-----0.0000000.000000NMTNoRSNMT 6 layer
74Senseen-ja2018/08/24 11:38:2320920.7962140.8007270.803453-----0.0000000.000000NMTNo
75NICT-5en-ja2018/09/03 16:49:5722190.8471340.8493990.853634-----0.0000000.000000NMTNoBig Bidirectional Transformer. 1.5M sentences only.
76EHRen-ja2018/09/08 12:42:0322450.8287460.8333330.837806-----0.0000000.000000NMTNoSMT reranked NMT
77Osaka-Uen-ja2018/09/15 23:02:0524390.8250610.8293280.833200-----0.0000000.000000NMTYesrewarding model
78TMUen-ja2018/09/16 12:10:4924670.8179480.8242390.827386-----0.0000000.000000NMTNoBaseline-NMT ( Single )
79TMUen-ja2018/09/16 12:11:5824680.8191480.8253670.827650-----0.0000000.000000NMTNoGAN-NMT ( Single )
80TMUen-ja2018/09/16 12:44:2324690.8236530.8291560.831219-----0.0000000.000000NMTNoEnsemble of 6 Baseline-NMT
81Osaka-Uen-ja2018/09/16 13:06:4524700.7168890.7264690.729323-----0.0000000.000000SMTNopreordering with neural network
82srcben-ja2018/09/16 15:26:3724790.8503180.8522090.857017-----0.0000000.000000NMTNoTransformer with relative position, ensemble of 3 models.
83srcben-ja2018/09/16 15:52:1724800.8476780.8501800.855073-----0.0000000.000000NMTNoTransformer with relative position, ensemble of 4 models, rerank.
84TMUen-ja2018/09/16 16:56:4924820.8171530.8240660.826013-----0.0000000.000000NMTNoReconstructor-NMT ( Single )
85TMUen-ja2018/09/16 17:25:3624840.8209610.8267790.829178-----0.0000000.000000NMTNoEnsemble of 6 NMT ( 2 Baseline + 2 Reconstructor + 2 GAN )
86srcben-ja2019/07/25 11:43:0629180.8489290.8529480.855857-------NMTNoTransformer (Big) with relative position, sentence-wise smooth
87KNU_Hyundaien-ja2019/07/27 09:19:3131720.8570600.8598850.863400-------NMTNoTransformer Base, relative position, BT, r2l reranking, ensemble of 3 models
88NICT-2en-ja2019/07/27 10:56:0231810.8504580.8539090.857275-------NMTNoTransformer, sigle model w/ long warm-up and self-training
89NICT-2en-ja2019/07/27 10:57:2631820.8529700.8567550.859059-------NMTNoTransformer, ensemble of 4 models w/ long warm-up and self-training
90srcben-ja2019/07/27 15:28:3032060.8562450.8590250.863177-------NMTNoTransformer(Big) with relative position, sentence-wise smooth, deep transformer, back translation, ensemble of 7 models, re-rank.
91srcben-ja2019/07/27 15:50:1932120.8575060.8606420.862819-------NMTNoTransformer(Big) with relative position, sentence-wise smooth, deep transformer, back translation, ensemble of 10 models, rerank.
92NTTen-ja2019/07/28 15:44:1232360.8619940.8656400.868175-------NMTNoASPEC first 1.5M + Synthetic 1.5M, 6 ensemble
93NTTen-ja2019/07/28 15:55:1832390.8596730.8635320.866183-------NMTYesParaCrawl + (ASPEC first 1.5M + Synthetic 1.5M) * 2 oversampling, fine-tune ASPEC, SINGLE MODEL
94AISTAIen-ja2019/07/29 07:33:5632510.8491290.8514000.856177-------NMTNoTransformer (big). 1.5M sentences, train_steps=131000 only. Averaged the last 10 ckpts.
95AISTAIen-ja2019/08/30 20:23:2433570.8560820.8591660.862434-------NMTNoTransformer, 1.5M sentences, relative position, ensemble of 3 models, by OpenNMT-py.
96AISTAIen-ja2019/09/05 08:18:4533730.8478630.8516730.854716-------NMTNoTransformer (big), 1.5M sentences, train_steps=300000, Averaged the last 20 ckpts, by Tensor2Tensor.

Notice:

Back to top

AMFM


# Team Task Date/Time DataID AMFM
Method
Other
Resources
System
Description
unuse unuse unuse unuse unuse unuse unuse unuse unuse unuse
1ORGANIZERen-ja2016/07/26 11:31:4710410.6770200.6770200.677020----0.0000000.0000000.000000OtherYesOnline A (2016)
2NICT-2en-ja2016/08/05 17:47:0510970.7530800.7530800.753080----0.0000000.0000000.000000SMTYesPhrase-based SMT with Preordering + Domain Adaptation (JPC and ASPEC) + Google 5-gram LM
3TOKYOMTen-ja2016/08/12 11:21:4311310.7052100.7052100.705210----0.0000000.0000000.000000NMTNochar 1 , ens 2 , version 1
4EHRen-ja2016/08/15 11:21:5111400.7467200.7467200.746720----0.0000000.0000000.000000SMTNoPBSMT with preordering (DL=6)
5bjtu_nlpen-ja2016/08/16 11:21:0711430.7043400.7043400.704340----0.0000000.0000000.000000NMTNoRNN Encoder-Decoder with attention mechanism, single model
6JAPIOen-ja2016/08/17 12:50:5011650.6607900.6607900.660790----0.0000000.0000000.000000SMTYesPhrase-based SMT with Preordering + JAPIO corpus + rule-based posteditor
7Kyoto-Uen-ja2016/08/18 03:45:5111720.7387000.7387000.738700----0.0000000.0000000.000000NMTNoBPE tgt/src: 52k 2-layer lstm self-ensemble of 3
8Kyoto-Uen-ja2016/08/19 10:18:0912010.7475100.7475100.747510----0.0000000.0000000.000000EBMTNoKyotoEBMT 2016 w/o reranking
9TOKYOMTen-ja2016/08/19 23:18:4612170.7208100.7208100.720810----0.0000000.0000000.000000NMTNoCombination of NMT and T2S
10UT-AKYen-ja2016/08/20 09:25:1512240.7081400.7081400.708140----0.0000000.0000000.000000NMTNotree-to-seq NMT model (character-based decoder)
11UT-AKYen-ja2016/08/20 12:34:4312280.7314400.7314400.731440----0.0000000.0000000.000000NMTNotree-to-seq NMT model (word-based decoder)
12ORGANIZERen-ja2016/11/16 10:50:1313340.7270400.7270400.727040----0.0000000.0000000.000000NMTYesOnline A (2016/11/14)
13UT-KAYen-ja2017/06/27 16:58:4213690.7530900.7530900.753090----0.0000000.0000000.000000NMTNoHashimoto and Tsuruoka (2017), https://arxiv.org/abs/1702.02265
14NICT-2en-ja2017/07/26 13:52:3514750.7595700.7595700.759570----0.0000000.0000000.000000NMTNoNMT Single Model: BPE50k, Bi-LSTM(500*2) Encoder, LSTM(1000) Left-to-Right Decoder
15NICT-2en-ja2017/07/26 14:03:0014790.7655800.7655800.765580----0.0000000.0000000.000000NMTNoNMT 6 Ensembles * Bi-directional Reranking
16NAIST-NICTen-ja2017/07/27 21:49:0415060.7633100.7633100.763310----0.0000000.0000000.000000NMTNoSingle Model: SPM16k/16k, BiLSTM Encoder 512*2*2, UniLSTM Decoder 512*2, One-best Search
17NAIST-NICTen-ja2017/07/27 21:50:1515070.7704800.7704800.770480----0.0000000.0000000.000000NMTNoSingle Model: SPM16k/16k, BiLSTM Encoder 512*2*2, UniLSTM Decoder 512*2, Adjusted Search
18NTTen-ja2017/07/30 16:16:2516080.7564800.7564800.756480----0.0000000.0000000.000000NMTNoSingle Model: joint BPE 16k, BiLSTM Encoder 512*2*2, LtoR LSTM Decoder 512*2, Beam Search 20 w/ length-based reranking
19NTTen-ja2017/08/01 02:22:5616730.7637700.7637700.763770----0.0000000.0000000.000000NMTNoSingle Model: joint BPE 16k, BiLSTM Encoder 512*2*2, LtoR LSTM Decoder 512*2, Beam Search 20 w/ length-based reranking, add 1M pseudo training data (generated from provided training data)
20NTTen-ja2017/08/01 07:12:2516840.7577400.7577400.757740----0.0000000.0000000.000000NMTNoEnsemble 8 Models: joint BPE 16k, BiLSTM Encoder 512*2*2, LtoR LSTM Decoder 512*2, Beam Search 20 w/ length-based reranking
21TMUen-ja2017/08/01 11:42:0317040.7406200.7406200.740620----0.0000000.0000000.000000NMTNoour baseline system in 2017
22TMUen-ja2017/08/01 11:56:3117090.7448900.7448900.744890----0.0000000.0000000.000000NMTNobaseline system with beam20
23UT-IISen-ja2017/08/01 12:03:2217100.7469100.7469100.746910----0.0000000.0000000.000000NMTNoNMT 8 Ensembles with beam search, Sentence Piece, Embedding layer initialization
24NTTen-ja2017/08/01 15:42:1317290.7621700.7621700.762170----0.0000000.0000000.000000NMTNoEnsemble 8 Models: joint BPE 16k, BiLSTM Encoder 512*2*2, LtoR LSTM Decoder 512*2, Beam Search 20 w/ length-based reranking, add 1M pseudo training data (generated from provided training data)
25Kyoto-Uen-ja2017/08/01 15:49:5117310.7542200.7542200.754220----0.0000000.0000000.000000NMTNoEnsemble of 4 BPE Averaged
26ORGANIZERen-ja2017/08/02 01:04:4917370.7686300.7686300.768630----0.0000000.0000000.000000NMTNoGoogle's "Attention Is All You Need"
27TMUen-ja2017/08/02 10:20:1417410.7516600.7516600.751660----0.0000000.0000000.000000NMTNothe ensemble system of different dropout rate.
28TMUen-ja2017/08/04 11:06:5117470.7494100.7494100.749410----0.0000000.0000000.000000NMTNobeam_size: 10, ensemble of different dropout rates.
29Kyoto-Uen-ja2017/09/04 22:31:1817590.7710200.7710200.771020----0.0000000.0000000.000000SMTNoSyscomb of AIAYN and KNMT
30ORGANIZERen-ja2018/08/14 10:58:4919000.7599100.7599100.759910-----0.0000000.000000NMTNoNMT with Attention
31NICT-5en-ja2018/08/22 18:43:0020480.7714000.7714000.771400-----0.0000000.000000NMTNoTensor2Tensor's transformer implementation. Used recurrently stacked layers model logic. Layer recurrence logic: 1-2-3-1-2-3-1-2-3-1-2-3. Codename: Megarecurrence.
32NICT-5en-ja2018/08/22 18:48:5120500.7705600.7705600.770560-----0.0000000.000000NMTNoRSNMT 6 layer
33Senseen-ja2018/08/24 11:38:2320920.7410900.7410900.741090-----0.0000000.000000NMTNo
34NICT-5en-ja2018/09/03 16:49:5722190.7795600.7795600.779560-----0.0000000.000000NMTNoBig Bidirectional Transformer. 1.5M sentences only.
35EHRen-ja2018/09/08 12:42:0322450.7587500.7587500.758750-----0.0000000.000000NMTNoSMT reranked NMT
36Osaka-Uen-ja2018/09/15 23:02:0524390.7631400.7631400.763140-----0.0000000.000000NMTYesrewarding model
37TMUen-ja2018/09/16 12:10:4924670.7491900.7491900.749190-----0.0000000.000000NMTNoBaseline-NMT ( Single )
38TMUen-ja2018/09/16 12:11:5824680.7503500.7503500.750350-----0.0000000.000000NMTNoGAN-NMT ( Single )
39TMUen-ja2018/09/16 12:44:2324690.7530400.7530400.753040-----0.0000000.000000NMTNoEnsemble of 6 Baseline-NMT
40Osaka-Uen-ja2018/09/16 13:06:4524700.7050500.7050500.705050-----0.0000000.000000SMTNopreordering with neural network
41srcben-ja2018/09/16 15:26:3724790.7810000.7810000.781000-----0.0000000.000000NMTNoTransformer with relative position, ensemble of 3 models.
42srcben-ja2018/09/16 15:52:1724800.7798200.7798200.779820-----0.0000000.000000NMTNoTransformer with relative position, ensemble of 4 models, rerank.
43TMUen-ja2018/09/16 16:56:4924820.7521000.7521000.752100-----0.0000000.000000NMTNoReconstructor-NMT ( Single )
44TMUen-ja2018/09/16 17:25:3624840.7524200.7524200.752420-----0.0000000.000000NMTNoEnsemble of 6 NMT ( 2 Baseline + 2 Reconstructor + 2 GAN )
45srcben-ja2019/07/25 11:43:0629180.7595100.7595100.759510-------NMTNoTransformer (Big) with relative position, sentence-wise smooth
46KNU_Hyundaien-ja2019/07/27 09:19:3131720.7600500.7600500.760050-------NMTNoTransformer Base, relative position, BT, r2l reranking, ensemble of 3 models
47NICT-2en-ja2019/07/27 10:56:0231810.7592600.7592600.759260-------NMTNoTransformer, sigle model w/ long warm-up and self-training
48NICT-2en-ja2019/07/27 10:57:2631820.7613400.7613400.761340-------NMTNoTransformer, ensemble of 4 models w/ long warm-up and self-training
49srcben-ja2019/07/27 15:28:3032060.7653300.7653300.765330-------NMTNoTransformer(Big) with relative position, sentence-wise smooth, deep transformer, back translation, ensemble of 7 models, re-rank.
50srcben-ja2019/07/27 15:50:1932120.7704400.7704400.770440-------NMTNoTransformer(Big) with relative position, sentence-wise smooth, deep transformer, back translation, ensemble of 10 models, rerank.
51NTTen-ja2019/07/28 15:44:1232360.7749500.7749500.774950-------NMTNoASPEC first 1.5M + Synthetic 1.5M, 6 ensemble
52NTTen-ja2019/07/28 15:55:1832390.7725900.7725900.772590-------NMTYesParaCrawl + (ASPEC first 1.5M + Synthetic 1.5M) * 2 oversampling, fine-tune ASPEC, SINGLE MODEL
53AISTAIen-ja2019/07/29 07:33:5632510.7575900.7575900.757590-------NMTNoTransformer (big). 1.5M sentences, train_steps=131000 only. Averaged the last 10 ckpts.
54AISTAIen-ja2019/08/30 20:23:2433570.7604100.7604100.760410-------NMTNoTransformer, 1.5M sentences, relative position, ensemble of 3 models, by OpenNMT-py.
55AISTAIen-ja2019/09/05 08:18:4533730.7599400.7599400.759940-------NMTNoTransformer (big), 1.5M sentences, train_steps=300000, Averaged the last 20 ckpts, by Tensor2Tensor.
56ORGANIZERen-ja2014/07/11 19:49:0350.7363800.7363800.7363800.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoPhrase-based SMT
57ORGANIZERen-ja2014/07/11 20:03:21120.7443700.7443700.7443700.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoTree-to-String SMT (2014)
58ORGANIZERen-ja2014/07/18 11:02:25340.6954200.6954200.6954200.0000000.0000000.0000000.0000000.0000000.0000000.000000OtherYesOnline A (2014)
59ORGANIZERen-ja2014/07/21 11:38:12660.6229300.6229300.6229300.0000000.0000000.0000000.0000000.0000000.0000000.000000OtherYesRBMT B (2014)
60ORGANIZERen-ja2014/07/21 11:40:50680.6269400.6269400.6269400.0000000.0000000.0000000.0000000.0000000.0000000.000000OtherYesRBMT A
61ORGANIZERen-ja2014/07/22 13:30:13910.6430700.6430700.6430700.0000000.0000000.0000000.0000000.0000000.0000000.000000OtherYesOnline B (2014)
62ORGANIZERen-ja2014/07/23 14:50:44950.5943800.5943800.5943800.0000000.0000000.0000000.0000000.0000000.0000000.000000OtherYesRBMT C
63NAISTen-ja2014/07/31 11:38:371180.7528500.7528500.7528500.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoTravatar-based Forest-to-String SMT System
64NAISTen-ja2014/08/01 17:37:231260.7587400.7587400.7587400.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoTravatar-based Forest-to-String SMT System (Tuned BLEU+RIBES)
65Wen-ja2014/08/16 00:57:161320.7465700.7465700.7465700.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoWeblio Pre-reordering SMT System Baseline
66Kyoto-Uen-ja2014/08/19 10:16:021340.7378900.7378900.7378900.0000000.0000000.0000000.0000000.0000000.0000000.000000EBMTNoOur baseline system using 3M parallel sentences.
67Senseen-ja2014/08/25 01:07:271840.7333600.7333600.7333600.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoBaseline SMT
68Kyoto-Uen-ja2014/08/25 14:16:151860.7323800.7323800.7323800.0000000.0000000.0000000.0000000.0000000.0000000.000000EBMTNoUsing n-best parses and RNNLM.
69Wen-ja2014/08/26 16:17:152020.7476500.7476500.7476500.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoWeblio Pre-reordering SMT System (with forest inputs)
70Kyoto-Uen-ja2014/08/31 10:33:232530.7417100.7417100.7417100.0000000.0000000.0000000.0000000.0000000.0000000.000000EBMTNoOur new baseline system after several modifications.
71SAS_MTen-ja2014/09/01 10:39:272640.7405800.7405800.7405800.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoSyntactic reordering Hierarchical SMT (using part of data)
72Kyoto-Uen-ja2014/09/01 21:06:322670.7386800.7386800.7386800.0000000.0000000.0000000.0000000.0000000.0000000.000000EBMTNoOur new baseline system after several modifications + 20-best parses, KN7, RNNLM reranking
73ORGANIZERen-ja2014/09/16 13:36:353670.7439000.7439000.7439000.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoHierarchical Phrase-based SMT (2014)
74TOSHIBAen-ja2015/07/28 16:24:375240.7421000.7421000.7421000.0000000.0000000.0000000.0000000.0000000.0000000.000000SMT and RBMTYesRBMT with SPE(Statistical Post Editing) system
75Senseen-ja2015/07/28 22:26:555310.7316200.7316200.7316200.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoBaseline-2015
76TOSHIBAen-ja2015/07/30 11:22:115400.7446600.7446600.7446600.0000000.0000000.0000000.0000000.0000000.0000000.000000SMT and RBMTYesSystem combination SMT and RBMT(SPE) with RNNLM language model
77naveren-ja2015/08/04 16:48:325810.7375900.7375900.7375900.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoSMT t2s
78Senseen-ja2015/08/18 21:52:017000.7292000.7292000.7292000.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoBaseline-dictmt
79Senseen-ja2015/08/18 22:04:097150.7314000.7314000.7314000.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTYesPassive JSTx3
80EHRen-ja2015/08/22 12:28:157420.6205000.6205000.6205000.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoPhrase based SMT with preordering.
81NAISTen-ja2015/08/25 12:39:237610.7488700.7488700.7488700.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoTravatar System with NeuralMT Reranking
82NAISTen-ja2015/08/25 12:47:497630.7522600.7522600.7522600.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoTravatar System Baseline
83naveren-ja2015/08/25 16:20:307700.7402100.7402100.7402100.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoSMT t2s + Spell correction
84ORGANIZERen-ja2015/08/25 18:54:297740.6772000.6772000.6772000.0000000.0000000.0000000.0000000.0000000.0000000.000000OtherYesOnline A (2015)
85Wen-ja2015/08/26 16:00:467860.7253700.7253700.7253700.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoNMT, LSTM Search, 5 ensembles, beam size 20, UNK replacing, System Combination with NMT score (Pick top-1k results from NMT)
86Kyoto-Uen-ja2015/08/27 22:56:138050.7410500.7410500.7410500.0000000.0000000.0000000.0000000.0000000.0000000.000000EBMTNoKyotoEBMT system without reranking
87Wen-ja2015/08/28 14:30:568130.7053400.7053400.7053400.0000000.0000000.0000000.0000000.0000000.0000000.000000OtherNoNMT, LSTM Search, Beam Size 20, Ensemble of 2 models, UNK replacing
88Senseen-ja2015/08/28 19:22:248210.7375600.7375600.7375600.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoBaseline-2015 (train1 only)
89Kyoto-Uen-ja2015/08/30 19:59:188320.7437100.7437100.7437100.0000000.0000000.0000000.0000000.0000000.0000000.000000EBMTNoKyotoEBMT system with bilingual RNNLM reranking
90naveren-ja2015/08/31 14:12:038360.7258600.7258600.7258600.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoNMT only
91naveren-ja2015/08/31 14:18:188370.7446300.7446300.7446300.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoSMT t2s + Spell correction + NMT reranking
92EHRen-ja2015/09/09 12:21:348730.6304100.6304100.6304100.0000000.0000000.0000000.0000000.0000000.0000000.000000SMT and RBMTYesSystem combination of RBMT with user dictionary plus SPE and phrase based SMT with preordering. Candidate selection by language model score.
93EHRen-ja2015/09/09 12:22:588740.6040900.6040900.6040900.0000000.0000000.0000000.0000000.0000000.0000000.000000SMT and RBMTYesRBMT with user dictionary plus SPE.
94ORGANIZERen-ja2015/09/10 13:29:358750.7443700.7443700.7443700.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoTree-to-String SMT (2015)
95ORGANIZERen-ja2015/09/10 14:26:288830.6229300.6229300.6229300.0000000.0000000.0000000.0000000.0000000.0000000.000000OtherYesRBMT B (2015)
96ORGANIZERen-ja2015/09/10 19:02:388890.6461600.6461600.6461600.0000000.0000000.0000000.0000000.0000000.0000000.000000OtherYesOnline B (2015)

Notice:

Back to top

HUMAN (WAT2022)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description

Notice:
Back to top

HUMAN (WAT2021)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description

Notice:
Back to top

HUMAN (WAT2020)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description

Notice:
Back to top

HUMAN (WAT2019)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description
1NTTen-ja2019/07/28 15:44:12323647.750NMTNoASPEC first 1.5M + Synthetic 1.5M, 6 ensemble
2NICT-2en-ja2019/07/27 10:57:26318242.250NMTNoTransformer, ensemble of 4 models w/ long warm-up and self-training
3srcben-ja2019/07/27 15:50:19321240.000NMTNoTransformer(Big) with relative position, sentence-wise smooth, deep transformer, back translation, ensemble of 10 models, rerank.
4AISTAIen-ja2019/07/29 07:33:56325136.750NMTNoTransformer (big). 1.5M sentences, train_steps=131000 only. Averaged the last 10 ckpts.
5KNU_Hyundaien-ja2019/07/27 09:19:31317236.000NMTNoTransformer Base, relative position, BT, r2l reranking, ensemble of 3 models

Notice:
Back to top

HUMAN (WAT2018)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description
1NICT-5en-ja2018/09/03 16:49:57221928.500NMTNoBig Bidirectional Transformer. 1.5M sentences only.
2srcben-ja2018/09/16 15:26:37247925.000NMTNoTransformer with relative position, ensemble of 3 models.
3NICT-5en-ja2018/08/22 18:43:00204820.250NMTNoTensor2Tensor's transformer implementation. Used recurrently stacked layers model logic. Layer recurrence logic: 1-2-3-1-2-3-1-2-3-1-2-3. Codename: Megarecurrence.
4Osaka-Uen-ja2018/09/15 23:02:0524394.500NMTYesrewarding model
5EHRen-ja2018/09/08 12:42:032245-0.500NMTNoSMT reranked NMT
6TMUen-ja2018/09/16 12:44:232469-12.000NMTNoEnsemble of 6 Baseline-NMT
7Osaka-Uen-ja2018/09/16 13:06:452470-82.250SMTNopreordering with neural network

Notice:
Back to top

HUMAN (WAT2017)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description
1NTTen-ja2017/08/01 15:42:13172975.750NMTNoEnsemble 8 Models: joint BPE 16k, BiLSTM Encoder 512*2*2, LtoR LSTM Decoder 512*2, Beam Search 20 w/ length-based reranking, add 1M pseudo training data (generated from provided training data)
2NICT-2en-ja2017/07/26 14:03:00147974.750NMTNoNMT 6 Ensembles * Bi-directional Reranking
3NTTen-ja2017/08/01 07:12:25168472.250NMTNoEnsemble 8 Models: joint BPE 16k, BiLSTM Encoder 512*2*2, LtoR LSTM Decoder 512*2, Beam Search 20 w/ length-based reranking
4NAIST-NICTen-ja2017/07/27 21:50:15150770.000NMTNoSingle Model: SPM16k/16k, BiLSTM Encoder 512*2*2, UniLSTM Decoder 512*2, Adjusted Search
5Kyoto-Uen-ja2017/08/01 15:49:51173169.750NMTNoEnsemble of 4 BPE Averaged
6ORGANIZERen-ja2017/08/02 01:04:49173769.750NMTNoGoogle's "Attention Is All You Need"
7UT-IISen-ja2017/08/01 12:03:22171068.000NMTNoNMT 8 Ensembles with beam search, Sentence Piece, Embedding layer initialization
8NAIST-NICTen-ja2017/07/27 21:49:04150663.500NMTNoSingle Model: SPM16k/16k, BiLSTM Encoder 512*2*2, UniLSTM Decoder 512*2, One-best Search
9NICT-2en-ja2017/07/26 13:52:35147562.000NMTNoNMT Single Model: BPE50k, Bi-LSTM(500*2) Encoder, LSTM(1000) Left-to-Right Decoder
10TMUen-ja2017/08/01 11:56:31170956.500NMTNobaseline system with beam20
11TMUen-ja2017/08/01 11:42:03170450.750NMTNoour baseline system in 2017

Notice:
Back to top

HUMAN (WAT2016)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description
1ORGANIZERen-ja2016/11/16 10:50:13133474.000NMTYesOnline A (2016/11/14)
2Kyoto-Uen-ja2016/08/18 03:45:51117255.250NMTNoBPE tgt/src: 52k 2-layer lstm self-ensemble of 3
3ORGANIZERen-ja2016/07/26 11:31:47104149.750OtherYesOnline A (2016)
4NICT-2en-ja2016/08/05 17:47:05109741.250SMTYesPhrase-based SMT with Preordering + Domain Adaptation (JPC and ASPEC) + Google 5-gram LM
5bjtu_nlpen-ja2016/08/16 11:21:07114339.500NMTNoRNN Encoder-Decoder with attention mechanism, single model
6EHRen-ja2016/08/15 11:21:51114039.000SMTNoPBSMT with preordering (DL=6)
7UT-AKYen-ja2016/08/20 12:34:43122836.250NMTNotree-to-seq NMT model (word-based decoder)
8TOKYOMTen-ja2016/08/19 23:18:46121730.500NMTNoCombination of NMT and T2S
9TOKYOMTen-ja2016/08/12 11:21:43113129.750NMTNochar 1 , ens 2 , version 1
10UT-AKYen-ja2016/08/20 09:25:15122421.750NMTNotree-to-seq NMT model (character-based decoder)
11JAPIOen-ja2016/08/17 12:50:5011654.250SMTYesPhrase-based SMT with Preordering + JAPIO corpus + rule-based posteditor

Notice:
Back to top

HUMAN (WAT2015)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description
1NAISTen-ja2015/08/25 12:39:2376162.250SMTNoTravatar System with NeuralMT Reranking
2Wen-ja2015/08/26 16:00:4678653.750SMTNoNMT, LSTM Search, 5 ensembles, beam size 20, UNK replacing, System Combination with NMT score (Pick top-1k results from NMT)
3naveren-ja2015/08/31 14:18:1883753.250SMTNoSMT t2s + Spell correction + NMT reranking
4Kyoto-Uen-ja2015/08/30 19:59:1883251.000EBMTNoKyotoEBMT system with bilingual RNNLM reranking
5NAISTen-ja2015/08/25 12:47:4976349.750SMTNoTravatar System Baseline
6naveren-ja2015/08/31 14:12:0383648.500SMTNoNMT only
7Wen-ja2015/08/28 14:30:5681343.500OtherNoNMT, LSTM Search, Beam Size 20, Ensemble of 2 models, UNK replacing
8Kyoto-Uen-ja2015/08/27 22:56:1380540.500EBMTNoKyotoEBMT system without reranking
9TOSHIBAen-ja2015/07/28 16:24:3752440.250SMT and RBMTYesRBMT with SPE(Statistical Post Editing) system
10ORGANIZERen-ja2015/08/25 18:54:2977434.250OtherYesOnline A (2015)
11EHRen-ja2015/08/22 12:28:1574232.500SMTNoPhrase based SMT with preordering.
12ORGANIZERen-ja2015/09/10 13:29:3587530.000SMTNoTree-to-String SMT (2015)
13ORGANIZERen-ja2015/09/10 14:26:288839.750OtherYesRBMT B (2015)
14Senseen-ja2015/08/18 22:04:09715-31.000SMTYesPassive JSTx3
15Senseen-ja2015/08/18 21:52:01700-36.250SMTNoBaseline-dictmt

Notice:
Back to top

HUMAN (WAT2014)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description
1NAISTen-ja2014/07/31 11:38:3711856.250SMTNoTravatar-based Forest-to-String SMT System
2NAISTen-ja2014/08/01 17:37:2312651.500SMTNoTravatar-based Forest-to-String SMT System (Tuned BLEU+RIBES)
3Wen-ja2014/08/16 00:57:1613243.250SMTNoWeblio Pre-reordering SMT System Baseline
4ORGANIZERen-ja2014/07/18 11:02:253442.500OtherYesOnline A (2014)
5Kyoto-Uen-ja2014/09/01 21:06:3226738.000EBMTNoOur new baseline system after several modifications + 20-best parses, KN7, RNNLM reranking
6Wen-ja2014/08/26 16:17:1520236.000SMTNoWeblio Pre-reordering SMT System (with forest inputs)
7ORGANIZERen-ja2014/07/11 20:03:211234.250SMTNoTree-to-String SMT (2014)
8Kyoto-Uen-ja2014/08/31 10:33:2325333.750EBMTNoOur new baseline system after several modifications.
9ORGANIZERen-ja2014/09/16 13:36:3536731.500SMTNoHierarchical Phrase-based SMT (2014)
10SAS_MTen-ja2014/09/01 10:39:2726427.500SMTNoSyntactic reordering Hierarchical SMT (using part of data)
11Senseen-ja2014/08/25 01:07:271843.750SMTNoBaseline SMT
12ORGANIZERen-ja2014/07/21 11:38:12660.750OtherYesRBMT B (2014)

Notice:
Back to top

EVALUATION RESULTS USAGE POLICY

When you use the WAT evaluation results for any purpose such as:
- writing technical papers,
- making presentations about your system,
- advertising your MT system to the customers,
you can use the information about translation directions, scores (including both automatic and human evaluations) and ranks of your system among others. You can also use the scores of the other systems, but you MUST anonymize the other system's names. In addition, you can show the links (URLs) to the WAT evaluation result pages.

NICT (National Institute of Information and Communications Technology)
Kyoto University
Last Modified: 2018-08-02