NICT_LOGO.JPG KYOTO-U_LOGO.JPG

WAT

The Workshop on Asian Translation
Evaluation Results

[EVALUATION RESULTS TOP] | [BLEU] | [RIBES] | [AMFM] | [HUMAN (WAT2022)] | [HUMAN (WAT2021)] | [HUMAN (WAT2020)] | [HUMAN (WAT2019)] | [HUMAN (WAT2018)] | [HUMAN (WAT2017)] | [HUMAN (WAT2016)] | [HUMAN (WAT2015)] | [HUMAN (WAT2014)] | [EVALUATION RESULTS USAGE POLICY]

BLEU


# Team Task Date/Time DataID BLEU
Method
Other
Resources
System
Description
juman kytea mecab moses-
tokenizer
stanford-
segmenter-
ctb
stanford-
segmenter-
pku
indic-
tokenizer
unuse myseg kmseg
1ORGANIZERJPCko-ja2015/06/10 18:44:0843869.2270.3669.73--- 0.00 0.00 0.00 0.00SMTNoPhrase-based SMT
2ORGANIZERJPCko-ja2015/06/10 18:46:3443967.4168.6568.00--- 0.00 0.00 0.00 0.00SMTNoHierarchical Phrase-based SMT (2015)
3EHRJPCko-ja2015/07/22 12:24:0150070.6771.5270.93--- 0.00 0.00 0.00 0.00SMT and RBMTYesSystem combination of RBMT with user dictionary plus SPE and phrase based SMT. Candidate selection by language model score.
4NICTJPCko-ja2015/07/23 14:13:3850170.6271.5270.92--- 0.00 0.00 0.00 0.00SMTNoour baseline: character-based / PB SMT in MOSES (DL=0, max-phrase-len=9, no lex-reordering) / SRILM (9-gram)
5NICTJPCko-ja2015/07/24 14:00:0651370.8171.7071.11--- 0.00 0.00 0.00 0.00SMTNoour baseline + post-processing of bracket balancing
6TOSHIBAJPCko-ja2015/07/31 19:11:0455268.8369.2869.04--- 0.00 0.00 0.00 0.00SMT and RBMTYesPhrase-based SPE with RNNLM reranking + post-processing
7TOSHIBAJPCko-ja2015/07/31 19:15:0055470.5170.8470.71--- 0.00 0.00 0.00 0.00SMT and RBMTYesSystem combination SMT, SPE and RBMT with RNNLM + post-processing
8TOSHIBAJPCko-ja2015/08/03 09:37:4956871.0171.4471.26--- 0.00 0.00 0.00 0.00SMTYesPhrase-based SMT with RNNLM reranking + post-processing
9naverJPCko-ja2015/08/11 17:38:5963070.9171.7671.18--- 0.00 0.00 0.00 0.00SMTNocombined two phrase-based systems, post-processing
10SenseJPCko-ja2015/08/14 10:07:0164384.2684.2384.26--- 0.00 0.00 0.00 0.00SMTNoUnicode2Stringwith devtest. MERT run1
11ORGANIZERJPCko-ja2015/08/14 17:17:3165136.4138.7237.01--- 0.00 0.00 0.00 0.00OtherYesOnline B (2015)
12ORGANIZERJPCko-ja2015/08/14 17:30:2165255.0556.8455.46--- 0.00 0.00 0.00 0.00OtherYesOnline A (2015)
13ORGANIZERJPCko-ja2015/08/14 17:37:5765342.0043.9742.45--- 0.00 0.00 0.00 0.00OtherYesRBMT A (2015)
14ORGANIZERJPCko-ja2015/08/14 17:40:1165434.7437.5135.54--- 0.00 0.00 0.00 0.00OtherYesRBMT B
15SenseJPCko-ja2015/08/15 15:36:3465785.2385.2085.23--- 0.00 0.00 0.00 0.00SMTNoUnicode2String with devtest MERT run2
16TOSHIBAJPCko-ja2015/08/17 11:55:2266858.6059.3458.49--- 0.00 0.00 0.00 0.00RBMTYesRBMT
17SenseJPCko-ja2015/08/19 02:23:0171985.2485.2185.23--- 0.00 0.00 0.00 0.00SMTNo에이트->エイト
18SenseJPCko-ja2015/08/19 21:23:1772371.6371.7270.90--- 0.00 0.00 0.00 0.00SMTNoBytes2String
19SenseJPCko-ja2015/08/19 21:26:2872446.0448.3046.62--- 0.00 0.00 0.00 0.00SMTNoUnicode2String
20naverJPCko-ja2015/08/28 17:07:1081671.3872.2771.68--- 0.00 0.00 0.00 0.00SMTNoSMT PB + NMT reranking
21EHRJPCko-ja2015/08/30 15:52:0783170.1370.8670.35--- 0.00 0.00 0.00 0.00SMT and RBMTYesRBMT with user dictionary plus SPE
22SenseJPCko-ja2015/08/31 05:00:3583370.2371.1170.51--- 0.00 0.00 0.00 0.00SMTNoBaseline with only train.ja/ko
23EHRJPCko-ja2015/09/02 16:39:4686570.8171.5771.00--- 0.00 0.00 0.00 0.00SMTYesPhrase based SMT
24ORGANIZERJPCko-ja2016/07/13 00:54:1096355.0556.8455.46---- 0.00 0.00 0.00OtherYesOnline A (2015)
25ORGANIZERJPCko-ja2016/07/13 01:34:5696442.0043.9742.45---- 0.00 0.00 0.00OtherYesRBMT A (2015)
26ORGANIZERJPCko-ja2016/07/13 01:35:3796567.4168.6568.00---- 0.00 0.00 0.00SMTNoHierarchical Phrase-based SMT (2015)
27EHRJPCko-ja2016/07/17 15:54:31100468.7869.6569.06---- 0.00 0.00 0.00SMTYesComb. of PBSMT by word and character. DL=0. Parens in Ko side and not in Ja side are added to Ja training and dev sets. After decoding, parens surrounding number are deleted.
28EHRJPCko-ja2016/07/17 18:23:55100571.5172.3271.77---- 0.00 0.00 0.00SMTYesCombination of word-based PBSMT and character-based PBSMT with DL=0. Parentheses surrounding number in Korean sentences are deleted.
29EHRJPCko-ja2016/07/17 18:25:46100662.3364.1762.75---- 0.00 0.00 0.00SMTYesCombination of word-based PBSMT and character-based PBSMT with DL=0. Parentheses in Korean side and not in Japanese side are added to Japanese for training and dev sets.
30ORGANIZERJPCko-ja2016/07/26 11:16:02103954.7856.6855.14---- 0.00 0.00 0.00OtherYesOnline A (2016)
31JAPIOJPCko-ja2016/08/19 15:34:39120668.6269.4968.90---- 0.00 0.00 0.00SMTYesPhrase-based SMT + JAPIO corpus + rule-based posteditor
32JAPIOJPCko-ja2016/08/19 17:10:56120970.3271.0770.52---- 0.00 0.00 0.00SMTYesPhrase-based SMT + JPC/JAPIO corpora + rule-based posteditor
33transictktJPCko-ja2016/08/21 19:13:09126769.3069.4269.77---- 0.00 0.00 0.00SMTNoPhrase-based SMT.
34JAPIOJPCko-ja2016/10/28 13:37:08133169.1070.0369.34---- 0.00 0.00 0.00SMTNoPhrase-based SMT + rule-based posteditor
35ORGANIZERJPCko-ja2016/11/25 11:27:15134444.4245.1444.72---- 0.00 0.00 0.00NMTNoOnline A (2016/11/25)
36EHRJPCko-ja2017/07/19 21:55:39141671.5272.3471.82---- 0.00 0.00 0.00NMTNoSMT reranked NMT (word based, by Moses and OpenNMT)
37EHRJPCko-ja2017/07/19 21:57:18141771.3672.2671.65---- 0.00 0.00 0.00NMTNoSMT reranked NMT (character based, by Moses and OpenNMT)
38EHRJPCko-ja2017/07/19 21:58:48141870.2371.1470.54---- 0.00 0.00 0.00NMTNoSimple NMT (word based, by OpenNMT)
39EHRJPCko-ja2017/07/19 22:00:25141969.4270.2869.70---- 0.00 0.00 0.00NMTNoSimple NMT (character based, by OpenNMT)
40JAPIOJPCko-ja2017/07/25 17:12:46144873.0073.7173.23---- 0.00 0.00 0.00SMTYesWord-based PBSMT + JPC/Japio corpora + rule-based posteditor
41JAPIOJPCko-ja2017/07/25 17:15:54144971.9772.9172.17---- 0.00 0.00 0.00SMTYesCharacter-based PBSMT + JPC/Japio corpora + rule-based posteditor
42JAPIOJPCko-ja2017/07/25 17:18:17145073.0073.7373.25---- 0.00 0.00 0.00SMTYesCombination of word- and character-based PBSMTs
43ORGANIZERJPCko-ja2018/08/15 18:44:10196671.4272.2171.68----- 0.00 0.00NMTNoNMT with Attention
44EHRJPCko-ja2018/08/31 19:01:15221571.4572.3571.74----- 0.00 0.00NMTNoSMT reranked NMT
45ryanJPCko-ja2019/07/23 11:57:16283873.5374.3173.78-------NMTNoBase Transformer
46sarahJPCko-ja2019/07/26 16:27:49301673.1673.9373.40-------NMTNoTransformer, ensemble of 4 models
47KNU_HyundaiJPCko-ja2019/07/26 23:59:03309373.7374.5574.03-------NMTNoTransformer + Relative Position + ensemble of 11 models
48TMUJPCko-ja2020/09/15 07:33:10372173.6174.4273.90-------NMTNoTransformer, domain adaptation (BERT Japanese), hanja loss, shared EMB, shared BPE, ensemble of 4 models
49TMUJPCko-ja2020/09/15 20:51:18375773.5674.4173.85-------NMTNoTransformer, domain adaptation (BERT Japanese), shared EMB, shared BPE, ensemble of 4 models
50goku20JPCko-ja2020/09/21 12:11:26409168.3969.3968.72-------NMTNomBART pre-training transformer, single model
51goku20JPCko-ja2020/09/22 10:17:13412069.6470.5469.87-------NMTNomBART pre-training transformer, ensemble of 3 models
52TMUJPCko-ja2020/10/13 09:43:00413873.6174.4273.90-------NMTNoTransformer, domain adaptation (BERT Korean), hanja loss, shared EMB, shared BPE, ensemble of 4 models
53TMUJPCko-ja2020/10/13 10:00:00414273.5674.4173.85-------NMTNoTransformer, domain adaptation (BERT Korean), shared EMB, shared BPE, ensemble of 4 models
54Bering LabJPCko-ja2021/04/25 02:59:58551376.2176.7176.36-------NMTYesTransformer Ensemble with additional crawled parallel corpus
55tpt_watJPCko-ja2021/04/27 02:01:36569966.7667.5466.61-------NMTNoBase Transformer model with joint vocab, size 8k

Notice:

Back to top

RIBES


# Team Task Date/Time DataID RIBES
Method
Other
Resources
System
Description
juman kytea mecab moses-
tokenizer
stanford-
segmenter-
ctb
stanford-
segmenter-
pku
indic-
tokenizer
unuse myseg kmseg
1ORGANIZERJPCko-ja2015/06/10 18:44:084380.9413020.9397290.940756---0.0000000.0000000.0000000.000000SMTNoPhrase-based SMT
2ORGANIZERJPCko-ja2015/06/10 18:46:344390.9371620.9359030.936570---0.0000000.0000000.0000000.000000SMTNoHierarchical Phrase-based SMT (2015)
3EHRJPCko-ja2015/07/22 12:24:015000.9430400.9418930.942771---0.0000000.0000000.0000000.000000SMT and RBMTYesSystem combination of RBMT with user dictionary plus SPE and phrase based SMT. Candidate selection by language model score.
4NICTJPCko-ja2015/07/23 14:13:385010.9433480.9424020.942748---0.0000000.0000000.0000000.000000SMTNoour baseline: character-based / PB SMT in MOSES (DL=0, max-phrase-len=9, no lex-reordering) / SRILM (9-gram)
5NICTJPCko-ja2015/07/24 14:00:065130.9434630.9425190.942904---0.0000000.0000000.0000000.000000SMTNoour baseline + post-processing of bracket balancing
6TOSHIBAJPCko-ja2015/07/31 19:11:045520.9397530.9372100.939180---0.0000000.0000000.0000000.000000SMT and RBMTYesPhrase-based SPE with RNNLM reranking + post-processing
7TOSHIBAJPCko-ja2015/07/31 19:15:005540.9421830.9395450.941471---0.0000000.0000000.0000000.000000SMT and RBMTYesSystem combination SMT, SPE and RBMT with RNNLM + post-processing
8TOSHIBAJPCko-ja2015/08/03 09:37:495680.9437940.9412870.943181---0.0000000.0000000.0000000.000000SMTYesPhrase-based SMT with RNNLM reranking + post-processing
9naverJPCko-ja2015/08/11 17:38:596300.9439280.9428000.943376---0.0000000.0000000.0000000.000000SMTNocombined two phrase-based systems, post-processing
10SenseJPCko-ja2015/08/14 10:07:016430.9503180.9501090.950235---0.0000000.0000000.0000000.000000SMTNoUnicode2Stringwith devtest. MERT run1
11ORGANIZERJPCko-ja2015/08/14 17:17:316510.8517450.8522630.851945---0.0000000.0000000.0000000.000000OtherYesOnline B (2015)
12ORGANIZERJPCko-ja2015/08/14 17:30:216520.9091520.9093850.908838---0.0000000.0000000.0000000.000000OtherYesOnline A (2015)
13ORGANIZERJPCko-ja2015/08/14 17:37:576530.8763960.8737340.875146---0.0000000.0000000.0000000.000000OtherYesRBMT A (2015)
14ORGANIZERJPCko-ja2015/08/14 17:40:116540.8457120.8490140.846228---0.0000000.0000000.0000000.000000OtherYesRBMT B
15SenseJPCko-ja2015/08/15 15:36:346570.9545060.9544480.954435---0.0000000.0000000.0000000.000000SMTNoUnicode2String with devtest MERT run2
16TOSHIBAJPCko-ja2015/08/17 11:55:226680.9167760.9139820.915122---0.0000000.0000000.0000000.000000RBMTYesRBMT
17SenseJPCko-ja2015/08/19 02:23:017190.9544980.9544630.954435---0.0000000.0000000.0000000.000000SMTNo에이트->エイト
18SenseJPCko-ja2015/08/19 21:23:177230.9328030.9312680.930872---0.0000000.0000000.0000000.000000SMTNoBytes2String
19SenseJPCko-ja2015/08/19 21:26:287240.8510410.8542980.851636---0.0000000.0000000.0000000.000000SMTNoUnicode2String
20naverJPCko-ja2015/08/28 17:07:108160.9438140.9428050.943584---0.0000000.0000000.0000000.000000SMTNoSMT PB + NMT reranking
21EHRJPCko-ja2015/08/30 15:52:078310.9418870.9403410.941517---0.0000000.0000000.0000000.000000SMT and RBMTYesRBMT with user dictionary plus SPE
22SenseJPCko-ja2015/08/31 05:00:358330.9424150.9408870.941687---0.0000000.0000000.0000000.000000SMTNoBaseline with only train.ja/ko
23EHRJPCko-ja2015/09/02 16:39:468650.9429990.9416000.942680---0.0000000.0000000.0000000.000000SMTYesPhrase based SMT
24ORGANIZERJPCko-ja2016/07/13 00:54:109630.9091520.9093850.908838----0.0000000.0000000.000000OtherYesOnline A (2015)
25ORGANIZERJPCko-ja2016/07/13 01:34:569640.8763960.8737340.875146----0.0000000.0000000.000000OtherYesRBMT A (2015)
26ORGANIZERJPCko-ja2016/07/13 01:35:379650.9371620.9359030.936570----0.0000000.0000000.000000SMTNoHierarchical Phrase-based SMT (2015)
27EHRJPCko-ja2016/07/17 15:54:3110040.9410710.9401690.940682----0.0000000.0000000.000000SMTYesComb. of PBSMT by word and character. DL=0. Parens in Ko side and not in Ja side are added to Ja training and dev sets. After decoding, parens surrounding number are deleted.
28EHRJPCko-ja2016/07/17 18:23:5510050.9446510.9435140.944606----0.0000000.0000000.000000SMTYesCombination of word-based PBSMT and character-based PBSMT with DL=0. Parentheses surrounding number in Korean sentences are deleted.
29EHRJPCko-ja2016/07/17 18:25:4610060.9270650.9272150.927017----0.0000000.0000000.000000SMTYesCombination of word-based PBSMT and character-based PBSMT with DL=0. Parentheses in Korean side and not in Japanese side are added to Japanese for training and dev sets.
30ORGANIZERJPCko-ja2016/07/26 11:16:0210390.9073200.9076520.906743----0.0000000.0000000.000000OtherYesOnline A (2016)
31JAPIOJPCko-ja2016/08/19 15:34:3912060.9384740.9370660.938230----0.0000000.0000000.000000SMTYesPhrase-based SMT + JAPIO corpus + rule-based posteditor
32JAPIOJPCko-ja2016/08/19 17:10:5612090.9421370.9405440.941746----0.0000000.0000000.000000SMTYesPhrase-based SMT + JPC/JAPIO corpora + rule-based posteditor
33transictktJPCko-ja2016/08/21 19:13:0912670.9412450.9376300.940497----0.0000000.0000000.000000SMTNoPhrase-based SMT.
34JAPIOJPCko-ja2016/10/28 13:37:0813310.9403670.9390570.939990----0.0000000.0000000.000000SMTNoPhrase-based SMT + rule-based posteditor
35ORGANIZERJPCko-ja2016/11/25 11:27:1513440.8576420.8541580.857083----0.0000000.0000000.000000NMTNoOnline A (2016/11/25)
36EHRJPCko-ja2017/07/19 21:55:3914160.9445160.9429400.944219----0.0000000.0000000.000000NMTNoSMT reranked NMT (word based, by Moses and OpenNMT)
37EHRJPCko-ja2017/07/19 21:57:1814170.9461260.9448120.945888----0.0000000.0000000.000000NMTNoSMT reranked NMT (character based, by Moses and OpenNMT)
38EHRJPCko-ja2017/07/19 21:58:4814180.9431790.9416950.942757----0.0000000.0000000.000000NMTNoSimple NMT (word based, by OpenNMT)
39EHRJPCko-ja2017/07/19 22:00:2514190.9364400.9356210.936136----0.0000000.0000000.000000NMTNoSimple NMT (character based, by OpenNMT)
40JAPIOJPCko-ja2017/07/25 17:12:4614480.9468800.9457540.946645----0.0000000.0000000.000000SMTYesWord-based PBSMT + JPC/Japio corpora + rule-based posteditor
41JAPIOJPCko-ja2017/07/25 17:15:5414490.9444350.9436180.943960----0.0000000.0000000.000000SMTYesCharacter-based PBSMT + JPC/Japio corpora + rule-based posteditor
42JAPIOJPCko-ja2017/07/25 17:18:1714500.9469850.9458410.946745----0.0000000.0000000.000000SMTYesCombination of word- and character-based PBSMTs
43ORGANIZERJPCko-ja2018/08/15 18:44:1019660.9455930.9446440.945126-----0.0000000.000000NMTNoNMT with Attention
44EHRJPCko-ja2018/08/31 19:01:1522150.9457710.9442840.945497-----0.0000000.000000NMTNoSMT reranked NMT
45ryanJPCko-ja2019/07/23 11:57:1628380.9499540.9489800.949567-------NMTNoBase Transformer
46sarahJPCko-ja2019/07/26 16:27:4930160.9495750.9485700.949128-------NMTNoTransformer, ensemble of 4 models
47KNU_HyundaiJPCko-ja2019/07/26 23:59:0330930.9500000.9489240.949410-------NMTNoTransformer + Relative Position + ensemble of 11 models
48TMUJPCko-ja2020/09/15 07:33:1037210.9501090.9492080.949567-------NMTNoTransformer, domain adaptation (BERT Japanese), hanja loss, shared EMB, shared BPE, ensemble of 4 models
49TMUJPCko-ja2020/09/15 20:51:1837570.9499830.9489080.949484-------NMTNoTransformer, domain adaptation (BERT Japanese), shared EMB, shared BPE, ensemble of 4 models
50goku20JPCko-ja2020/09/21 12:11:2640910.9430290.9419260.942480-------NMTNomBART pre-training transformer, single model
51goku20JPCko-ja2020/09/22 10:17:1341200.9454490.9443210.944852-------NMTNomBART pre-training transformer, ensemble of 3 models
52TMUJPCko-ja2020/10/13 09:43:0041380.9501090.9492080.949567-------NMTNoTransformer, domain adaptation (BERT Korean), hanja loss, shared EMB, shared BPE, ensemble of 4 models
53TMUJPCko-ja2020/10/13 10:00:0041420.9499830.9489080.949484-------NMTNoTransformer, domain adaptation (BERT Korean), shared EMB, shared BPE, ensemble of 4 models
54Bering LabJPCko-ja2021/04/25 02:59:5855130.9521010.9512610.951440-------NMTYesTransformer Ensemble with additional crawled parallel corpus
55tpt_watJPCko-ja2021/04/27 02:01:3656990.9390240.9376990.938299-------NMTNoBase Transformer model with joint vocab, size 8k

Notice:

Back to top

AMFM


# Team Task Date/Time DataID AMFM
Method
Other
Resources
System
Description
unuse unuse unuse unuse unuse unuse unuse unuse unuse unuse
1ORGANIZERJPCko-ja2016/07/13 00:54:109630.8006100.8006100.800610----0.0000000.0000000.000000OtherYesOnline A (2015)
2ORGANIZERJPCko-ja2016/07/13 01:34:569640.7127000.7127000.712700----0.0000000.0000000.000000OtherYesRBMT A (2015)
3ORGANIZERJPCko-ja2016/07/13 01:35:379650.8506100.8506100.850610----0.0000000.0000000.000000SMTNoHierarchical Phrase-based SMT (2015)
4EHRJPCko-ja2016/07/17 15:54:3110040.8516600.8516600.851660----0.0000000.0000000.000000SMTYesComb. of PBSMT by word and character. DL=0. Parens in Ko side and not in Ja side are added to Ja training and dev sets. After decoding, parens surrounding number are deleted.
5EHRJPCko-ja2016/07/17 18:23:5510050.8663700.8663700.866370----0.0000000.0000000.000000SMTYesCombination of word-based PBSMT and character-based PBSMT with DL=0. Parentheses surrounding number in Korean sentences are deleted.
6EHRJPCko-ja2016/07/17 18:25:4610060.8180300.8180300.818030----0.0000000.0000000.000000SMTYesCombination of word-based PBSMT and character-based PBSMT with DL=0. Parentheses in Korean side and not in Japanese side are added to Japanese for training and dev sets.
7ORGANIZERJPCko-ja2016/07/26 11:16:0210390.7987500.7987500.798750----0.0000000.0000000.000000OtherYesOnline A (2016)
8JAPIOJPCko-ja2016/08/19 15:34:3912060.8581900.8581900.858190----0.0000000.0000000.000000SMTYesPhrase-based SMT + JAPIO corpus + rule-based posteditor
9JAPIOJPCko-ja2016/08/19 17:10:5612090.8636600.8636600.863660----0.0000000.0000000.000000SMTYesPhrase-based SMT + JPC/JAPIO corpora + rule-based posteditor
10transictktJPCko-ja2016/08/21 19:13:0912670.8568300.8568300.856830----0.0000000.0000000.000000SMTNoPhrase-based SMT.
11JAPIOJPCko-ja2016/10/28 13:37:0813310.8597900.8597900.859790----0.0000000.0000000.000000SMTNoPhrase-based SMT + rule-based posteditor
12ORGANIZERJPCko-ja2016/11/25 11:27:1513440.7838500.7838500.783850----0.0000000.0000000.000000NMTNoOnline A (2016/11/25)
13EHRJPCko-ja2017/07/19 21:55:3914160.8660600.8660600.866060----0.0000000.0000000.000000NMTNoSMT reranked NMT (word based, by Moses and OpenNMT)
14EHRJPCko-ja2017/07/19 21:57:1814170.8711100.8711100.871110----0.0000000.0000000.000000NMTNoSMT reranked NMT (character based, by Moses and OpenNMT)
15EHRJPCko-ja2017/07/19 21:58:4814180.8623400.8623400.862340----0.0000000.0000000.000000NMTNoSimple NMT (word based, by OpenNMT)
16EHRJPCko-ja2017/07/19 22:00:2514190.8605200.8605200.860520----0.0000000.0000000.000000NMTNoSimple NMT (character based, by OpenNMT)
17JAPIOJPCko-ja2017/07/25 17:12:4614480.8725100.8725100.872510----0.0000000.0000000.000000SMTYesWord-based PBSMT + JPC/Japio corpora + rule-based posteditor
18JAPIOJPCko-ja2017/07/25 17:15:5414490.8681700.8681700.868170----0.0000000.0000000.000000SMTYesCharacter-based PBSMT + JPC/Japio corpora + rule-based posteditor
19JAPIOJPCko-ja2017/07/25 17:18:1714500.8732000.8732000.873200----0.0000000.0000000.000000SMTYesCombination of word- and character-based PBSMTs
20ORGANIZERJPCko-ja2018/08/15 18:44:1019660.8687600.8687600.868760-----0.0000000.000000NMTNoNMT with Attention
21EHRJPCko-ja2018/08/31 19:01:1522150.8695400.8695400.869540-----0.0000000.000000NMTNoSMT reranked NMT
22ryanJPCko-ja2019/07/23 11:57:1628380.0000000.0000000.000000-------NMTNoBase Transformer
23sarahJPCko-ja2019/07/26 16:27:4930160.0000000.0000000.000000-------NMTNoTransformer, ensemble of 4 models
24KNU_HyundaiJPCko-ja2019/07/26 23:59:0330930.0000000.0000000.000000-------NMTNoTransformer + Relative Position + ensemble of 11 models
25TMUJPCko-ja2020/09/15 07:33:1037210.0000000.0000000.000000-------NMTNoTransformer, domain adaptation (BERT Japanese), hanja loss, shared EMB, shared BPE, ensemble of 4 models
26TMUJPCko-ja2020/09/15 20:51:1837570.0000000.0000000.000000-------NMTNoTransformer, domain adaptation (BERT Japanese), shared EMB, shared BPE, ensemble of 4 models
27goku20JPCko-ja2020/09/21 12:11:2640910.0000000.0000000.000000-------NMTNomBART pre-training transformer, single model
28goku20JPCko-ja2020/09/22 10:17:1341200.0000000.0000000.000000-------NMTNomBART pre-training transformer, ensemble of 3 models
29TMUJPCko-ja2020/10/13 09:43:0041380.0000000.0000000.000000-------NMTNoTransformer, domain adaptation (BERT Korean), hanja loss, shared EMB, shared BPE, ensemble of 4 models
30TMUJPCko-ja2020/10/13 10:00:0041420.0000000.0000000.000000-------NMTNoTransformer, domain adaptation (BERT Korean), shared EMB, shared BPE, ensemble of 4 models
31Bering LabJPCko-ja2021/04/25 02:59:5855130.9419310.9419310.941931-------NMTYesTransformer Ensemble with additional crawled parallel corpus
32tpt_watJPCko-ja2021/04/27 02:01:3656990.9275900.9275900.927590-------NMTNoBase Transformer model with joint vocab, size 8k
33ORGANIZERJPCko-ja2015/06/10 18:44:084380.8562200.8562200.8562200.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoPhrase-based SMT
34ORGANIZERJPCko-ja2015/06/10 18:46:344390.8505600.8505600.8505600.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoHierarchical Phrase-based SMT (2015)
35EHRJPCko-ja2015/07/22 12:24:015000.8296500.8296500.8296500.0000000.0000000.0000000.0000000.0000000.0000000.000000SMT and RBMTYesSystem combination of RBMT with user dictionary plus SPE and phrase based SMT. Candidate selection by language model score.
36NICTJPCko-ja2015/07/23 14:13:385010.8635400.8635400.8635400.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoour baseline: character-based / PB SMT in MOSES (DL=0, max-phrase-len=9, no lex-reordering) / SRILM (9-gram)
37NICTJPCko-ja2015/07/24 14:00:065130.8643500.8643500.8643500.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoour baseline + post-processing of bracket balancing
38TOSHIBAJPCko-ja2015/07/31 19:11:045520.8592300.8592300.8592300.0000000.0000000.0000000.0000000.0000000.0000000.000000SMT and RBMTYesPhrase-based SPE with RNNLM reranking + post-processing
39TOSHIBAJPCko-ja2015/07/31 19:15:005540.8613000.8613000.8613000.0000000.0000000.0000000.0000000.0000000.0000000.000000SMT and RBMTYesSystem combination SMT, SPE and RBMT with RNNLM + post-processing
40TOSHIBAJPCko-ja2015/08/03 09:37:495680.8661300.8661300.8661300.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTYesPhrase-based SMT with RNNLM reranking + post-processing
41naverJPCko-ja2015/08/11 17:38:596300.8655500.8655500.8655500.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNocombined two phrase-based systems, post-processing
42SenseJPCko-ja2015/08/14 10:07:016430.9159600.9159600.9159600.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoUnicode2Stringwith devtest. MERT run1
43ORGANIZERJPCko-ja2015/08/14 17:17:316510.7287500.7287500.7287500.0000000.0000000.0000000.0000000.0000000.0000000.000000OtherYesOnline B (2015)
44ORGANIZERJPCko-ja2015/08/14 17:30:216520.8004600.8004600.8004600.0000000.0000000.0000000.0000000.0000000.0000000.000000OtherYesOnline A (2015)
45ORGANIZERJPCko-ja2015/08/14 17:37:576530.7120200.7120200.7120200.0000000.0000000.0000000.0000000.0000000.0000000.000000OtherYesRBMT A (2015)
46ORGANIZERJPCko-ja2015/08/14 17:40:116540.6431500.6431500.6431500.0000000.0000000.0000000.0000000.0000000.0000000.000000OtherYesRBMT B
47SenseJPCko-ja2015/08/15 15:36:346570.9192100.9192100.9192100.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoUnicode2String with devtest MERT run2
48TOSHIBAJPCko-ja2015/08/17 11:55:226680.8197700.8197700.8197700.0000000.0000000.0000000.0000000.0000000.0000000.000000RBMTYesRBMT
49SenseJPCko-ja2015/08/19 02:23:017190.9193800.9193800.9193800.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNo에이트->エイト
50SenseJPCko-ja2015/08/19 21:23:177230.7788600.7788600.7788600.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoBytes2String
51SenseJPCko-ja2015/08/19 21:26:287240.8159900.8159900.8159900.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoUnicode2String
52naverJPCko-ja2015/08/28 17:07:108160.8644500.8644500.8644500.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoSMT PB + NMT reranking
53EHRJPCko-ja2015/08/30 15:52:078310.8224200.8224200.8224200.0000000.0000000.0000000.0000000.0000000.0000000.000000SMT and RBMTYesRBMT with user dictionary plus SPE
54SenseJPCko-ja2015/08/31 05:00:358330.8607400.8607400.8607400.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTNoBaseline with only train.ja/ko
55EHRJPCko-ja2015/09/02 16:39:468650.8247500.8247500.8247500.0000000.0000000.0000000.0000000.0000000.0000000.000000SMTYesPhrase based SMT

Notice:

Back to top

HUMAN (WAT2022)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description

Notice:
Back to top

HUMAN (WAT2021)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description

Notice:
Back to top

HUMAN (WAT2020)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description

Notice:
Back to top

HUMAN (WAT2019)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description
1sarahJPCko-ja2019/07/26 16:27:493016UnderwayNMTNoTransformer, ensemble of 4 models
2KNU_HyundaiJPCko-ja2019/07/26 23:59:033093UnderwayNMTNoTransformer + Relative Position + ensemble of 11 models

Notice:
Back to top

HUMAN (WAT2018)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description

Notice:
Back to top

HUMAN (WAT2017)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description
1JAPIOJPCko-ja2017/07/25 17:12:46144848.750SMTYesWord-based PBSMT + JPC/Japio corpora + rule-based posteditor
2JAPIOJPCko-ja2017/07/25 17:18:17145048.500SMTYesCombination of word- and character-based PBSMTs
3EHRJPCko-ja2017/07/19 21:57:18141711.250NMTNoSMT reranked NMT (character based, by Moses and OpenNMT)
4EHRJPCko-ja2017/07/19 21:55:3914166.250NMTNoSMT reranked NMT (word based, by Moses and OpenNMT)

Notice:
Back to top

HUMAN (WAT2016)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description
1EHRJPCko-ja2016/07/17 18:25:46100621.750SMTYesCombination of word-based PBSMT and character-based PBSMT with DL=0. Parentheses in Korean side and not in Japanese side are added to Japanese for training and dev sets.
2JAPIOJPCko-ja2016/08/19 17:10:56120917.500SMTYesPhrase-based SMT + JPC/JAPIO corpora + rule-based posteditor
3ORGANIZERJPCko-ja2016/07/26 11:16:0210398.000OtherYesOnline A (2016)
4EHRJPCko-ja2016/07/17 18:23:551005-3.000SMTYesCombination of word-based PBSMT and character-based PBSMT with DL=0. Parentheses surrounding number in Korean sentences are deleted.
5JAPIOJPCko-ja2016/08/19 15:34:391206-9.000SMTYesPhrase-based SMT + JAPIO corpus + rule-based posteditor
6ORGANIZERJPCko-ja2016/11/25 11:27:151344-55.750NMTNoOnline A (2016/11/25)

Notice:
Back to top

HUMAN (WAT2015)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description
1ORGANIZERJPCko-ja2015/08/14 17:30:2165238.750OtherYesOnline A (2015)
2naverJPCko-ja2015/08/28 17:07:1081614.750SMTNoSMT PB + NMT reranking
3NICTJPCko-ja2015/07/24 14:00:0651310.500SMTNoour baseline + post-processing of bracket balancing
4EHRJPCko-ja2015/07/22 12:24:0150010.250SMT and RBMTYesSystem combination of RBMT with user dictionary plus SPE and phrase based SMT. Candidate selection by language model score.
5NICTJPCko-ja2015/07/23 14:13:385018.250SMTNoour baseline: character-based / PB SMT in MOSES (DL=0, max-phrase-len=9, no lex-reordering) / SRILM (9-gram)
6naverJPCko-ja2015/08/11 17:38:596306.750SMTNocombined two phrase-based systems, post-processing
7EHRJPCko-ja2015/08/30 15:52:078316.500SMT and RBMTYesRBMT with user dictionary plus SPE
8TOSHIBAJPCko-ja2015/08/03 09:37:495684.500SMTYesPhrase-based SMT with RNNLM reranking + post-processing
9SenseJPCko-ja2015/08/31 05:00:358333.500SMTNoBaseline with only train.ja/ko
10TOSHIBAJPCko-ja2015/07/31 19:15:005543.000SMT and RBMTYesSystem combination SMT, SPE and RBMT with RNNLM + post-processing
11ORGANIZERJPCko-ja2015/06/10 18:46:344392.750SMTNoHierarchical Phrase-based SMT (2015)
12ORGANIZERJPCko-ja2015/08/14 17:37:57653-7.250OtherYesRBMT A (2015)
13SenseJPCko-ja2015/08/15 15:36:34657-17.750SMTNoUnicode2String with devtest MERT run2

Notice:
Back to top

HUMAN (WAT2014)


# Team Task Date/Time DataID HUMAN
Method
Other
Resources
System
Description

Notice:
Back to top

EVALUATION RESULTS USAGE POLICY

When you use the WAT evaluation results for any purpose such as:
- writing technical papers,
- making presentations about your system,
- advertising your MT system to the customers,
you can use the information about translation directions, scores (including both automatic and human evaluations) and ranks of your system among others. You can also use the scores of the other systems, but you MUST anonymize the other system's names. In addition, you can show the links (URLs) to the WAT evaluation result pages.

NICT (National Institute of Information and Communications Technology)
Kyoto University
Last Modified: 2018-08-02