WAT

The Workshop on Asian Translation

Evaluation Results

BLEU

#	Team	Task	Date/Time	DataID	BLEU										Method	Other Resources	System Description
#	Team	Task	Date/Time	DataID	juman	kytea	mecab	moses- tokenizer	stanford- segmenter- ctb	stanford- segmenter- pku	indic- tokenizer	unuse	myseg	kmseg	Method	Other Resources	System Description
1	ryan	JPCko-ja	2019/07/23 11:57:16	2838	73.53	74.31	73.78	-	-	-	-	-	-	-	NMT	No	Base Transformer
2	sarah	JPCko-ja	2019/07/26 16:27:49	3016	73.16	73.93	73.40	-	-	-	-	-	-	-	NMT	No	Transformer, ensemble of 4 models
3	KNU_Hyundai	JPCko-ja	2019/07/26 23:59:03	3093	73.73	74.55	74.03	-	-	-	-	-	-	-	NMT	No	Transformer + Relative Position + ensemble of 11 models
4	TMU	JPCko-ja	2020/09/15 07:33:10	3721	73.61	74.42	73.90	-	-	-	-	-	-	-	NMT	No	Transformer, domain adaptation (BERT Japanese), hanja loss, shared EMB, shared BPE, ensemble of 4 models
5	TMU	JPCko-ja	2020/09/15 20:51:18	3757	73.56	74.41	73.85	-	-	-	-	-	-	-	NMT	No	Transformer, domain adaptation (BERT Japanese), shared EMB, shared BPE, ensemble of 4 models
6	goku20	JPCko-ja	2020/09/21 12:11:26	4091	68.39	69.39	68.72	-	-	-	-	-	-	-	NMT	No	mBART pre-training transformer, single model
7	goku20	JPCko-ja	2020/09/22 10:17:13	4120	69.64	70.54	69.87	-	-	-	-	-	-	-	NMT	No	mBART pre-training transformer, ensemble of 3 models
8	TMU	JPCko-ja	2020/10/13 09:43:00	4138	73.61	74.42	73.90	-	-	-	-	-	-	-	NMT	No	Transformer, domain adaptation (BERT Korean), hanja loss, shared EMB, shared BPE, ensemble of 4 models
9	TMU	JPCko-ja	2020/10/13 10:00:00	4142	73.56	74.41	73.85	-	-	-	-	-	-	-	NMT	No	Transformer, domain adaptation (BERT Korean), shared EMB, shared BPE, ensemble of 4 models
10	Bering Lab	JPCko-ja	2021/04/25 02:59:58	5513	76.21	76.71	76.36	-	-	-	-	-	-	-	NMT	Yes	Transformer Ensemble with additional crawled parallel corpus
11	tpt_wat	JPCko-ja	2021/04/27 02:01:36	5699	66.76	67.54	66.61	-	-	-	-	-	-	-	NMT	No	Base Transformer model with joint vocab, size 8k
12	ORGANIZER	JPCko-ja	2015/06/10 18:44:08	438	69.22	70.36	69.73	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Phrase-based SMT
13	ORGANIZER	JPCko-ja	2015/06/10 18:46:34	439	67.41	68.65	68.00	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Hierarchical Phrase-based SMT (2015)
14	EHR	JPCko-ja	2015/07/22 12:24:01	500	70.67	71.52	70.93	-	-	-	0.00	0.00	0.00	0.00	SMT and RBMT	Yes	System combination of RBMT with user dictionary plus SPE and phrase based SMT. Candidate selection by language model score.
15	NICT	JPCko-ja	2015/07/23 14:13:38	501	70.62	71.52	70.92	-	-	-	0.00	0.00	0.00	0.00	SMT	No	our baseline: character-based / PB SMT in MOSES (DL=0, max-phrase-len=9, no lex-reordering) / SRILM (9-gram)
16	NICT	JPCko-ja	2015/07/24 14:00:06	513	70.81	71.70	71.11	-	-	-	0.00	0.00	0.00	0.00	SMT	No	our baseline + post-processing of bracket balancing
17	TOSHIBA	JPCko-ja	2015/07/31 19:11:04	552	68.83	69.28	69.04	-	-	-	0.00	0.00	0.00	0.00	SMT and RBMT	Yes	Phrase-based SPE with RNNLM reranking + post-processing
18	TOSHIBA	JPCko-ja	2015/07/31 19:15:00	554	70.51	70.84	70.71	-	-	-	0.00	0.00	0.00	0.00	SMT and RBMT	Yes	System combination SMT, SPE and RBMT with RNNLM + post-processing
19	TOSHIBA	JPCko-ja	2015/08/03 09:37:49	568	71.01	71.44	71.26	-	-	-	0.00	0.00	0.00	0.00	SMT	Yes	Phrase-based SMT with RNNLM reranking + post-processing
20	naver	JPCko-ja	2015/08/11 17:38:59	630	70.91	71.76	71.18	-	-	-	0.00	0.00	0.00	0.00	SMT	No	combined two phrase-based systems, post-processing
21	Sense	JPCko-ja	2015/08/14 10:07:01	643	84.26	84.23	84.26	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Unicode2Stringwith devtest. MERT run1
22	ORGANIZER	JPCko-ja	2015/08/14 17:17:31	651	36.41	38.72	37.01	-	-	-	0.00	0.00	0.00	0.00	Other	Yes	Online B (2015)
23	ORGANIZER	JPCko-ja	2015/08/14 17:30:21	652	55.05	56.84	55.46	-	-	-	0.00	0.00	0.00	0.00	Other	Yes	Online A (2015)
24	ORGANIZER	JPCko-ja	2015/08/14 17:37:57	653	42.00	43.97	42.45	-	-	-	0.00	0.00	0.00	0.00	Other	Yes	RBMT A (2015)
25	ORGANIZER	JPCko-ja	2015/08/14 17:40:11	654	34.74	37.51	35.54	-	-	-	0.00	0.00	0.00	0.00	Other	Yes	RBMT B
26	Sense	JPCko-ja	2015/08/15 15:36:34	657	85.23	85.20	85.23	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Unicode2String with devtest MERT run2
27	TOSHIBA	JPCko-ja	2015/08/17 11:55:22	668	58.60	59.34	58.49	-	-	-	0.00	0.00	0.00	0.00	RBMT	Yes	RBMT
28	Sense	JPCko-ja	2015/08/19 02:23:01	719	85.24	85.21	85.23	-	-	-	0.00	0.00	0.00	0.00	SMT	No	에이트->エイト
29	Sense	JPCko-ja	2015/08/19 21:23:17	723	71.63	71.72	70.90	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Bytes2String
30	Sense	JPCko-ja	2015/08/19 21:26:28	724	46.04	48.30	46.62	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Unicode2String
31	naver	JPCko-ja	2015/08/28 17:07:10	816	71.38	72.27	71.68	-	-	-	0.00	0.00	0.00	0.00	SMT	No	SMT PB + NMT reranking
32	EHR	JPCko-ja	2015/08/30 15:52:07	831	70.13	70.86	70.35	-	-	-	0.00	0.00	0.00	0.00	SMT and RBMT	Yes	RBMT with user dictionary plus SPE
33	Sense	JPCko-ja	2015/08/31 05:00:35	833	70.23	71.11	70.51	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Baseline with only train.ja/ko
34	EHR	JPCko-ja	2015/09/02 16:39:46	865	70.81	71.57	71.00	-	-	-	0.00	0.00	0.00	0.00	SMT	Yes	Phrase based SMT
35	ORGANIZER	JPCko-ja	2016/07/13 00:54:10	963	55.05	56.84	55.46	-	-	-	-	0.00	0.00	0.00	Other	Yes	Online A (2015)
36	ORGANIZER	JPCko-ja	2016/07/13 01:34:56	964	42.00	43.97	42.45	-	-	-	-	0.00	0.00	0.00	Other	Yes	RBMT A (2015)
37	ORGANIZER	JPCko-ja	2016/07/13 01:35:37	965	67.41	68.65	68.00	-	-	-	-	0.00	0.00	0.00	SMT	No	Hierarchical Phrase-based SMT (2015)
38	EHR	JPCko-ja	2016/07/17 15:54:31	1004	68.78	69.65	69.06	-	-	-	-	0.00	0.00	0.00	SMT	Yes	Comb. of PBSMT by word and character. DL=0. Parens in Ko side and not in Ja side are added to Ja training and dev sets. After decoding, parens surrounding number are deleted.
39	EHR	JPCko-ja	2016/07/17 18:23:55	1005	71.51	72.32	71.77	-	-	-	-	0.00	0.00	0.00	SMT	Yes	Combination of word-based PBSMT and character-based PBSMT with DL=0. Parentheses surrounding number in Korean sentences are deleted.
40	EHR	JPCko-ja	2016/07/17 18:25:46	1006	62.33	64.17	62.75	-	-	-	-	0.00	0.00	0.00	SMT	Yes	Combination of word-based PBSMT and character-based PBSMT with DL=0. Parentheses in Korean side and not in Japanese side are added to Japanese for training and dev sets.
41	ORGANIZER	JPCko-ja	2016/07/26 11:16:02	1039	54.78	56.68	55.14	-	-	-	-	0.00	0.00	0.00	Other	Yes	Online A (2016)
42	JAPIO	JPCko-ja	2016/08/19 15:34:39	1206	68.62	69.49	68.90	-	-	-	-	0.00	0.00	0.00	SMT	Yes	Phrase-based SMT + JAPIO corpus + rule-based posteditor
43	JAPIO	JPCko-ja	2016/08/19 17:10:56	1209	70.32	71.07	70.52	-	-	-	-	0.00	0.00	0.00	SMT	Yes	Phrase-based SMT + JPC/JAPIO corpora + rule-based posteditor
44	transictkt	JPCko-ja	2016/08/21 19:13:09	1267	69.30	69.42	69.77	-	-	-	-	0.00	0.00	0.00	SMT	No	Phrase-based SMT.
45	JAPIO	JPCko-ja	2016/10/28 13:37:08	1331	69.10	70.03	69.34	-	-	-	-	0.00	0.00	0.00	SMT	No	Phrase-based SMT + rule-based posteditor
46	ORGANIZER	JPCko-ja	2016/11/25 11:27:15	1344	44.42	45.14	44.72	-	-	-	-	0.00	0.00	0.00	NMT	No	Online A (2016/11/25)
47	EHR	JPCko-ja	2017/07/19 21:55:39	1416	71.52	72.34	71.82	-	-	-	-	0.00	0.00	0.00	NMT	No	SMT reranked NMT (word based, by Moses and OpenNMT)
48	EHR	JPCko-ja	2017/07/19 21:57:18	1417	71.36	72.26	71.65	-	-	-	-	0.00	0.00	0.00	NMT	No	SMT reranked NMT (character based, by Moses and OpenNMT)
49	EHR	JPCko-ja	2017/07/19 21:58:48	1418	70.23	71.14	70.54	-	-	-	-	0.00	0.00	0.00	NMT	No	Simple NMT (word based, by OpenNMT)
50	EHR	JPCko-ja	2017/07/19 22:00:25	1419	69.42	70.28	69.70	-	-	-	-	0.00	0.00	0.00	NMT	No	Simple NMT (character based, by OpenNMT)
51	JAPIO	JPCko-ja	2017/07/25 17:12:46	1448	73.00	73.71	73.23	-	-	-	-	0.00	0.00	0.00	SMT	Yes	Word-based PBSMT + JPC/Japio corpora + rule-based posteditor
52	JAPIO	JPCko-ja	2017/07/25 17:15:54	1449	71.97	72.91	72.17	-	-	-	-	0.00	0.00	0.00	SMT	Yes	Character-based PBSMT + JPC/Japio corpora + rule-based posteditor
53	JAPIO	JPCko-ja	2017/07/25 17:18:17	1450	73.00	73.73	73.25	-	-	-	-	0.00	0.00	0.00	SMT	Yes	Combination of word- and character-based PBSMTs
54	ORGANIZER	JPCko-ja	2018/08/15 18:44:10	1966	71.42	72.21	71.68	-	-	-	-	-	0.00	0.00	NMT	No	NMT with Attention
55	EHR	JPCko-ja	2018/08/31 19:01:15	2215	71.45	72.35	71.74	-	-	-	-	-	0.00	0.00	NMT	No	SMT reranked NMT

Notice:

This table is sorted by the leftmost segmenters. You can change the segmenter used to sort by clicking each segmenter link.

RIBES

#	Team	Task	Date/Time	DataID	RIBES										Method	Other Resources	System Description
#	Team	Task	Date/Time	DataID	juman	kytea	mecab	moses- tokenizer	stanford- segmenter- ctb	stanford- segmenter- pku	indic- tokenizer	unuse	myseg	kmseg	Method	Other Resources	System Description
1	ryan	JPCko-ja	2019/07/23 11:57:16	2838	0.949954	0.948980	0.949567	-	-	-	-	-	-	-	NMT	No	Base Transformer
2	sarah	JPCko-ja	2019/07/26 16:27:49	3016	0.949575	0.948570	0.949128	-	-	-	-	-	-	-	NMT	No	Transformer, ensemble of 4 models
3	KNU_Hyundai	JPCko-ja	2019/07/26 23:59:03	3093	0.950000	0.948924	0.949410	-	-	-	-	-	-	-	NMT	No	Transformer + Relative Position + ensemble of 11 models
4	TMU	JPCko-ja	2020/09/15 07:33:10	3721	0.950109	0.949208	0.949567	-	-	-	-	-	-	-	NMT	No	Transformer, domain adaptation (BERT Japanese), hanja loss, shared EMB, shared BPE, ensemble of 4 models
5	TMU	JPCko-ja	2020/09/15 20:51:18	3757	0.949983	0.948908	0.949484	-	-	-	-	-	-	-	NMT	No	Transformer, domain adaptation (BERT Japanese), shared EMB, shared BPE, ensemble of 4 models
6	goku20	JPCko-ja	2020/09/21 12:11:26	4091	0.943029	0.941926	0.942480	-	-	-	-	-	-	-	NMT	No	mBART pre-training transformer, single model
7	goku20	JPCko-ja	2020/09/22 10:17:13	4120	0.945449	0.944321	0.944852	-	-	-	-	-	-	-	NMT	No	mBART pre-training transformer, ensemble of 3 models
8	TMU	JPCko-ja	2020/10/13 09:43:00	4138	0.950109	0.949208	0.949567	-	-	-	-	-	-	-	NMT	No	Transformer, domain adaptation (BERT Korean), hanja loss, shared EMB, shared BPE, ensemble of 4 models
9	TMU	JPCko-ja	2020/10/13 10:00:00	4142	0.949983	0.948908	0.949484	-	-	-	-	-	-	-	NMT	No	Transformer, domain adaptation (BERT Korean), shared EMB, shared BPE, ensemble of 4 models
10	Bering Lab	JPCko-ja	2021/04/25 02:59:58	5513	0.952101	0.951261	0.951440	-	-	-	-	-	-	-	NMT	Yes	Transformer Ensemble with additional crawled parallel corpus
11	tpt_wat	JPCko-ja	2021/04/27 02:01:36	5699	0.939024	0.937699	0.938299	-	-	-	-	-	-	-	NMT	No	Base Transformer model with joint vocab, size 8k
12	ORGANIZER	JPCko-ja	2015/06/10 18:44:08	438	0.941302	0.939729	0.940756	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Phrase-based SMT
13	ORGANIZER	JPCko-ja	2015/06/10 18:46:34	439	0.937162	0.935903	0.936570	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Hierarchical Phrase-based SMT (2015)
14	EHR	JPCko-ja	2015/07/22 12:24:01	500	0.943040	0.941893	0.942771	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	System combination of RBMT with user dictionary plus SPE and phrase based SMT. Candidate selection by language model score.
15	NICT	JPCko-ja	2015/07/23 14:13:38	501	0.943348	0.942402	0.942748	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	our baseline: character-based / PB SMT in MOSES (DL=0, max-phrase-len=9, no lex-reordering) / SRILM (9-gram)
16	NICT	JPCko-ja	2015/07/24 14:00:06	513	0.943463	0.942519	0.942904	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	our baseline + post-processing of bracket balancing
17	TOSHIBA	JPCko-ja	2015/07/31 19:11:04	552	0.939753	0.937210	0.939180	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	Phrase-based SPE with RNNLM reranking + post-processing
18	TOSHIBA	JPCko-ja	2015/07/31 19:15:00	554	0.942183	0.939545	0.941471	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	System combination SMT, SPE and RBMT with RNNLM + post-processing
19	TOSHIBA	JPCko-ja	2015/08/03 09:37:49	568	0.943794	0.941287	0.943181	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Phrase-based SMT with RNNLM reranking + post-processing
20	naver	JPCko-ja	2015/08/11 17:38:59	630	0.943928	0.942800	0.943376	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	combined two phrase-based systems, post-processing
21	Sense	JPCko-ja	2015/08/14 10:07:01	643	0.950318	0.950109	0.950235	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Unicode2Stringwith devtest. MERT run1
22	ORGANIZER	JPCko-ja	2015/08/14 17:17:31	651	0.851745	0.852263	0.851945	-	-	-	0.000000	0.000000	0.000000	0.000000	Other	Yes	Online B (2015)
23	ORGANIZER	JPCko-ja	2015/08/14 17:30:21	652	0.909152	0.909385	0.908838	-	-	-	0.000000	0.000000	0.000000	0.000000	Other	Yes	Online A (2015)
24	ORGANIZER	JPCko-ja	2015/08/14 17:37:57	653	0.876396	0.873734	0.875146	-	-	-	0.000000	0.000000	0.000000	0.000000	Other	Yes	RBMT A (2015)
25	ORGANIZER	JPCko-ja	2015/08/14 17:40:11	654	0.845712	0.849014	0.846228	-	-	-	0.000000	0.000000	0.000000	0.000000	Other	Yes	RBMT B
26	Sense	JPCko-ja	2015/08/15 15:36:34	657	0.954506	0.954448	0.954435	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Unicode2String with devtest MERT run2
27	TOSHIBA	JPCko-ja	2015/08/17 11:55:22	668	0.916776	0.913982	0.915122	-	-	-	0.000000	0.000000	0.000000	0.000000	RBMT	Yes	RBMT
28	Sense	JPCko-ja	2015/08/19 02:23:01	719	0.954498	0.954463	0.954435	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	에이트->エイト
29	Sense	JPCko-ja	2015/08/19 21:23:17	723	0.932803	0.931268	0.930872	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Bytes2String
30	Sense	JPCko-ja	2015/08/19 21:26:28	724	0.851041	0.854298	0.851636	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Unicode2String
31	naver	JPCko-ja	2015/08/28 17:07:10	816	0.943814	0.942805	0.943584	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	SMT PB + NMT reranking
32	EHR	JPCko-ja	2015/08/30 15:52:07	831	0.941887	0.940341	0.941517	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	RBMT with user dictionary plus SPE
33	Sense	JPCko-ja	2015/08/31 05:00:35	833	0.942415	0.940887	0.941687	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Baseline with only train.ja/ko
34	EHR	JPCko-ja	2015/09/02 16:39:46	865	0.942999	0.941600	0.942680	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Phrase based SMT
35	ORGANIZER	JPCko-ja	2016/07/13 00:54:10	963	0.909152	0.909385	0.908838	-	-	-	-	0.000000	0.000000	0.000000	Other	Yes	Online A (2015)
36	ORGANIZER	JPCko-ja	2016/07/13 01:34:56	964	0.876396	0.873734	0.875146	-	-	-	-	0.000000	0.000000	0.000000	Other	Yes	RBMT A (2015)
37	ORGANIZER	JPCko-ja	2016/07/13 01:35:37	965	0.937162	0.935903	0.936570	-	-	-	-	0.000000	0.000000	0.000000	SMT	No	Hierarchical Phrase-based SMT (2015)
38	EHR	JPCko-ja	2016/07/17 15:54:31	1004	0.941071	0.940169	0.940682	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	Comb. of PBSMT by word and character. DL=0. Parens in Ko side and not in Ja side are added to Ja training and dev sets. After decoding, parens surrounding number are deleted.
39	EHR	JPCko-ja	2016/07/17 18:23:55	1005	0.944651	0.943514	0.944606	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	Combination of word-based PBSMT and character-based PBSMT with DL=0. Parentheses surrounding number in Korean sentences are deleted.
40	EHR	JPCko-ja	2016/07/17 18:25:46	1006	0.927065	0.927215	0.927017	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	Combination of word-based PBSMT and character-based PBSMT with DL=0. Parentheses in Korean side and not in Japanese side are added to Japanese for training and dev sets.
41	ORGANIZER	JPCko-ja	2016/07/26 11:16:02	1039	0.907320	0.907652	0.906743	-	-	-	-	0.000000	0.000000	0.000000	Other	Yes	Online A (2016)
42	JAPIO	JPCko-ja	2016/08/19 15:34:39	1206	0.938474	0.937066	0.938230	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	Phrase-based SMT + JAPIO corpus + rule-based posteditor
43	JAPIO	JPCko-ja	2016/08/19 17:10:56	1209	0.942137	0.940544	0.941746	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	Phrase-based SMT + JPC/JAPIO corpora + rule-based posteditor
44	transictkt	JPCko-ja	2016/08/21 19:13:09	1267	0.941245	0.937630	0.940497	-	-	-	-	0.000000	0.000000	0.000000	SMT	No	Phrase-based SMT.
45	JAPIO	JPCko-ja	2016/10/28 13:37:08	1331	0.940367	0.939057	0.939990	-	-	-	-	0.000000	0.000000	0.000000	SMT	No	Phrase-based SMT + rule-based posteditor
46	ORGANIZER	JPCko-ja	2016/11/25 11:27:15	1344	0.857642	0.854158	0.857083	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	Online A (2016/11/25)
47	EHR	JPCko-ja	2017/07/19 21:55:39	1416	0.944516	0.942940	0.944219	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	SMT reranked NMT (word based, by Moses and OpenNMT)
48	EHR	JPCko-ja	2017/07/19 21:57:18	1417	0.946126	0.944812	0.945888	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	SMT reranked NMT (character based, by Moses and OpenNMT)
49	EHR	JPCko-ja	2017/07/19 21:58:48	1418	0.943179	0.941695	0.942757	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	Simple NMT (word based, by OpenNMT)
50	EHR	JPCko-ja	2017/07/19 22:00:25	1419	0.936440	0.935621	0.936136	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	Simple NMT (character based, by OpenNMT)
51	JAPIO	JPCko-ja	2017/07/25 17:12:46	1448	0.946880	0.945754	0.946645	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	Word-based PBSMT + JPC/Japio corpora + rule-based posteditor
52	JAPIO	JPCko-ja	2017/07/25 17:15:54	1449	0.944435	0.943618	0.943960	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	Character-based PBSMT + JPC/Japio corpora + rule-based posteditor
53	JAPIO	JPCko-ja	2017/07/25 17:18:17	1450	0.946985	0.945841	0.946745	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	Combination of word- and character-based PBSMTs
54	ORGANIZER	JPCko-ja	2018/08/15 18:44:10	1966	0.945593	0.944644	0.945126	-	-	-	-	-	0.000000	0.000000	NMT	No	NMT with Attention
55	EHR	JPCko-ja	2018/08/31 19:01:15	2215	0.945771	0.944284	0.945497	-	-	-	-	-	0.000000	0.000000	NMT	No	SMT reranked NMT

Notice:

This table is sorted by the leftmost segmenters. You can change the segmenter used to sort by clicking each segmenter link.

AMFM

#	Team	Task	Date/Time	DataID	AMFM										Method	Other Resources	System Description
#	Team	Task	Date/Time	DataID	unuse	unuse	unuse	unuse	unuse	unuse	unuse	unuse	unuse	unuse	Method	Other Resources	System Description
1	ryan	JPCko-ja	2019/07/23 11:57:16	2838	0.000000	0.000000	0.000000	-	-	-	-	-	-	-	NMT	No	Base Transformer
2	sarah	JPCko-ja	2019/07/26 16:27:49	3016	0.000000	0.000000	0.000000	-	-	-	-	-	-	-	NMT	No	Transformer, ensemble of 4 models
3	KNU_Hyundai	JPCko-ja	2019/07/26 23:59:03	3093	0.000000	0.000000	0.000000	-	-	-	-	-	-	-	NMT	No	Transformer + Relative Position + ensemble of 11 models
4	TMU	JPCko-ja	2020/09/15 07:33:10	3721	0.000000	0.000000	0.000000	-	-	-	-	-	-	-	NMT	No	Transformer, domain adaptation (BERT Japanese), hanja loss, shared EMB, shared BPE, ensemble of 4 models
5	TMU	JPCko-ja	2020/09/15 20:51:18	3757	0.000000	0.000000	0.000000	-	-	-	-	-	-	-	NMT	No	Transformer, domain adaptation (BERT Japanese), shared EMB, shared BPE, ensemble of 4 models
6	goku20	JPCko-ja	2020/09/21 12:11:26	4091	0.000000	0.000000	0.000000	-	-	-	-	-	-	-	NMT	No	mBART pre-training transformer, single model
7	goku20	JPCko-ja	2020/09/22 10:17:13	4120	0.000000	0.000000	0.000000	-	-	-	-	-	-	-	NMT	No	mBART pre-training transformer, ensemble of 3 models
8	TMU	JPCko-ja	2020/10/13 09:43:00	4138	0.000000	0.000000	0.000000	-	-	-	-	-	-	-	NMT	No	Transformer, domain adaptation (BERT Korean), hanja loss, shared EMB, shared BPE, ensemble of 4 models
9	TMU	JPCko-ja	2020/10/13 10:00:00	4142	0.000000	0.000000	0.000000	-	-	-	-	-	-	-	NMT	No	Transformer, domain adaptation (BERT Korean), shared EMB, shared BPE, ensemble of 4 models
10	Bering Lab	JPCko-ja	2021/04/25 02:59:58	5513	0.941931	0.941931	0.941931	-	-	-	-	-	-	-	NMT	Yes	Transformer Ensemble with additional crawled parallel corpus
11	tpt_wat	JPCko-ja	2021/04/27 02:01:36	5699	0.927590	0.927590	0.927590	-	-	-	-	-	-	-	NMT	No	Base Transformer model with joint vocab, size 8k
12	ORGANIZER	JPCko-ja	2015/06/10 18:44:08	438	0.856220	0.856220	0.856220	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Phrase-based SMT
13	ORGANIZER	JPCko-ja	2015/06/10 18:46:34	439	0.850560	0.850560	0.850560	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Hierarchical Phrase-based SMT (2015)
14	EHR	JPCko-ja	2015/07/22 12:24:01	500	0.829650	0.829650	0.829650	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	System combination of RBMT with user dictionary plus SPE and phrase based SMT. Candidate selection by language model score.
15	NICT	JPCko-ja	2015/07/23 14:13:38	501	0.863540	0.863540	0.863540	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	our baseline: character-based / PB SMT in MOSES (DL=0, max-phrase-len=9, no lex-reordering) / SRILM (9-gram)
16	NICT	JPCko-ja	2015/07/24 14:00:06	513	0.864350	0.864350	0.864350	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	our baseline + post-processing of bracket balancing
17	TOSHIBA	JPCko-ja	2015/07/31 19:11:04	552	0.859230	0.859230	0.859230	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	Phrase-based SPE with RNNLM reranking + post-processing
18	TOSHIBA	JPCko-ja	2015/07/31 19:15:00	554	0.861300	0.861300	0.861300	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	System combination SMT, SPE and RBMT with RNNLM + post-processing
19	TOSHIBA	JPCko-ja	2015/08/03 09:37:49	568	0.866130	0.866130	0.866130	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Phrase-based SMT with RNNLM reranking + post-processing
20	naver	JPCko-ja	2015/08/11 17:38:59	630	0.865550	0.865550	0.865550	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	combined two phrase-based systems, post-processing
21	Sense	JPCko-ja	2015/08/14 10:07:01	643	0.915960	0.915960	0.915960	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Unicode2Stringwith devtest. MERT run1
22	ORGANIZER	JPCko-ja	2015/08/14 17:17:31	651	0.728750	0.728750	0.728750	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	Other	Yes	Online B (2015)
23	ORGANIZER	JPCko-ja	2015/08/14 17:30:21	652	0.800460	0.800460	0.800460	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	Other	Yes	Online A (2015)
24	ORGANIZER	JPCko-ja	2015/08/14 17:37:57	653	0.712020	0.712020	0.712020	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	Other	Yes	RBMT A (2015)
25	ORGANIZER	JPCko-ja	2015/08/14 17:40:11	654	0.643150	0.643150	0.643150	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	Other	Yes	RBMT B
26	Sense	JPCko-ja	2015/08/15 15:36:34	657	0.919210	0.919210	0.919210	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Unicode2String with devtest MERT run2
27	TOSHIBA	JPCko-ja	2015/08/17 11:55:22	668	0.819770	0.819770	0.819770	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	RBMT	Yes	RBMT
28	Sense	JPCko-ja	2015/08/19 02:23:01	719	0.919380	0.919380	0.919380	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	에이트->エイト
29	Sense	JPCko-ja	2015/08/19 21:23:17	723	0.778860	0.778860	0.778860	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Bytes2String
30	Sense	JPCko-ja	2015/08/19 21:26:28	724	0.815990	0.815990	0.815990	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Unicode2String
31	naver	JPCko-ja	2015/08/28 17:07:10	816	0.864450	0.864450	0.864450	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	SMT PB + NMT reranking
32	EHR	JPCko-ja	2015/08/30 15:52:07	831	0.822420	0.822420	0.822420	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	RBMT with user dictionary plus SPE
33	Sense	JPCko-ja	2015/08/31 05:00:35	833	0.860740	0.860740	0.860740	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Baseline with only train.ja/ko
34	EHR	JPCko-ja	2015/09/02 16:39:46	865	0.824750	0.824750	0.824750	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Phrase based SMT
35	ORGANIZER	JPCko-ja	2016/07/13 00:54:10	963	0.800610	0.800610	0.800610	-	-	-	-	0.000000	0.000000	0.000000	Other	Yes	Online A (2015)
36	ORGANIZER	JPCko-ja	2016/07/13 01:34:56	964	0.712700	0.712700	0.712700	-	-	-	-	0.000000	0.000000	0.000000	Other	Yes	RBMT A (2015)
37	ORGANIZER	JPCko-ja	2016/07/13 01:35:37	965	0.850610	0.850610	0.850610	-	-	-	-	0.000000	0.000000	0.000000	SMT	No	Hierarchical Phrase-based SMT (2015)
38	EHR	JPCko-ja	2016/07/17 15:54:31	1004	0.851660	0.851660	0.851660	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	Comb. of PBSMT by word and character. DL=0. Parens in Ko side and not in Ja side are added to Ja training and dev sets. After decoding, parens surrounding number are deleted.
39	EHR	JPCko-ja	2016/07/17 18:23:55	1005	0.866370	0.866370	0.866370	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	Combination of word-based PBSMT and character-based PBSMT with DL=0. Parentheses surrounding number in Korean sentences are deleted.
40	EHR	JPCko-ja	2016/07/17 18:25:46	1006	0.818030	0.818030	0.818030	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	Combination of word-based PBSMT and character-based PBSMT with DL=0. Parentheses in Korean side and not in Japanese side are added to Japanese for training and dev sets.
41	ORGANIZER	JPCko-ja	2016/07/26 11:16:02	1039	0.798750	0.798750	0.798750	-	-	-	-	0.000000	0.000000	0.000000	Other	Yes	Online A (2016)
42	JAPIO	JPCko-ja	2016/08/19 15:34:39	1206	0.858190	0.858190	0.858190	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	Phrase-based SMT + JAPIO corpus + rule-based posteditor
43	JAPIO	JPCko-ja	2016/08/19 17:10:56	1209	0.863660	0.863660	0.863660	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	Phrase-based SMT + JPC/JAPIO corpora + rule-based posteditor
44	transictkt	JPCko-ja	2016/08/21 19:13:09	1267	0.856830	0.856830	0.856830	-	-	-	-	0.000000	0.000000	0.000000	SMT	No	Phrase-based SMT.
45	JAPIO	JPCko-ja	2016/10/28 13:37:08	1331	0.859790	0.859790	0.859790	-	-	-	-	0.000000	0.000000	0.000000	SMT	No	Phrase-based SMT + rule-based posteditor
46	ORGANIZER	JPCko-ja	2016/11/25 11:27:15	1344	0.783850	0.783850	0.783850	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	Online A (2016/11/25)
47	EHR	JPCko-ja	2017/07/19 21:55:39	1416	0.866060	0.866060	0.866060	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	SMT reranked NMT (word based, by Moses and OpenNMT)
48	EHR	JPCko-ja	2017/07/19 21:57:18	1417	0.871110	0.871110	0.871110	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	SMT reranked NMT (character based, by Moses and OpenNMT)
49	EHR	JPCko-ja	2017/07/19 21:58:48	1418	0.862340	0.862340	0.862340	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	Simple NMT (word based, by OpenNMT)
50	EHR	JPCko-ja	2017/07/19 22:00:25	1419	0.860520	0.860520	0.860520	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	Simple NMT (character based, by OpenNMT)
51	JAPIO	JPCko-ja	2017/07/25 17:12:46	1448	0.872510	0.872510	0.872510	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	Word-based PBSMT + JPC/Japio corpora + rule-based posteditor
52	JAPIO	JPCko-ja	2017/07/25 17:15:54	1449	0.868170	0.868170	0.868170	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	Character-based PBSMT + JPC/Japio corpora + rule-based posteditor
53	JAPIO	JPCko-ja	2017/07/25 17:18:17	1450	0.873200	0.873200	0.873200	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	Combination of word- and character-based PBSMTs
54	ORGANIZER	JPCko-ja	2018/08/15 18:44:10	1966	0.868760	0.868760	0.868760	-	-	-	-	-	0.000000	0.000000	NMT	No	NMT with Attention
55	EHR	JPCko-ja	2018/08/31 19:01:15	2215	0.869540	0.869540	0.869540	-	-	-	-	-	0.000000	0.000000	NMT	No	SMT reranked NMT

Notice:

This table is sorted by the leftmost segmenters. You can change the segmenter used to sort by clicking each segmenter link.
Adequacy-Fluency Metrics (AMFM) is a two-dimensional automatic evaluation metric for machine translation, designed to operate at the sentence level. It is based on adequacy and fluency, to decouple semantic and syntactic components of the translation process to provide a balanced view on translation quality.
AMFM is calculated without tokenizers.
The detail of AMFM is shown on the following paper: "Adequacy–Fluency Metrics: Evaluating MT in the Continuous Space Model Framework" [pdf]. Invited Talk in WAT2015 also helps understanding [slide].

HUMAN (WAT2022)

Notice:

HUMAN (WAT2022) is the result of the Pairwise Crowdsourcing Evaluation on WAT2022.
HUMAN (WAT2022) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.

HUMAN (WAT2021)

Notice:

HUMAN (WAT2021) is the result of the Pairwise Crowdsourcing Evaluation on WAT2021.
HUMAN (WAT2021) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.

HUMAN (WAT2020)

Notice:

HUMAN (WAT2020) is the result of the Pairwise Crowdsourcing Evaluation on WAT2020.
HUMAN (WAT2020) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.

HUMAN (WAT2019)

Notice:

HUMAN (WAT2019) is the result of the Pairwise Crowdsourcing Evaluation on WAT2019.
HUMAN (WAT2019) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.

HUMAN (WAT2018)

Notice:

HUMAN (WAT2018) is the result of the Pairwise Crowdsourcing Evaluation on WAT2018.
HUMAN (WAT2018) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.

HUMAN (WAT2017)

Notice:

HUMAN (WAT2017) is the result of the Pairwise Crowdsourcing Evaluation on WAT2017.
HUMAN (WAT2017) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.

HUMAN (WAT2016)

#	Team	Task	Date/Time	DataID	HUMAN	Method	Other Resources	System Description
1	EHR	JPCko-ja	2016/07/17 18:25:46	1006	21.750	SMT	Yes	Combination of word-based PBSMT and character-based PBSMT with DL=0. Parentheses in Korean side and not in Japanese side are added to Japanese for training and dev sets.
2	JAPIO	JPCko-ja	2016/08/19 17:10:56	1209	17.500	SMT	Yes	Phrase-based SMT + JPC/JAPIO corpora + rule-based posteditor
3	ORGANIZER	JPCko-ja	2016/07/26 11:16:02	1039	8.000	Other	Yes	Online A (2016)
4	EHR	JPCko-ja	2016/07/17 18:23:55	1005	-3.000	SMT	Yes	Combination of word-based PBSMT and character-based PBSMT with DL=0. Parentheses surrounding number in Korean sentences are deleted.
5	JAPIO	JPCko-ja	2016/08/19 15:34:39	1206	-9.000	SMT	Yes	Phrase-based SMT + JAPIO corpus + rule-based posteditor
6	ORGANIZER	JPCko-ja	2016/11/25 11:27:15	1344	-55.750	NMT	No	Online A (2016/11/25)

Notice:

HUMAN (WAT2016) is the result of the Pairwise Crowdsourcing Evaluation on WAT2016.
HUMAN (WAT2016) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.

HUMAN (WAT2015)

#	Team	Task	Date/Time	DataID	HUMAN	Method	Other Resources	System Description
1	ORGANIZER	JPCko-ja	2015/08/14 17:30:21	652	38.750	Other	Yes	Online A (2015)
2	naver	JPCko-ja	2015/08/28 17:07:10	816	14.750	SMT	No	SMT PB + NMT reranking
3	NICT	JPCko-ja	2015/07/24 14:00:06	513	10.500	SMT	No	our baseline + post-processing of bracket balancing
4	EHR	JPCko-ja	2015/07/22 12:24:01	500	10.250	SMT and RBMT	Yes	System combination of RBMT with user dictionary plus SPE and phrase based SMT. Candidate selection by language model score.
5	NICT	JPCko-ja	2015/07/23 14:13:38	501	8.250	SMT	No	our baseline: character-based / PB SMT in MOSES (DL=0, max-phrase-len=9, no lex-reordering) / SRILM (9-gram)
6	naver	JPCko-ja	2015/08/11 17:38:59	630	6.750	SMT	No	combined two phrase-based systems, post-processing
7	EHR	JPCko-ja	2015/08/30 15:52:07	831	6.500	SMT and RBMT	Yes	RBMT with user dictionary plus SPE
8	TOSHIBA	JPCko-ja	2015/08/03 09:37:49	568	4.500	SMT	Yes	Phrase-based SMT with RNNLM reranking + post-processing
9	Sense	JPCko-ja	2015/08/31 05:00:35	833	3.500	SMT	No	Baseline with only train.ja/ko
10	TOSHIBA	JPCko-ja	2015/07/31 19:15:00	554	3.000	SMT and RBMT	Yes	System combination SMT, SPE and RBMT with RNNLM + post-processing
11	ORGANIZER	JPCko-ja	2015/06/10 18:46:34	439	2.750	SMT	No	Hierarchical Phrase-based SMT (2015)
12	ORGANIZER	JPCko-ja	2015/08/14 17:37:57	653	-7.250	Other	Yes	RBMT A (2015)
13	Sense	JPCko-ja	2015/08/15 15:36:34	657	-17.750	SMT	No	Unicode2String with devtest MERT run2

Notice:

HUMAN (WAT2015) is the result of the Pairwise Crowdsourcing Evaluation on WAT2015.
HUMAN (WAT2015) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.
The detail of the evaluation can be found in the pdf document (PDF file).

HUMAN (WAT2014)

Notice:

HUMAN (WAT2014) is the result of the Pairwise Crowdsourcing Evaluation on WAT2014.
HUMAN (WAT2014) was evaluated by 3 different workers and the final decision is made by the voting of the judgements.
The detail of the evaluation can be found in the pdf document (PDF file).

EVALUATION RESULTS USAGE POLICY

When you use the WAT evaluation results for any purpose such as:
- writing technical papers,
- making presentations about your system,
- advertising your MT system to the customers,
you can use the information about translation directions, scores (including both automatic and human evaluations) and ranks of your system among others. You can also use the scores of the other systems, but you MUST anonymize the other system's names. In addition, you can show the links (URLs) to the WAT evaluation result pages.

NICT (National Institute of Information and Communications Technology)
Kyoto University
Last Modified: 2018-08-02

WAT The Workshop on Asian Translation Evaluation Results

BLEU

RIBES

AMFM

HUMAN (WAT2022)

HUMAN (WAT2021)

HUMAN (WAT2020)

HUMAN (WAT2019)

HUMAN (WAT2018)

HUMAN (WAT2017)

HUMAN (WAT2016)

HUMAN (WAT2015)

HUMAN (WAT2014)

EVALUATION RESULTS USAGE POLICY

WAT

The Workshop on Asian Translation

Evaluation Results