WAT

The Workshop on Asian Translation

Evaluation Results

BLEU

#	Team	Task	Date/Time	DataID	BLEU										Method	Other Resources	System Description
#	Team	Task	Date/Time	DataID	juman	kytea	mecab	moses- tokenizer	stanford- segmenter- ctb	stanford- segmenter- pku	indic- tokenizer	unuse	myseg	kmseg	Method	Other Resources	System Description
1	Sense	JPCko-ja	2015/08/15 15:36:34	657	85.23	85.20	85.23	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Unicode2String with devtest MERT run2
2	Sense	JPCko-ja	2015/08/19 02:23:01	719	85.24	85.21	85.23	-	-	-	0.00	0.00	0.00	0.00	SMT	No	에이트->エイト
3	Sense	JPCko-ja	2015/08/14 10:07:01	643	84.26	84.23	84.26	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Unicode2Stringwith devtest. MERT run1
4	Bering Lab	JPCko-ja	2021/04/25 02:59:58	5513	76.21	76.71	76.36	-	-	-	-	-	-	-	NMT	Yes	Transformer Ensemble with additional crawled parallel corpus
5	KNU_Hyundai	JPCko-ja	2019/07/26 23:59:03	3093	73.73	74.55	74.03	-	-	-	-	-	-	-	NMT	No	Transformer + Relative Position + ensemble of 11 models
6	TMU	JPCko-ja	2020/09/15 07:33:10	3721	73.61	74.42	73.90	-	-	-	-	-	-	-	NMT	No	Transformer, domain adaptation (BERT Japanese), hanja loss, shared EMB, shared BPE, ensemble of 4 models
7	TMU	JPCko-ja	2020/10/13 09:43:00	4138	73.61	74.42	73.90	-	-	-	-	-	-	-	NMT	No	Transformer, domain adaptation (BERT Korean), hanja loss, shared EMB, shared BPE, ensemble of 4 models
8	TMU	JPCko-ja	2020/09/15 20:51:18	3757	73.56	74.41	73.85	-	-	-	-	-	-	-	NMT	No	Transformer, domain adaptation (BERT Japanese), shared EMB, shared BPE, ensemble of 4 models
9	TMU	JPCko-ja	2020/10/13 10:00:00	4142	73.56	74.41	73.85	-	-	-	-	-	-	-	NMT	No	Transformer, domain adaptation (BERT Korean), shared EMB, shared BPE, ensemble of 4 models
10	ryan	JPCko-ja	2019/07/23 11:57:16	2838	73.53	74.31	73.78	-	-	-	-	-	-	-	NMT	No	Base Transformer
11	sarah	JPCko-ja	2019/07/26 16:27:49	3016	73.16	73.93	73.40	-	-	-	-	-	-	-	NMT	No	Transformer, ensemble of 4 models
12	JAPIO	JPCko-ja	2017/07/25 17:18:17	1450	73.00	73.73	73.25	-	-	-	-	0.00	0.00	0.00	SMT	Yes	Combination of word- and character-based PBSMTs
13	JAPIO	JPCko-ja	2017/07/25 17:12:46	1448	73.00	73.71	73.23	-	-	-	-	0.00	0.00	0.00	SMT	Yes	Word-based PBSMT + JPC/Japio corpora + rule-based posteditor
14	JAPIO	JPCko-ja	2017/07/25 17:15:54	1449	71.97	72.91	72.17	-	-	-	-	0.00	0.00	0.00	SMT	Yes	Character-based PBSMT + JPC/Japio corpora + rule-based posteditor
15	EHR	JPCko-ja	2017/07/19 21:55:39	1416	71.52	72.34	71.82	-	-	-	-	0.00	0.00	0.00	NMT	No	SMT reranked NMT (word based, by Moses and OpenNMT)
16	EHR	JPCko-ja	2016/07/17 18:23:55	1005	71.51	72.32	71.77	-	-	-	-	0.00	0.00	0.00	SMT	Yes	Combination of word-based PBSMT and character-based PBSMT with DL=0. Parentheses surrounding number in Korean sentences are deleted.
17	EHR	JPCko-ja	2018/08/31 19:01:15	2215	71.45	72.35	71.74	-	-	-	-	-	0.00	0.00	NMT	No	SMT reranked NMT
18	naver	JPCko-ja	2015/08/28 17:07:10	816	71.38	72.27	71.68	-	-	-	0.00	0.00	0.00	0.00	SMT	No	SMT PB + NMT reranking
19	ORGANIZER	JPCko-ja	2018/08/15 18:44:10	1966	71.42	72.21	71.68	-	-	-	-	-	0.00	0.00	NMT	No	NMT with Attention
20	EHR	JPCko-ja	2017/07/19 21:57:18	1417	71.36	72.26	71.65	-	-	-	-	0.00	0.00	0.00	NMT	No	SMT reranked NMT (character based, by Moses and OpenNMT)
21	TOSHIBA	JPCko-ja	2015/08/03 09:37:49	568	71.01	71.44	71.26	-	-	-	0.00	0.00	0.00	0.00	SMT	Yes	Phrase-based SMT with RNNLM reranking + post-processing
22	naver	JPCko-ja	2015/08/11 17:38:59	630	70.91	71.76	71.18	-	-	-	0.00	0.00	0.00	0.00	SMT	No	combined two phrase-based systems, post-processing
23	NICT	JPCko-ja	2015/07/24 14:00:06	513	70.81	71.70	71.11	-	-	-	0.00	0.00	0.00	0.00	SMT	No	our baseline + post-processing of bracket balancing
24	EHR	JPCko-ja	2015/09/02 16:39:46	865	70.81	71.57	71.00	-	-	-	0.00	0.00	0.00	0.00	SMT	Yes	Phrase based SMT
25	EHR	JPCko-ja	2015/07/22 12:24:01	500	70.67	71.52	70.93	-	-	-	0.00	0.00	0.00	0.00	SMT and RBMT	Yes	System combination of RBMT with user dictionary plus SPE and phrase based SMT. Candidate selection by language model score.
26	NICT	JPCko-ja	2015/07/23 14:13:38	501	70.62	71.52	70.92	-	-	-	0.00	0.00	0.00	0.00	SMT	No	our baseline: character-based / PB SMT in MOSES (DL=0, max-phrase-len=9, no lex-reordering) / SRILM (9-gram)
27	Sense	JPCko-ja	2015/08/19 21:23:17	723	71.63	71.72	70.90	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Bytes2String
28	TOSHIBA	JPCko-ja	2015/07/31 19:15:00	554	70.51	70.84	70.71	-	-	-	0.00	0.00	0.00	0.00	SMT and RBMT	Yes	System combination SMT, SPE and RBMT with RNNLM + post-processing
29	EHR	JPCko-ja	2017/07/19 21:58:48	1418	70.23	71.14	70.54	-	-	-	-	0.00	0.00	0.00	NMT	No	Simple NMT (word based, by OpenNMT)
30	JAPIO	JPCko-ja	2016/08/19 17:10:56	1209	70.32	71.07	70.52	-	-	-	-	0.00	0.00	0.00	SMT	Yes	Phrase-based SMT + JPC/JAPIO corpora + rule-based posteditor
31	Sense	JPCko-ja	2015/08/31 05:00:35	833	70.23	71.11	70.51	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Baseline with only train.ja/ko
32	EHR	JPCko-ja	2015/08/30 15:52:07	831	70.13	70.86	70.35	-	-	-	0.00	0.00	0.00	0.00	SMT and RBMT	Yes	RBMT with user dictionary plus SPE
33	goku20	JPCko-ja	2020/09/22 10:17:13	4120	69.64	70.54	69.87	-	-	-	-	-	-	-	NMT	No	mBART pre-training transformer, ensemble of 3 models
34	transictkt	JPCko-ja	2016/08/21 19:13:09	1267	69.30	69.42	69.77	-	-	-	-	0.00	0.00	0.00	SMT	No	Phrase-based SMT.
35	ORGANIZER	JPCko-ja	2015/06/10 18:44:08	438	69.22	70.36	69.73	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Phrase-based SMT
36	EHR	JPCko-ja	2017/07/19 22:00:25	1419	69.42	70.28	69.70	-	-	-	-	0.00	0.00	0.00	NMT	No	Simple NMT (character based, by OpenNMT)
37	JAPIO	JPCko-ja	2016/10/28 13:37:08	1331	69.10	70.03	69.34	-	-	-	-	0.00	0.00	0.00	SMT	No	Phrase-based SMT + rule-based posteditor
38	EHR	JPCko-ja	2016/07/17 15:54:31	1004	68.78	69.65	69.06	-	-	-	-	0.00	0.00	0.00	SMT	Yes	Comb. of PBSMT by word and character. DL=0. Parens in Ko side and not in Ja side are added to Ja training and dev sets. After decoding, parens surrounding number are deleted.
39	TOSHIBA	JPCko-ja	2015/07/31 19:11:04	552	68.83	69.28	69.04	-	-	-	0.00	0.00	0.00	0.00	SMT and RBMT	Yes	Phrase-based SPE with RNNLM reranking + post-processing
40	JAPIO	JPCko-ja	2016/08/19 15:34:39	1206	68.62	69.49	68.90	-	-	-	-	0.00	0.00	0.00	SMT	Yes	Phrase-based SMT + JAPIO corpus + rule-based posteditor
41	goku20	JPCko-ja	2020/09/21 12:11:26	4091	68.39	69.39	68.72	-	-	-	-	-	-	-	NMT	No	mBART pre-training transformer, single model
42	ORGANIZER	JPCko-ja	2015/06/10 18:46:34	439	67.41	68.65	68.00	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Hierarchical Phrase-based SMT (2015)
43	ORGANIZER	JPCko-ja	2016/07/13 01:35:37	965	67.41	68.65	68.00	-	-	-	-	0.00	0.00	0.00	SMT	No	Hierarchical Phrase-based SMT (2015)
44	tpt_wat	JPCko-ja	2021/04/27 02:01:36	5699	66.76	67.54	66.61	-	-	-	-	-	-	-	NMT	No	Base Transformer model with joint vocab, size 8k
45	EHR	JPCko-ja	2016/07/17 18:25:46	1006	62.33	64.17	62.75	-	-	-	-	0.00	0.00	0.00	SMT	Yes	Combination of word-based PBSMT and character-based PBSMT with DL=0. Parentheses in Korean side and not in Japanese side are added to Japanese for training and dev sets.
46	TOSHIBA	JPCko-ja	2015/08/17 11:55:22	668	58.60	59.34	58.49	-	-	-	0.00	0.00	0.00	0.00	RBMT	Yes	RBMT
47	ORGANIZER	JPCko-ja	2015/08/14 17:30:21	652	55.05	56.84	55.46	-	-	-	0.00	0.00	0.00	0.00	Other	Yes	Online A (2015)
48	ORGANIZER	JPCko-ja	2016/07/13 00:54:10	963	55.05	56.84	55.46	-	-	-	-	0.00	0.00	0.00	Other	Yes	Online A (2015)
49	ORGANIZER	JPCko-ja	2016/07/26 11:16:02	1039	54.78	56.68	55.14	-	-	-	-	0.00	0.00	0.00	Other	Yes	Online A (2016)
50	Sense	JPCko-ja	2015/08/19 21:26:28	724	46.04	48.30	46.62	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Unicode2String
51	ORGANIZER	JPCko-ja	2016/11/25 11:27:15	1344	44.42	45.14	44.72	-	-	-	-	0.00	0.00	0.00	NMT	No	Online A (2016/11/25)
52	ORGANIZER	JPCko-ja	2015/08/14 17:37:57	653	42.00	43.97	42.45	-	-	-	0.00	0.00	0.00	0.00	Other	Yes	RBMT A (2015)
53	ORGANIZER	JPCko-ja	2016/07/13 01:34:56	964	42.00	43.97	42.45	-	-	-	-	0.00	0.00	0.00	Other	Yes	RBMT A (2015)
54	ORGANIZER	JPCko-ja	2015/08/14 17:17:31	651	36.41	38.72	37.01	-	-	-	0.00	0.00	0.00	0.00	Other	Yes	Online B (2015)
55	ORGANIZER	JPCko-ja	2015/08/14 17:40:11	654	34.74	37.51	35.54	-	-	-	0.00	0.00	0.00	0.00	Other	Yes	RBMT B

Notice:

This table is sorted by the leftmost segmenters. You can change the segmenter used to sort by clicking each segmenter link.

RIBES

#	Team	Task	Date/Time	DataID	RIBES										Method	Other Resources	System Description
#	Team	Task	Date/Time	DataID	juman	kytea	mecab	moses- tokenizer	stanford- segmenter- ctb	stanford- segmenter- pku	indic- tokenizer	unuse	myseg	kmseg	Method	Other Resources	System Description
1	Sense	JPCko-ja	2015/08/15 15:36:34	657	0.954506	0.954448	0.954435	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Unicode2String with devtest MERT run2
2	Sense	JPCko-ja	2015/08/19 02:23:01	719	0.954498	0.954463	0.954435	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	에이트->エイト
3	Bering Lab	JPCko-ja	2021/04/25 02:59:58	5513	0.952101	0.951261	0.951440	-	-	-	-	-	-	-	NMT	Yes	Transformer Ensemble with additional crawled parallel corpus
4	Sense	JPCko-ja	2015/08/14 10:07:01	643	0.950318	0.950109	0.950235	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Unicode2Stringwith devtest. MERT run1
5	ryan	JPCko-ja	2019/07/23 11:57:16	2838	0.949954	0.948980	0.949567	-	-	-	-	-	-	-	NMT	No	Base Transformer
6	TMU	JPCko-ja	2020/09/15 07:33:10	3721	0.950109	0.949208	0.949567	-	-	-	-	-	-	-	NMT	No	Transformer, domain adaptation (BERT Japanese), hanja loss, shared EMB, shared BPE, ensemble of 4 models
7	TMU	JPCko-ja	2020/10/13 09:43:00	4138	0.950109	0.949208	0.949567	-	-	-	-	-	-	-	NMT	No	Transformer, domain adaptation (BERT Korean), hanja loss, shared EMB, shared BPE, ensemble of 4 models
8	TMU	JPCko-ja	2020/09/15 20:51:18	3757	0.949983	0.948908	0.949484	-	-	-	-	-	-	-	NMT	No	Transformer, domain adaptation (BERT Japanese), shared EMB, shared BPE, ensemble of 4 models
9	TMU	JPCko-ja	2020/10/13 10:00:00	4142	0.949983	0.948908	0.949484	-	-	-	-	-	-	-	NMT	No	Transformer, domain adaptation (BERT Korean), shared EMB, shared BPE, ensemble of 4 models
10	KNU_Hyundai	JPCko-ja	2019/07/26 23:59:03	3093	0.950000	0.948924	0.949410	-	-	-	-	-	-	-	NMT	No	Transformer + Relative Position + ensemble of 11 models
11	sarah	JPCko-ja	2019/07/26 16:27:49	3016	0.949575	0.948570	0.949128	-	-	-	-	-	-	-	NMT	No	Transformer, ensemble of 4 models
12	JAPIO	JPCko-ja	2017/07/25 17:18:17	1450	0.946985	0.945841	0.946745	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	Combination of word- and character-based PBSMTs
13	JAPIO	JPCko-ja	2017/07/25 17:12:46	1448	0.946880	0.945754	0.946645	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	Word-based PBSMT + JPC/Japio corpora + rule-based posteditor
14	EHR	JPCko-ja	2017/07/19 21:57:18	1417	0.946126	0.944812	0.945888	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	SMT reranked NMT (character based, by Moses and OpenNMT)
15	EHR	JPCko-ja	2018/08/31 19:01:15	2215	0.945771	0.944284	0.945497	-	-	-	-	-	0.000000	0.000000	NMT	No	SMT reranked NMT
16	ORGANIZER	JPCko-ja	2018/08/15 18:44:10	1966	0.945593	0.944644	0.945126	-	-	-	-	-	0.000000	0.000000	NMT	No	NMT with Attention
17	goku20	JPCko-ja	2020/09/22 10:17:13	4120	0.945449	0.944321	0.944852	-	-	-	-	-	-	-	NMT	No	mBART pre-training transformer, ensemble of 3 models
18	EHR	JPCko-ja	2016/07/17 18:23:55	1005	0.944651	0.943514	0.944606	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	Combination of word-based PBSMT and character-based PBSMT with DL=0. Parentheses surrounding number in Korean sentences are deleted.
19	EHR	JPCko-ja	2017/07/19 21:55:39	1416	0.944516	0.942940	0.944219	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	SMT reranked NMT (word based, by Moses and OpenNMT)
20	JAPIO	JPCko-ja	2017/07/25 17:15:54	1449	0.944435	0.943618	0.943960	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	Character-based PBSMT + JPC/Japio corpora + rule-based posteditor
21	naver	JPCko-ja	2015/08/28 17:07:10	816	0.943814	0.942805	0.943584	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	SMT PB + NMT reranking
22	naver	JPCko-ja	2015/08/11 17:38:59	630	0.943928	0.942800	0.943376	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	combined two phrase-based systems, post-processing
23	TOSHIBA	JPCko-ja	2015/08/03 09:37:49	568	0.943794	0.941287	0.943181	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Phrase-based SMT with RNNLM reranking + post-processing
24	NICT	JPCko-ja	2015/07/24 14:00:06	513	0.943463	0.942519	0.942904	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	our baseline + post-processing of bracket balancing
25	EHR	JPCko-ja	2015/07/22 12:24:01	500	0.943040	0.941893	0.942771	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	System combination of RBMT with user dictionary plus SPE and phrase based SMT. Candidate selection by language model score.
26	EHR	JPCko-ja	2017/07/19 21:58:48	1418	0.943179	0.941695	0.942757	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	Simple NMT (word based, by OpenNMT)
27	NICT	JPCko-ja	2015/07/23 14:13:38	501	0.943348	0.942402	0.942748	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	our baseline: character-based / PB SMT in MOSES (DL=0, max-phrase-len=9, no lex-reordering) / SRILM (9-gram)
28	EHR	JPCko-ja	2015/09/02 16:39:46	865	0.942999	0.941600	0.942680	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Phrase based SMT
29	goku20	JPCko-ja	2020/09/21 12:11:26	4091	0.943029	0.941926	0.942480	-	-	-	-	-	-	-	NMT	No	mBART pre-training transformer, single model
30	JAPIO	JPCko-ja	2016/08/19 17:10:56	1209	0.942137	0.940544	0.941746	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	Phrase-based SMT + JPC/JAPIO corpora + rule-based posteditor
31	Sense	JPCko-ja	2015/08/31 05:00:35	833	0.942415	0.940887	0.941687	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Baseline with only train.ja/ko
32	EHR	JPCko-ja	2015/08/30 15:52:07	831	0.941887	0.940341	0.941517	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	RBMT with user dictionary plus SPE
33	TOSHIBA	JPCko-ja	2015/07/31 19:15:00	554	0.942183	0.939545	0.941471	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	System combination SMT, SPE and RBMT with RNNLM + post-processing
34	ORGANIZER	JPCko-ja	2015/06/10 18:44:08	438	0.941302	0.939729	0.940756	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Phrase-based SMT
35	EHR	JPCko-ja	2016/07/17 15:54:31	1004	0.941071	0.940169	0.940682	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	Comb. of PBSMT by word and character. DL=0. Parens in Ko side and not in Ja side are added to Ja training and dev sets. After decoding, parens surrounding number are deleted.
36	transictkt	JPCko-ja	2016/08/21 19:13:09	1267	0.941245	0.937630	0.940497	-	-	-	-	0.000000	0.000000	0.000000	SMT	No	Phrase-based SMT.
37	JAPIO	JPCko-ja	2016/10/28 13:37:08	1331	0.940367	0.939057	0.939990	-	-	-	-	0.000000	0.000000	0.000000	SMT	No	Phrase-based SMT + rule-based posteditor
38	TOSHIBA	JPCko-ja	2015/07/31 19:11:04	552	0.939753	0.937210	0.939180	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	Phrase-based SPE with RNNLM reranking + post-processing
39	tpt_wat	JPCko-ja	2021/04/27 02:01:36	5699	0.939024	0.937699	0.938299	-	-	-	-	-	-	-	NMT	No	Base Transformer model with joint vocab, size 8k
40	JAPIO	JPCko-ja	2016/08/19 15:34:39	1206	0.938474	0.937066	0.938230	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	Phrase-based SMT + JAPIO corpus + rule-based posteditor
41	ORGANIZER	JPCko-ja	2015/06/10 18:46:34	439	0.937162	0.935903	0.936570	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Hierarchical Phrase-based SMT (2015)
42	ORGANIZER	JPCko-ja	2016/07/13 01:35:37	965	0.937162	0.935903	0.936570	-	-	-	-	0.000000	0.000000	0.000000	SMT	No	Hierarchical Phrase-based SMT (2015)
43	EHR	JPCko-ja	2017/07/19 22:00:25	1419	0.936440	0.935621	0.936136	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	Simple NMT (character based, by OpenNMT)
44	Sense	JPCko-ja	2015/08/19 21:23:17	723	0.932803	0.931268	0.930872	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Bytes2String
45	EHR	JPCko-ja	2016/07/17 18:25:46	1006	0.927065	0.927215	0.927017	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	Combination of word-based PBSMT and character-based PBSMT with DL=0. Parentheses in Korean side and not in Japanese side are added to Japanese for training and dev sets.
46	TOSHIBA	JPCko-ja	2015/08/17 11:55:22	668	0.916776	0.913982	0.915122	-	-	-	0.000000	0.000000	0.000000	0.000000	RBMT	Yes	RBMT
47	ORGANIZER	JPCko-ja	2015/08/14 17:30:21	652	0.909152	0.909385	0.908838	-	-	-	0.000000	0.000000	0.000000	0.000000	Other	Yes	Online A (2015)
48	ORGANIZER	JPCko-ja	2016/07/13 00:54:10	963	0.909152	0.909385	0.908838	-	-	-	-	0.000000	0.000000	0.000000	Other	Yes	Online A (2015)
49	ORGANIZER	JPCko-ja	2016/07/26 11:16:02	1039	0.907320	0.907652	0.906743	-	-	-	-	0.000000	0.000000	0.000000	Other	Yes	Online A (2016)
50	ORGANIZER	JPCko-ja	2015/08/14 17:37:57	653	0.876396	0.873734	0.875146	-	-	-	0.000000	0.000000	0.000000	0.000000	Other	Yes	RBMT A (2015)
51	ORGANIZER	JPCko-ja	2016/07/13 01:34:56	964	0.876396	0.873734	0.875146	-	-	-	-	0.000000	0.000000	0.000000	Other	Yes	RBMT A (2015)
52	ORGANIZER	JPCko-ja	2016/11/25 11:27:15	1344	0.857642	0.854158	0.857083	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	Online A (2016/11/25)
53	ORGANIZER	JPCko-ja	2015/08/14 17:17:31	651	0.851745	0.852263	0.851945	-	-	-	0.000000	0.000000	0.000000	0.000000	Other	Yes	Online B (2015)
54	Sense	JPCko-ja	2015/08/19 21:26:28	724	0.851041	0.854298	0.851636	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Unicode2String
55	ORGANIZER	JPCko-ja	2015/08/14 17:40:11	654	0.845712	0.849014	0.846228	-	-	-	0.000000	0.000000	0.000000	0.000000	Other	Yes	RBMT B

Notice:

This table is sorted by the leftmost segmenters. You can change the segmenter used to sort by clicking each segmenter link.

AMFM

#	Team	Task	Date/Time	DataID	AMFM										Method	Other Resources	System Description
#	Team	Task	Date/Time	DataID	unuse	unuse	unuse	unuse	unuse	unuse	unuse	unuse	unuse	unuse	Method	Other Resources	System Description
1	Bering Lab	JPCko-ja	2021/04/25 02:59:58	5513	0.941931	0.941931	0.941931	-	-	-	-	-	-	-	NMT	Yes	Transformer Ensemble with additional crawled parallel corpus
2	tpt_wat	JPCko-ja	2021/04/27 02:01:36	5699	0.927590	0.927590	0.927590	-	-	-	-	-	-	-	NMT	No	Base Transformer model with joint vocab, size 8k
3	Sense	JPCko-ja	2015/08/19 02:23:01	719	0.919380	0.919380	0.919380	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	에이트->エイト
4	Sense	JPCko-ja	2015/08/15 15:36:34	657	0.919210	0.919210	0.919210	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Unicode2String with devtest MERT run2
5	Sense	JPCko-ja	2015/08/14 10:07:01	643	0.915960	0.915960	0.915960	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Unicode2Stringwith devtest. MERT run1
6	JAPIO	JPCko-ja	2017/07/25 17:18:17	1450	0.873200	0.873200	0.873200	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	Combination of word- and character-based PBSMTs
7	JAPIO	JPCko-ja	2017/07/25 17:12:46	1448	0.872510	0.872510	0.872510	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	Word-based PBSMT + JPC/Japio corpora + rule-based posteditor
8	EHR	JPCko-ja	2017/07/19 21:57:18	1417	0.871110	0.871110	0.871110	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	SMT reranked NMT (character based, by Moses and OpenNMT)
9	EHR	JPCko-ja	2018/08/31 19:01:15	2215	0.869540	0.869540	0.869540	-	-	-	-	-	0.000000	0.000000	NMT	No	SMT reranked NMT
10	ORGANIZER	JPCko-ja	2018/08/15 18:44:10	1966	0.868760	0.868760	0.868760	-	-	-	-	-	0.000000	0.000000	NMT	No	NMT with Attention
11	JAPIO	JPCko-ja	2017/07/25 17:15:54	1449	0.868170	0.868170	0.868170	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	Character-based PBSMT + JPC/Japio corpora + rule-based posteditor
12	EHR	JPCko-ja	2016/07/17 18:23:55	1005	0.866370	0.866370	0.866370	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	Combination of word-based PBSMT and character-based PBSMT with DL=0. Parentheses surrounding number in Korean sentences are deleted.
13	TOSHIBA	JPCko-ja	2015/08/03 09:37:49	568	0.866130	0.866130	0.866130	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Phrase-based SMT with RNNLM reranking + post-processing
14	EHR	JPCko-ja	2017/07/19 21:55:39	1416	0.866060	0.866060	0.866060	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	SMT reranked NMT (word based, by Moses and OpenNMT)
15	naver	JPCko-ja	2015/08/11 17:38:59	630	0.865550	0.865550	0.865550	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	combined two phrase-based systems, post-processing
16	naver	JPCko-ja	2015/08/28 17:07:10	816	0.864450	0.864450	0.864450	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	SMT PB + NMT reranking
17	NICT	JPCko-ja	2015/07/24 14:00:06	513	0.864350	0.864350	0.864350	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	our baseline + post-processing of bracket balancing
18	JAPIO	JPCko-ja	2016/08/19 17:10:56	1209	0.863660	0.863660	0.863660	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	Phrase-based SMT + JPC/JAPIO corpora + rule-based posteditor
19	NICT	JPCko-ja	2015/07/23 14:13:38	501	0.863540	0.863540	0.863540	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	our baseline: character-based / PB SMT in MOSES (DL=0, max-phrase-len=9, no lex-reordering) / SRILM (9-gram)
20	EHR	JPCko-ja	2017/07/19 21:58:48	1418	0.862340	0.862340	0.862340	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	Simple NMT (word based, by OpenNMT)
21	TOSHIBA	JPCko-ja	2015/07/31 19:15:00	554	0.861300	0.861300	0.861300	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	System combination SMT, SPE and RBMT with RNNLM + post-processing
22	Sense	JPCko-ja	2015/08/31 05:00:35	833	0.860740	0.860740	0.860740	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Baseline with only train.ja/ko
23	EHR	JPCko-ja	2017/07/19 22:00:25	1419	0.860520	0.860520	0.860520	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	Simple NMT (character based, by OpenNMT)
24	JAPIO	JPCko-ja	2016/10/28 13:37:08	1331	0.859790	0.859790	0.859790	-	-	-	-	0.000000	0.000000	0.000000	SMT	No	Phrase-based SMT + rule-based posteditor
25	TOSHIBA	JPCko-ja	2015/07/31 19:11:04	552	0.859230	0.859230	0.859230	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	Phrase-based SPE with RNNLM reranking + post-processing
26	JAPIO	JPCko-ja	2016/08/19 15:34:39	1206	0.858190	0.858190	0.858190	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	Phrase-based SMT + JAPIO corpus + rule-based posteditor
27	transictkt	JPCko-ja	2016/08/21 19:13:09	1267	0.856830	0.856830	0.856830	-	-	-	-	0.000000	0.000000	0.000000	SMT	No	Phrase-based SMT.
28	ORGANIZER	JPCko-ja	2015/06/10 18:44:08	438	0.856220	0.856220	0.856220	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Phrase-based SMT
29	EHR	JPCko-ja	2016/07/17 15:54:31	1004	0.851660	0.851660	0.851660	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	Comb. of PBSMT by word and character. DL=0. Parens in Ko side and not in Ja side are added to Ja training and dev sets. After decoding, parens surrounding number are deleted.
30	ORGANIZER	JPCko-ja	2016/07/13 01:35:37	965	0.850610	0.850610	0.850610	-	-	-	-	0.000000	0.000000	0.000000	SMT	No	Hierarchical Phrase-based SMT (2015)
31	ORGANIZER	JPCko-ja	2015/06/10 18:46:34	439	0.850560	0.850560	0.850560	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Hierarchical Phrase-based SMT (2015)
32	EHR	JPCko-ja	2015/07/22 12:24:01	500	0.829650	0.829650	0.829650	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	System combination of RBMT with user dictionary plus SPE and phrase based SMT. Candidate selection by language model score.
33	EHR	JPCko-ja	2015/09/02 16:39:46	865	0.824750	0.824750	0.824750	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Phrase based SMT
34	EHR	JPCko-ja	2015/08/30 15:52:07	831	0.822420	0.822420	0.822420	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	RBMT with user dictionary plus SPE
35	TOSHIBA	JPCko-ja	2015/08/17 11:55:22	668	0.819770	0.819770	0.819770	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	RBMT	Yes	RBMT
36	EHR	JPCko-ja	2016/07/17 18:25:46	1006	0.818030	0.818030	0.818030	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	Combination of word-based PBSMT and character-based PBSMT with DL=0. Parentheses in Korean side and not in Japanese side are added to Japanese for training and dev sets.
37	Sense	JPCko-ja	2015/08/19 21:26:28	724	0.815990	0.815990	0.815990	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Unicode2String
38	ORGANIZER	JPCko-ja	2016/07/13 00:54:10	963	0.800610	0.800610	0.800610	-	-	-	-	0.000000	0.000000	0.000000	Other	Yes	Online A (2015)
39	ORGANIZER	JPCko-ja	2015/08/14 17:30:21	652	0.800460	0.800460	0.800460	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	Other	Yes	Online A (2015)
40	ORGANIZER	JPCko-ja	2016/07/26 11:16:02	1039	0.798750	0.798750	0.798750	-	-	-	-	0.000000	0.000000	0.000000	Other	Yes	Online A (2016)
41	ORGANIZER	JPCko-ja	2016/11/25 11:27:15	1344	0.783850	0.783850	0.783850	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	Online A (2016/11/25)
42	Sense	JPCko-ja	2015/08/19 21:23:17	723	0.778860	0.778860	0.778860	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Bytes2String
43	ORGANIZER	JPCko-ja	2015/08/14 17:17:31	651	0.728750	0.728750	0.728750	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	Other	Yes	Online B (2015)
44	ORGANIZER	JPCko-ja	2016/07/13 01:34:56	964	0.712700	0.712700	0.712700	-	-	-	-	0.000000	0.000000	0.000000	Other	Yes	RBMT A (2015)
45	ORGANIZER	JPCko-ja	2015/08/14 17:37:57	653	0.712020	0.712020	0.712020	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	Other	Yes	RBMT A (2015)
46	ORGANIZER	JPCko-ja	2015/08/14 17:40:11	654	0.643150	0.643150	0.643150	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	Other	Yes	RBMT B
47	ryan	JPCko-ja	2019/07/23 11:57:16	2838	0.000000	0.000000	0.000000	-	-	-	-	-	-	-	NMT	No	Base Transformer
48	sarah	JPCko-ja	2019/07/26 16:27:49	3016	0.000000	0.000000	0.000000	-	-	-	-	-	-	-	NMT	No	Transformer, ensemble of 4 models
49	KNU_Hyundai	JPCko-ja	2019/07/26 23:59:03	3093	0.000000	0.000000	0.000000	-	-	-	-	-	-	-	NMT	No	Transformer + Relative Position + ensemble of 11 models
50	TMU	JPCko-ja	2020/09/15 07:33:10	3721	0.000000	0.000000	0.000000	-	-	-	-	-	-	-	NMT	No	Transformer, domain adaptation (BERT Japanese), hanja loss, shared EMB, shared BPE, ensemble of 4 models
51	TMU	JPCko-ja	2020/09/15 20:51:18	3757	0.000000	0.000000	0.000000	-	-	-	-	-	-	-	NMT	No	Transformer, domain adaptation (BERT Japanese), shared EMB, shared BPE, ensemble of 4 models
52	goku20	JPCko-ja	2020/09/21 12:11:26	4091	0.000000	0.000000	0.000000	-	-	-	-	-	-	-	NMT	No	mBART pre-training transformer, single model
53	goku20	JPCko-ja	2020/09/22 10:17:13	4120	0.000000	0.000000	0.000000	-	-	-	-	-	-	-	NMT	No	mBART pre-training transformer, ensemble of 3 models
54	TMU	JPCko-ja	2020/10/13 09:43:00	4138	0.000000	0.000000	0.000000	-	-	-	-	-	-	-	NMT	No	Transformer, domain adaptation (BERT Korean), hanja loss, shared EMB, shared BPE, ensemble of 4 models
55	TMU	JPCko-ja	2020/10/13 10:00:00	4142	0.000000	0.000000	0.000000	-	-	-	-	-	-	-	NMT	No	Transformer, domain adaptation (BERT Korean), shared EMB, shared BPE, ensemble of 4 models

Notice:

This table is sorted by the leftmost segmenters. You can change the segmenter used to sort by clicking each segmenter link.
Adequacy-Fluency Metrics (AMFM) is a two-dimensional automatic evaluation metric for machine translation, designed to operate at the sentence level. It is based on adequacy and fluency, to decouple semantic and syntactic components of the translation process to provide a balanced view on translation quality.
AMFM is calculated without tokenizers.
The detail of AMFM is shown on the following paper: "Adequacy–Fluency Metrics: Evaluating MT in the Continuous Space Model Framework" [pdf]. Invited Talk in WAT2015 also helps understanding [slide].

HUMAN (WAT2022)

Notice:

HUMAN (WAT2022) is the result of the Pairwise Crowdsourcing Evaluation on WAT2022.
HUMAN (WAT2022) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.

HUMAN (WAT2021)

Notice:

HUMAN (WAT2021) is the result of the Pairwise Crowdsourcing Evaluation on WAT2021.
HUMAN (WAT2021) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.

HUMAN (WAT2020)

Notice:

HUMAN (WAT2020) is the result of the Pairwise Crowdsourcing Evaluation on WAT2020.
HUMAN (WAT2020) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.

HUMAN (WAT2019)

Notice:

HUMAN (WAT2019) is the result of the Pairwise Crowdsourcing Evaluation on WAT2019.
HUMAN (WAT2019) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.

HUMAN (WAT2018)

Notice:

HUMAN (WAT2018) is the result of the Pairwise Crowdsourcing Evaluation on WAT2018.
HUMAN (WAT2018) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.

HUMAN (WAT2017)

Notice:

HUMAN (WAT2017) is the result of the Pairwise Crowdsourcing Evaluation on WAT2017.
HUMAN (WAT2017) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.

HUMAN (WAT2016)

#	Team	Task	Date/Time	DataID	HUMAN	Method	Other Resources	System Description
1	EHR	JPCko-ja	2016/07/17 18:25:46	1006	21.750	SMT	Yes	Combination of word-based PBSMT and character-based PBSMT with DL=0. Parentheses in Korean side and not in Japanese side are added to Japanese for training and dev sets.
2	JAPIO	JPCko-ja	2016/08/19 17:10:56	1209	17.500	SMT	Yes	Phrase-based SMT + JPC/JAPIO corpora + rule-based posteditor
3	ORGANIZER	JPCko-ja	2016/07/26 11:16:02	1039	8.000	Other	Yes	Online A (2016)
4	EHR	JPCko-ja	2016/07/17 18:23:55	1005	-3.000	SMT	Yes	Combination of word-based PBSMT and character-based PBSMT with DL=0. Parentheses surrounding number in Korean sentences are deleted.
5	JAPIO	JPCko-ja	2016/08/19 15:34:39	1206	-9.000	SMT	Yes	Phrase-based SMT + JAPIO corpus + rule-based posteditor
6	ORGANIZER	JPCko-ja	2016/11/25 11:27:15	1344	-55.750	NMT	No	Online A (2016/11/25)

Notice:

HUMAN (WAT2016) is the result of the Pairwise Crowdsourcing Evaluation on WAT2016.
HUMAN (WAT2016) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.

HUMAN (WAT2015)

#	Team	Task	Date/Time	DataID	HUMAN	Method	Other Resources	System Description
1	ORGANIZER	JPCko-ja	2015/08/14 17:30:21	652	38.750	Other	Yes	Online A (2015)
2	naver	JPCko-ja	2015/08/28 17:07:10	816	14.750	SMT	No	SMT PB + NMT reranking
3	NICT	JPCko-ja	2015/07/24 14:00:06	513	10.500	SMT	No	our baseline + post-processing of bracket balancing
4	EHR	JPCko-ja	2015/07/22 12:24:01	500	10.250	SMT and RBMT	Yes	System combination of RBMT with user dictionary plus SPE and phrase based SMT. Candidate selection by language model score.
5	NICT	JPCko-ja	2015/07/23 14:13:38	501	8.250	SMT	No	our baseline: character-based / PB SMT in MOSES (DL=0, max-phrase-len=9, no lex-reordering) / SRILM (9-gram)
6	naver	JPCko-ja	2015/08/11 17:38:59	630	6.750	SMT	No	combined two phrase-based systems, post-processing
7	EHR	JPCko-ja	2015/08/30 15:52:07	831	6.500	SMT and RBMT	Yes	RBMT with user dictionary plus SPE
8	TOSHIBA	JPCko-ja	2015/08/03 09:37:49	568	4.500	SMT	Yes	Phrase-based SMT with RNNLM reranking + post-processing
9	Sense	JPCko-ja	2015/08/31 05:00:35	833	3.500	SMT	No	Baseline with only train.ja/ko
10	TOSHIBA	JPCko-ja	2015/07/31 19:15:00	554	3.000	SMT and RBMT	Yes	System combination SMT, SPE and RBMT with RNNLM + post-processing
11	ORGANIZER	JPCko-ja	2015/06/10 18:46:34	439	2.750	SMT	No	Hierarchical Phrase-based SMT (2015)
12	ORGANIZER	JPCko-ja	2015/08/14 17:37:57	653	-7.250	Other	Yes	RBMT A (2015)
13	Sense	JPCko-ja	2015/08/15 15:36:34	657	-17.750	SMT	No	Unicode2String with devtest MERT run2

Notice:

HUMAN (WAT2015) is the result of the Pairwise Crowdsourcing Evaluation on WAT2015.
HUMAN (WAT2015) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.
The detail of the evaluation can be found in the pdf document (PDF file).

HUMAN (WAT2014)

Notice:

HUMAN (WAT2014) is the result of the Pairwise Crowdsourcing Evaluation on WAT2014.
HUMAN (WAT2014) was evaluated by 3 different workers and the final decision is made by the voting of the judgements.
The detail of the evaluation can be found in the pdf document (PDF file).

EVALUATION RESULTS USAGE POLICY

When you use the WAT evaluation results for any purpose such as:
- writing technical papers,
- making presentations about your system,
- advertising your MT system to the customers,
you can use the information about translation directions, scores (including both automatic and human evaluations) and ranks of your system among others. You can also use the scores of the other systems, but you MUST anonymize the other system's names. In addition, you can show the links (URLs) to the WAT evaluation result pages.

NICT (National Institute of Information and Communications Technology)
Kyoto University
Last Modified: 2018-08-02

WAT The Workshop on Asian Translation Evaluation Results

BLEU

RIBES

AMFM

HUMAN (WAT2022)

HUMAN (WAT2021)

HUMAN (WAT2020)

HUMAN (WAT2019)

HUMAN (WAT2018)

HUMAN (WAT2017)

HUMAN (WAT2016)

HUMAN (WAT2015)

HUMAN (WAT2014)

EVALUATION RESULTS USAGE POLICY

WAT

The Workshop on Asian Translation

Evaluation Results