WAT

The Workshop on Asian Translation

Evaluation Results

BLEU

#	Team	Task	Date/Time	DataID	BLEU										Method	Other Resources	System Description
#	Team	Task	Date/Time	DataID	juman	kytea	mecab	moses- tokenizer	stanford- segmenter- ctb	stanford- segmenter- pku	indic- tokenizer	unuse	myseg	kmseg	Method	Other Resources	System Description
1	ORGANIZER	zh-ja	2018/08/14 11:33:03	1902	43.31	43.53	43.34	-	-	-	-	-	0.00	0.00	NMT	No	NMT with Attention
2	NICT-5	zh-ja	2018/08/22 18:51:44	2052	48.43	48.78	48.52	-	-	-	-	-	0.00	0.00	NMT	No	Mixed fine tuning by first pretraining on En-Ja ASPEC data and then continue on the En-Ja+Zh-Ja data. Transformer.
3	NICT-5	zh-ja	2018/08/27 14:40:35	2169	49.67	50.46	49.79	-	-	-	-	-	0.00	0.00	NMT	No	Combining En-Ja corpus with Zh-Ja as a multilingual model. ADDITIONAL ASPEC CORPUS USED
4	NICT-5	zh-ja	2018/09/10 14:14:05	2267	49.79	50.66	49.89	-	-	-	-	-	0.00	0.00	NMT	No	MLNMT
5	TMU	zh-ja	2018/09/14 17:30:33	2343	6.21	7.02	6.27	-	-	-	-	-	0.00	0.00	NMT	Yes	Unsupervised NMT with sub-character information. Both ASPEC and JPC 4.0 data (zh-ja) were also used as monolingual data in the training.
6	srcb	zh-ja	2019/07/25 11:37:44	2917	49.83	50.89	50.00	-	-	-	-	-	-	-	NMT	No	Transformer (Big) with relative position, layer attention, sentence-wise smooth.
7	KNU_Hyundai	zh-ja	2019/07/27 10:30:04	3179	50.02	50.84	50.23	-	-	-	-	-	-	-	NMT	No	Transformer(base) + Used ASPEC ja-en corpus with relative position, bt, multi source, r2l rerank, 6-model ensemble
8	srcb	zh-ja	2019/07/27 15:48:24	3210	52.37	53.58	52.57	-	-	-	-	-	-	-	NMT	No	Transformer(Big) with relative position, sentence-wise smooth, deep transformer, back translation, ensemble of 7 models.
9	Kyoto-U+ECNU	zh-ja	2020/09/10 23:58:13	3677	50.37	51.27	50.60	-	-	-	-	-	-	-	NMT	No	back-translation by using ja monolingual data from ASPEC-JE; lightconv (pay less attention) single model without ensemble
10	Kyoto-U+ECNU	zh-ja	2020/09/17 18:41:34	3813	52.80	53.64	52.92	-	-	-	-	-	-	-	NMT	Yes	ensemble 9 models: structures(LSTM, Transformer, ConvS2S, Lightconv), training data(BT, out-of-domain parallel), S2S settings(deeper transformer, deep encoder shallow decoder)
11	Kyoto-U+ECNU	zh-ja	2020/09/18 17:38:55	3933	52.65	53.48	52.80	-	-	-	-	-	-	-	NMT	No	without out-of-domain parallel data; others same as DataID:3813
12	ORGANIZER	zh-ja	2014/07/11 19:47:27	4	35.43	35.91	35.64	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Hierarchical Phrase-based SMT (2014)
13	ORGANIZER	zh-ja	2014/07/11 19:54:58	8	34.65	35.16	34.77	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Phrase-based SMT
14	ORGANIZER	zh-ja	2014/07/11 20:04:10	13	36.52	37.07	36.64	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Tree-to-String SMT (2014)
15	ORGANIZER	zh-ja	2014/07/18 11:09:12	36	11.63	13.21	11.87	-	-	-	0.00	0.00	0.00	0.00	Other	Yes	Online A (2014)
16	NAIST	zh-ja	2014/07/31 11:42:31	120	40.11	41.29	40.30	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Travatar-based Forest-to-String SMT System
17	NAIST	zh-ja	2014/08/01 17:33:01	124	40.21	40.82	40.15	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Travatar-based Forest-to-String SMT System (Tuned BLEU+RIBES)
18	Kyoto-U	zh-ja	2014/08/19 09:31:08	133	33.26	35.09	33.62	-	-	-	0.00	0.00	0.00	0.00	EBMT	No	Using n-best parses and RNNLM.
19	Kyoto-U	zh-ja	2014/08/19 10:21:37	135	32.68	33.30	32.45	-	-	-	0.00	0.00	0.00	0.00	EBMT	No	Our baseline system.
20	EIWA	zh-ja	2014/08/20 11:52:45	137	18.69	18.33	18.32	-	-	-	0.00	0.00	0.00	0.00	RBMT	Yes	RBMT plus user dictionary
21	EIWA	zh-ja	2014/08/20 11:56:00	138	33.53	33.74	33.87	-	-	-	0.00	0.00	0.00	0.00	SMT and RBMT	Yes	RBMT with user dictionary plus SPE(statistical post editing)
22	Sense	zh-ja	2014/08/26 15:17:49	200	33.66	33.86	33.46	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Character based SMT
23	ORGANIZER	zh-ja	2014/08/28 12:10:13	215	10.48	11.26	10.47	-	-	-	0.00	0.00	0.00	0.00	Other	Yes	Online B (2014)
24	SAS_MT	zh-ja	2014/08/29 15:33:07	232	36.58	36.22	36.10	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Syntactic reordering phrase-based SMT （SAS token tool）
25	ORGANIZER	zh-ja	2014/08/29 18:45:03	239	9.37	9.87	9.35	-	-	-	0.00	0.00	0.00	0.00	RBMT	No	RBMT A (2014)
26	ORGANIZER	zh-ja	2014/08/29 18:48:29	242	8.39	8.70	8.30	-	-	-	0.00	0.00	0.00	0.00	RBMT	No	RBMT D
27	Kyoto-U	zh-ja	2014/08/31 23:42:41	258	33.57	34.43	33.45	-	-	-	0.00	0.00	0.00	0.00	EBMT	No	Our new baseline system after several modifications.
28	SAS_MT	zh-ja	2014/09/01 10:38:13	263	37.42	37.65	37.07	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Syntactic reordering Hierarchical SMT （using SAS token tool）
29	Kyoto-U	zh-ja	2014/09/01 21:33:23	268	34.75	35.89	34.83	-	-	-	0.00	0.00	0.00	0.00	EBMT	No	Our new baseline system after several modifications + 20-best parses, KN7, RNNLM reranking
30	WASUIPS	zh-ja	2014/09/17 00:43:38	369	27.66	28.09	28.20	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Our baseline system (segmentation tools: urheen and mecab, moses: 1.0).
31	WASUIPS	zh-ja	2014/09/17 00:46:07	370	30.44	30.92	30.86	-	-	-	0.00	0.00	0.00	0.00	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: urheen and mecab, moses: 1.0).
32	WASUIPS	zh-ja	2014/09/17 01:03:57	374	31.87	32.26	32.26	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Our baseline system (segmentation tools: urheen and mecab, moses: 2.1.1).
33	WASUIPS	zh-ja	2014/09/17 01:05:38	375	32.19	32.55	32.54	-	-	-	0.00	0.00	0.00	0.00	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: urheen and mecab, moses: 2.1.1).
34	WASUIPS	zh-ja	2014/09/17 10:07:44	379	27.37	28.28	27.43	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Our baseline system (segmentation tools: kytea, moses: 1.0).
35	WASUIPS	zh-ja	2014/09/17 10:10:47	380	27.86	28.89	28.00	-	-	-	0.00	0.00	0.00	0.00	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: kytea, moses: 1.0).
36	WASUIPS	zh-ja	2014/09/17 10:24:50	383	32.08	33.09	32.18	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Our baseline system (segmentation tools: kytea, moses: 2.1.1).
37	WASUIPS	zh-ja	2014/09/17 10:26:43	384	32.43	33.36	32.48	-	-	-	0.00	0.00	0.00	0.00	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: kytea, moses: 2.1.1).
38	WASUIPS	zh-ja	2014/09/17 11:03:46	387	32.52	32.69	32.47	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Our baseline system (segmentation tools: stanford-ctb and juman, moses: 2.1.1).
39	WASUIPS	zh-ja	2014/09/17 12:00:46	388	32.65	32.81	32.59	-	-	-	0.00	0.00	0.00	0.00	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: stanford-ctb and juman, moses: 2.1.1).
40	Kyoto-U	zh-ja	2015/07/17 09:01:42	490	35.66	36.71	35.81	-	-	-	0.00	0.00	0.00	0.00	EBMT	No	WAT2015 baseline
41	Kyoto-U	zh-ja	2015/07/17 09:04:22	491	36.76	37.82	36.94	-	-	-	0.00	0.00	0.00	0.00	EBMT	No	WAT2015 baseline with reranking
42	TOSHIBA	zh-ja	2015/07/23 15:14:53	508	37.47	37.44	37.34	-	-	-	0.00	0.00	0.00	0.00	SMT and RBMT	Yes	System combination SMT and RBMT(SPE) with RNNLM language model
43	TOSHIBA	zh-ja	2015/07/28 16:27:32	525	35.85	36.02	35.73	-	-	-	0.00	0.00	0.00	0.00	SMT and RBMT	Yes	RBMT with SPE(Statistical Post Editing) system
44	Sense	zh-ja	2015/07/29 07:20:20	533	29.29	30.52	29.45	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Baseline-2015
45	Kyoto-U	zh-ja	2015/08/07 13:24:55	597	37.30	38.26	37.45	-	-	-	0.00	0.00	0.00	0.00	EBMT	No	Updated JUMAN and added one reordering feature, w/ reranking
46	TOSHIBA	zh-ja	2015/08/17 12:11:52	669	19.24	19.48	19.12	-	-	-	0.00	0.00	0.00	0.00	RBMT	Yes	RBMT
47	EHR	zh-ja	2015/08/19 11:23:36	720	37.90	38.68	37.98	-	-	-	0.00	0.00	0.00	0.00	SMT and RBMT	Yes	System combination of RBMT with user dictionary plus SPE and phrase based SMT with preordering. Candidate selection by language model score.
48	BJTUNLP	zh-ja	2015/08/25 14:55:20	769	34.72	34.87	34.79	-	-	-	0.00	0.00	0.00	0.00	SMT	No
49	ORGANIZER	zh-ja	2015/08/25 18:58:08	776	11.53	12.82	11.68	-	-	-	0.00	0.00	0.00	0.00	Other	Yes	Online A (2015)
50	NAIST	zh-ja	2015/08/31 08:23:30	834	41.75	42.95	41.93	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Travatar System with NeuralMT Reranking
51	NAIST	zh-ja	2015/08/31 08:26:31	835	39.36	40.51	39.47	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Travatar System Baseline
52	Kyoto-U	zh-ja	2015/08/31 22:38:22	844	36.30	37.22	36.44	-	-	-	0.00	0.00	0.00	0.00	EBMT	No	KyotoEBMT system without reranking
53	Kyoto-U	zh-ja	2015/08/31 22:39:36	845	38.53	39.41	38.66	-	-	-	0.00	0.00	0.00	0.00	EBMT	No	KyotoEBMT system with bilingual RNNLM reranking
54	BJTUNLP	zh-ja	2015/09/01 21:08:10	862	34.72	34.87	34.79	-	-	-	0.00	0.00	0.00	0.00	SMT	No	a dependency-to-string model for SMT
55	EHR	zh-ja	2015/09/02 17:00:16	867	39.43	39.98	39.58	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Phrase based SMT with preordering.
56	EHR	zh-ja	2015/09/04 11:44:26	868	35.59	35.56	35.37	-	-	-	0.00	0.00	0.00	0.00	SMT and RBMT	Yes	RBMT with user dictionary plus SPE.
57	ORGANIZER	zh-ja	2015/09/10 14:00:33	879	36.52	37.07	36.64	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Tree-to-String SMT (2015)
58	ORGANIZER	zh-ja	2015/09/10 14:30:56	885	9.37	9.87	9.35	-	-	-	0.00	0.00	0.00	0.00	Other	Yes	RBMT A (2015)
59	ORGANIZER	zh-ja	2015/09/11 10:09:23	890	10.41	11.03	10.36	-	-	-	0.00	0.00	0.00	0.00	Other	Yes	Online B (2015)
60	ORGANIZER	zh-ja	2016/07/26 11:54:14	1043	11.56	12.87	11.69	-	-	-	-	0.00	0.00	0.00	Other	Yes	Online A (2016)
61	EHR	zh-ja	2016/07/31 17:06:57	1063	39.75	39.85	39.40	-	-	-	-	0.00	0.00	0.00	SMT	Yes	LM-based merging of outputs of preordered word-based PBSMT(DL=6) and preordered character-based PBSMT(DL=6).
62	NICT-2	zh-ja	2016/08/05 18:05:03	1099	40.02	40.45	40.29	-	-	-	-	0.00	0.00	0.00	SMT	Yes	Phrase-based SMT with Preordering + Domain Adaptation (JPC and ASPEC) + Google 5-gram LM
63	Kyoto-U	zh-ja	2016/08/07 18:28:23	1110	36.63	37.54	36.70	-	-	-	-	0.00	0.00	0.00	EBMT	No	KyotoEBMT 2016 w/o reranking
64	bjtu_nlp	zh-ja	2016/08/12 12:50:38	1138	38.83	39.25	38.68	-	-	-	-	0.00	0.00	0.00	NMT	No	RNN Encoder-Decoder with attention mechanism, single model
65	JAPIO	zh-ja	2016/08/19 16:44:49	1208	26.24	27.87	26.37	-	-	-	-	0.00	0.00	0.00	SMT	Yes	Phrase-based SMT with Preordering + JAPIO corpus + rule-based posteditor
66	UT-KAY	zh-ja	2016/08/20 07:09:54	1220	37.63	39.07	37.82	-	-	-	-	0.00	0.00	0.00	NMT	No	An end-to-end NMT with 512 dimensional single-layer LSTMs, UNK replacement, and domain adaptation
67	UT-KAY	zh-ja	2016/08/20 07:12:52	1221	40.50	41.81	40.67	-	-	-	-	0.00	0.00	0.00	NMT	No	Ensemble of our NMT models with and without domain adaptation
68	Kyoto-U	zh-ja	2016/08/20 22:48:16	1255	44.29	45.05	44.32	-	-	-	-	0.00	0.00	0.00	NMT	No	src: 200k tgt: 50k 2-layers self-ensembling
69	Kyoto-U	zh-ja	2016/08/20 22:50:33	1256	46.04	46.70	46.05	-	-	-	-	0.00	0.00	0.00	NMT	No	voc: 30k ensemble of 3 independent model + reverse rescoring
70	Kyoto-U	zh-ja	2016/10/11 10:46:03	1324	46.36	47.02	46.50	-	-	-	-	0.00	0.00	0.00	NMT	No	voc: 32k ensemble of 4 independent model + Chinese short unit
71	ORGANIZER	zh-ja	2016/11/16 11:28:00	1342	18.75	20.64	19.04	-	-	-	-	0.00	0.00	0.00	NMT	Yes	Online A (2016/11/14)
72	NICT-2	zh-ja	2017/07/26 13:58:44	1477	44.26	44.90	44.50	-	-	-	-	0.00	0.00	0.00	NMT	No	NMT Single Model: BPE50k, Bi-LSTM(500*2) Encoder, LSTM(1000) Left-to-Right Decoder
73	NICT-2	zh-ja	2017/07/26 14:08:45	1481	46.84	47.51	47.27	-	-	-	-	0.00	0.00	0.00	NMT	No	NMT 6 Ensembles * Bi-directional Reranking
74	Kyoto-U	zh-ja	2017/07/29 08:02:07	1577	48.43	48.84	48.51	-	-	-	-	0.00	0.00	0.00	NMT	No	Ensemble of 5 Shared BPE 40k
75	Kyoto-U	zh-ja	2017/07/31 15:27:21	1643	46.74	47.79	46.67	-	-	-	-	0.00	0.00	0.00	NMT	No	KW replacement without KW in the test set, BPE, 6 ensemble
76	Kyoto-U	zh-ja	2017/08/01 14:14:49	1720	48.34	48.76	48.40	-	-	-	-	0.00	0.00	0.00	NMT	No	Ensemble of 7 shared BPE, averaged
77	ORGANIZER	zh-ja	2017/08/02 09:59:33	1740	46.87	47.30	47.00	-	-	-	-	0.00	0.00	0.00	NMT	No	Google's "Attention Is All You Need"

Notice:

This table is sorted by the leftmost segmenters. You can change the segmenter used to sort by clicking each segmenter link.

RIBES

#	Team	Task	Date/Time	DataID	RIBES										Method	Other Resources	System Description
#	Team	Task	Date/Time	DataID	juman	kytea	mecab	moses- tokenizer	stanford- segmenter- ctb	stanford- segmenter- pku	indic- tokenizer	unuse	myseg	kmseg	Method	Other Resources	System Description
1	ORGANIZER	zh-ja	2018/08/14 11:33:03	1902	0.870734	0.866281	0.870886	-	-	-	-	-	0.000000	0.000000	NMT	No	NMT with Attention
2	NICT-5	zh-ja	2018/08/22 18:51:44	2052	0.884426	0.879456	0.884782	-	-	-	-	-	0.000000	0.000000	NMT	No	Mixed fine tuning by first pretraining on En-Ja ASPEC data and then continue on the En-Ja+Zh-Ja data. Transformer.
3	NICT-5	zh-ja	2018/08/27 14:40:35	2169	0.886161	0.882989	0.886367	-	-	-	-	-	0.000000	0.000000	NMT	No	Combining En-Ja corpus with Zh-Ja as a multilingual model. ADDITIONAL ASPEC CORPUS USED
4	NICT-5	zh-ja	2018/09/10 14:14:05	2267	0.889674	0.886490	0.889853	-	-	-	-	-	0.000000	0.000000	NMT	No	MLNMT
5	TMU	zh-ja	2018/09/14 17:30:33	2343	0.570040	0.562331	0.565795	-	-	-	-	-	0.000000	0.000000	NMT	Yes	Unsupervised NMT with sub-character information. Both ASPEC and JPC 4.0 data (zh-ja) were also used as monolingual data in the training.
6	srcb	zh-ja	2019/07/25 11:37:44	2917	0.886069	0.882712	0.886854	-	-	-	-	-	-	-	NMT	No	Transformer (Big) with relative position, layer attention, sentence-wise smooth.
7	KNU_Hyundai	zh-ja	2019/07/27 10:30:04	3179	0.888530	0.886248	0.888706	-	-	-	-	-	-	-	NMT	No	Transformer(base) + Used ASPEC ja-en corpus with relative position, bt, multi source, r2l rerank, 6-model ensemble
8	srcb	zh-ja	2019/07/27 15:48:24	3210	0.895231	0.891872	0.895663	-	-	-	-	-	-	-	NMT	No	Transformer(Big) with relative position, sentence-wise smooth, deep transformer, back translation, ensemble of 7 models.
9	Kyoto-U+ECNU	zh-ja	2020/09/10 23:58:13	3677	0.889269	0.885500	0.889503	-	-	-	-	-	-	-	NMT	No	back-translation by using ja monolingual data from ASPEC-JE; lightconv (pay less attention) single model without ensemble
10	Kyoto-U+ECNU	zh-ja	2020/09/17 18:41:34	3813	0.897053	0.894441	0.897199	-	-	-	-	-	-	-	NMT	Yes	ensemble 9 models: structures(LSTM, Transformer, ConvS2S, Lightconv), training data(BT, out-of-domain parallel), S2S settings(deeper transformer, deep encoder shallow decoder)
11	Kyoto-U+ECNU	zh-ja	2020/09/18 17:38:55	3933	0.896551	0.894073	0.896743	-	-	-	-	-	-	-	NMT	No	without out-of-domain parallel data; others same as DataID:3813
12	ORGANIZER	zh-ja	2014/07/11 19:47:27	4	0.810406	0.798726	0.807665	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Hierarchical Phrase-based SMT (2014)
13	ORGANIZER	zh-ja	2014/07/11 19:54:58	8	0.772498	0.766384	0.771005	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Phrase-based SMT
14	ORGANIZER	zh-ja	2014/07/11 20:04:10	13	0.825292	0.820490	0.825025	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Tree-to-String SMT (2014)
15	ORGANIZER	zh-ja	2014/07/18 11:09:12	36	0.595925	0.598172	0.598573	-	-	-	0.000000	0.000000	0.000000	0.000000	Other	Yes	Online A (2014)
16	NAIST	zh-ja	2014/07/31 11:42:31	120	0.842477	0.834824	0.842235	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Travatar-based Forest-to-String SMT System
17	NAIST	zh-ja	2014/08/01 17:33:01	124	0.845486	0.838092	0.845625	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Travatar-based Forest-to-String SMT System (Tuned BLEU+RIBES)
18	Kyoto-U	zh-ja	2014/08/19 09:31:08	133	0.791680	0.787105	0.791269	-	-	-	0.000000	0.000000	0.000000	0.000000	EBMT	No	Using n-best parses and RNNLM.
19	Kyoto-U	zh-ja	2014/08/19 10:21:37	135	0.786229	0.783016	0.786352	-	-	-	0.000000	0.000000	0.000000	0.000000	EBMT	No	Our baseline system.
20	EIWA	zh-ja	2014/08/20 11:52:45	137	0.740183	0.720281	0.732466	-	-	-	0.000000	0.000000	0.000000	0.000000	RBMT	Yes	RBMT plus user dictionary
21	EIWA	zh-ja	2014/08/20 11:56:00	138	0.811350	0.800506	0.808504	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	RBMT with user dictionary plus SPE(statistical post editing)
22	Sense	zh-ja	2014/08/26 15:17:49	200	0.789495	0.774338	0.784012	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Character based SMT
23	ORGANIZER	zh-ja	2014/08/28 12:10:13	215	0.600733	0.596006	0.600706	-	-	-	0.000000	0.000000	0.000000	0.000000	Other	Yes	Online B (2014)
24	SAS_MT	zh-ja	2014/08/29 15:33:07	232	0.822180	0.807535	0.817368	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Syntactic reordering phrase-based SMT （SAS token tool）
25	ORGANIZER	zh-ja	2014/08/29 18:45:03	239	0.666277	0.652402	0.661730	-	-	-	0.000000	0.000000	0.000000	0.000000	RBMT	No	RBMT A (2014)
26	ORGANIZER	zh-ja	2014/08/29 18:48:29	242	0.641189	0.626400	0.633319	-	-	-	0.000000	0.000000	0.000000	0.000000	RBMT	No	RBMT D
27	Kyoto-U	zh-ja	2014/08/31 23:42:41	258	0.800949	0.795390	0.800986	-	-	-	0.000000	0.000000	0.000000	0.000000	EBMT	No	Our new baseline system after several modifications.
28	SAS_MT	zh-ja	2014/09/01 10:38:13	263	0.834170	0.825551	0.833048	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Syntactic reordering Hierarchical SMT （using SAS token tool）
29	Kyoto-U	zh-ja	2014/09/01 21:33:23	268	0.802629	0.798631	0.802930	-	-	-	0.000000	0.000000	0.000000	0.000000	EBMT	No	Our new baseline system after several modifications + 20-best parses, KN7, RNNLM reranking
30	WASUIPS	zh-ja	2014/09/17 00:43:38	369	0.779183	0.762949	0.770846	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Our baseline system (segmentation tools: urheen and mecab, moses: 1.0).
31	WASUIPS	zh-ja	2014/09/17 00:46:07	370	0.789824	0.773142	0.781475	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: urheen and mecab, moses: 1.0).
32	WASUIPS	zh-ja	2014/09/17 01:03:57	374	0.794303	0.777876	0.786422	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Our baseline system (segmentation tools: urheen and mecab, moses: 2.1.1).
33	WASUIPS	zh-ja	2014/09/17 01:05:38	375	0.795838	0.780027	0.787591	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: urheen and mecab, moses: 2.1.1).
34	WASUIPS	zh-ja	2014/09/17 10:07:44	379	0.774423	0.753749	0.767073	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Our baseline system (segmentation tools: kytea, moses: 1.0).
35	WASUIPS	zh-ja	2014/09/17 10:10:47	380	0.776550	0.756721	0.769409	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: kytea, moses: 1.0).
36	WASUIPS	zh-ja	2014/09/17 10:24:50	383	0.793230	0.775168	0.787665	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Our baseline system (segmentation tools: kytea, moses: 2.1.1).
37	WASUIPS	zh-ja	2014/09/17 10:26:43	384	0.796220	0.778075	0.789657	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: kytea, moses: 2.1.1).
38	WASUIPS	zh-ja	2014/09/17 11:03:46	387	0.796059	0.780402	0.790107	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Our baseline system (segmentation tools: stanford-ctb and juman, moses: 2.1.1).
39	WASUIPS	zh-ja	2014/09/17 12:00:46	388	0.796777	0.781733	0.791219	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: stanford-ctb and juman, moses: 2.1.1).
40	Kyoto-U	zh-ja	2015/07/17 09:01:42	490	0.809395	0.803780	0.808692	-	-	-	0.000000	0.000000	0.000000	0.000000	EBMT	No	WAT2015 baseline
41	Kyoto-U	zh-ja	2015/07/17 09:04:22	491	0.818445	0.812910	0.817522	-	-	-	0.000000	0.000000	0.000000	0.000000	EBMT	No	WAT2015 baseline with reranking
42	TOSHIBA	zh-ja	2015/07/23 15:14:53	508	0.827291	0.817395	0.825472	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	System combination SMT and RBMT(SPE) with RNNLM language model
43	TOSHIBA	zh-ja	2015/07/28 16:27:32	525	0.824740	0.815388	0.822423	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	RBMT with SPE(Statistical Post Editing) system
44	Sense	zh-ja	2015/07/29 07:20:20	533	0.774692	0.764847	0.772410	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Baseline-2015
45	Kyoto-U	zh-ja	2015/08/07 13:24:55	597	0.822672	0.817037	0.822340	-	-	-	0.000000	0.000000	0.000000	0.000000	EBMT	No	Updated JUMAN and added one reordering feature, w/ reranking
46	TOSHIBA	zh-ja	2015/08/17 12:11:52	669	0.741665	0.727155	0.738298	-	-	-	0.000000	0.000000	0.000000	0.000000	RBMT	Yes	RBMT
47	EHR	zh-ja	2015/08/19 11:23:36	720	0.826003	0.818620	0.824806	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	System combination of RBMT with user dictionary plus SPE and phrase based SMT with preordering. Candidate selection by language model score.
48	BJTUNLP	zh-ja	2015/08/25 14:55:20	769	0.807012	0.792488	0.802430	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No
49	ORGANIZER	zh-ja	2015/08/25 18:58:08	776	0.588285	0.590393	0.592887	-	-	-	0.000000	0.000000	0.000000	0.000000	Other	Yes	Online A (2015)
50	NAIST	zh-ja	2015/08/31 08:23:30	834	0.855089	0.847746	0.854587	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Travatar System with NeuralMT Reranking
51	NAIST	zh-ja	2015/08/31 08:26:31	835	0.834388	0.827148	0.834130	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Travatar System Baseline
52	Kyoto-U	zh-ja	2015/08/31 22:38:22	844	0.819743	0.814581	0.818794	-	-	-	0.000000	0.000000	0.000000	0.000000	EBMT	No	KyotoEBMT system without reranking
53	Kyoto-U	zh-ja	2015/08/31 22:39:36	845	0.840681	0.834451	0.839063	-	-	-	0.000000	0.000000	0.000000	0.000000	EBMT	No	KyotoEBMT system with bilingual RNNLM reranking
54	BJTUNLP	zh-ja	2015/09/01 21:08:10	862	0.807012	0.792488	0.802430	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	a dependency-to-string model for SMT
55	EHR	zh-ja	2015/09/02 17:00:16	867	0.837678	0.831682	0.837227	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Phrase based SMT with preordering.
56	EHR	zh-ja	2015/09/04 11:44:26	868	0.815842	0.806726	0.813996	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	RBMT with user dictionary plus SPE.
57	ORGANIZER	zh-ja	2015/09/10 14:00:33	879	0.825292	0.820490	0.825025	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Tree-to-String SMT (2015)
58	ORGANIZER	zh-ja	2015/09/10 14:30:56	885	0.666277	0.652402	0.661730	-	-	-	0.000000	0.000000	0.000000	0.000000	Other	Yes	RBMT A (2015)
59	ORGANIZER	zh-ja	2015/09/11 10:09:23	890	0.597355	0.592841	0.597298	-	-	-	0.000000	0.000000	0.000000	0.000000	Other	Yes	Online B (2015)
60	ORGANIZER	zh-ja	2016/07/26 11:54:14	1043	0.589802	0.589397	0.593361	-	-	-	-	0.000000	0.000000	0.000000	Other	Yes	Online A (2016)
61	EHR	zh-ja	2016/07/31 17:06:57	1063	0.843723	0.836156	0.841952	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	LM-based merging of outputs of preordered word-based PBSMT(DL=6) and preordered character-based PBSMT(DL=6).
62	NICT-2	zh-ja	2016/08/05 18:05:03	1099	0.843941	0.837707	0.842513	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	Phrase-based SMT with Preordering + Domain Adaptation (JPC and ASPEC) + Google 5-gram LM
63	Kyoto-U	zh-ja	2016/08/07 18:28:23	1110	0.820259	0.814661	0.819963	-	-	-	-	0.000000	0.000000	0.000000	EBMT	No	KyotoEBMT 2016 w/o reranking
64	bjtu_nlp	zh-ja	2016/08/12 12:50:38	1138	0.852818	0.846301	0.852298	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	RNN Encoder-Decoder with attention mechanism, single model
65	JAPIO	zh-ja	2016/08/19 16:44:49	1208	0.790553	0.780637	0.785917	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	Phrase-based SMT with Preordering + JAPIO corpus + rule-based posteditor
66	UT-KAY	zh-ja	2016/08/20 07:09:54	1220	0.847407	0.842055	0.848040	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	An end-to-end NMT with 512 dimensional single-layer LSTMs, UNK replacement, and domain adaptation
67	UT-KAY	zh-ja	2016/08/20 07:12:52	1221	0.860214	0.854690	0.860449	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	Ensemble of our NMT models with and without domain adaptation
68	Kyoto-U	zh-ja	2016/08/20 22:48:16	1255	0.869360	0.864748	0.869913	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	src: 200k tgt: 50k 2-layers self-ensembling
69	Kyoto-U	zh-ja	2016/08/20 22:50:33	1256	0.876531	0.872904	0.876946	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	voc: 30k ensemble of 3 independent model + reverse rescoring
70	Kyoto-U	zh-ja	2016/10/11 10:46:03	1324	0.875279	0.870175	0.875564	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	voc: 32k ensemble of 4 independent model + Chinese short unit
71	ORGANIZER	zh-ja	2016/11/16 11:28:00	1342	0.719022	0.717173	0.720095	-	-	-	-	0.000000	0.000000	0.000000	NMT	Yes	Online A (2016/11/14)
72	NICT-2	zh-ja	2017/07/26 13:58:44	1477	0.871438	0.868359	0.871736	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	NMT Single Model: BPE50k, Bi-LSTM(500*2) Encoder, LSTM(1000) Left-to-Right Decoder
73	NICT-2	zh-ja	2017/07/26 14:08:45	1481	0.882356	0.878580	0.882195	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	NMT 6 Ensembles * Bi-directional Reranking
74	Kyoto-U	zh-ja	2017/07/29 08:02:07	1577	0.883457	0.878964	0.884137	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	Ensemble of 5 Shared BPE 40k
75	Kyoto-U	zh-ja	2017/07/31 15:27:21	1643	0.878008	0.872944	0.878627	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	KW replacement without KW in the test set, BPE, 6 ensemble
76	Kyoto-U	zh-ja	2017/08/01 14:14:49	1720	0.884210	0.880069	0.884745	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	Ensemble of 7 shared BPE, averaged
77	ORGANIZER	zh-ja	2017/08/02 09:59:33	1740	0.880815	0.875511	0.880368	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	Google's "Attention Is All You Need"

Notice:

This table is sorted by the leftmost segmenters. You can change the segmenter used to sort by clicking each segmenter link.

AMFM

#	Team	Task	Date/Time	DataID	AMFM										Method	Other Resources	System Description
#	Team	Task	Date/Time	DataID	unuse	unuse	unuse	unuse	unuse	unuse	unuse	unuse	unuse	unuse	Method	Other Resources	System Description
1	ORGANIZER	zh-ja	2018/08/14 11:33:03	1902	0.782100	0.782100	0.782100	-	-	-	-	-	0.000000	0.000000	NMT	No	NMT with Attention
2	NICT-5	zh-ja	2018/08/22 18:51:44	2052	0.800670	0.800670	0.800670	-	-	-	-	-	0.000000	0.000000	NMT	No	Mixed fine tuning by first pretraining on En-Ja ASPEC data and then continue on the En-Ja+Zh-Ja data. Transformer.
3	NICT-5	zh-ja	2018/08/27 14:40:35	2169	0.805750	0.805750	0.805750	-	-	-	-	-	0.000000	0.000000	NMT	No	Combining En-Ja corpus with Zh-Ja as a multilingual model. ADDITIONAL ASPEC CORPUS USED
4	NICT-5	zh-ja	2018/09/10 14:14:05	2267	0.804920	0.804920	0.804920	-	-	-	-	-	0.000000	0.000000	NMT	No	MLNMT
5	TMU	zh-ja	2018/09/14 17:30:33	2343	0.512430	0.512430	0.512430	-	-	-	-	-	0.000000	0.000000	NMT	Yes	Unsupervised NMT with sub-character information. Both ASPEC and JPC 4.0 data (zh-ja) were also used as monolingual data in the training.
6	srcb	zh-ja	2019/07/25 11:37:44	2917	0.812450	0.812450	0.812450	-	-	-	-	-	-	-	NMT	No	Transformer (Big) with relative position, layer attention, sentence-wise smooth.
7	KNU_Hyundai	zh-ja	2019/07/27 10:30:04	3179	0.809990	0.809990	0.809990	-	-	-	-	-	-	-	NMT	No	Transformer(base) + Used ASPEC ja-en corpus with relative position, bt, multi source, r2l rerank, 6-model ensemble
8	srcb	zh-ja	2019/07/27 15:48:24	3210	0.819020	0.819020	0.819020	-	-	-	-	-	-	-	NMT	No	Transformer(Big) with relative position, sentence-wise smooth, deep transformer, back translation, ensemble of 7 models.
9	Kyoto-U+ECNU	zh-ja	2020/09/10 23:58:13	3677	0.817610	0.817610	0.817610	-	-	-	-	-	-	-	NMT	No	back-translation by using ja monolingual data from ASPEC-JE; lightconv (pay less attention) single model without ensemble
10	Kyoto-U+ECNU	zh-ja	2020/09/17 18:41:34	3813	0.823390	0.823390	0.823390	-	-	-	-	-	-	-	NMT	Yes	ensemble 9 models: structures(LSTM, Transformer, ConvS2S, Lightconv), training data(BT, out-of-domain parallel), S2S settings(deeper transformer, deep encoder shallow decoder)
11	Kyoto-U+ECNU	zh-ja	2020/09/18 17:38:55	3933	0.821660	0.821660	0.821660	-	-	-	-	-	-	-	NMT	No	without out-of-domain parallel data; others same as DataID:3813
12	ORGANIZER	zh-ja	2014/07/11 19:47:27	4	0.750950	0.750950	0.750950	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Hierarchical Phrase-based SMT (2014)
13	ORGANIZER	zh-ja	2014/07/11 19:54:58	8	0.753010	0.753010	0.753010	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Phrase-based SMT
14	ORGANIZER	zh-ja	2014/07/11 20:04:10	13	0.754870	0.754870	0.754870	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Tree-to-String SMT (2014)
15	ORGANIZER	zh-ja	2014/07/18 11:09:12	36	0.658060	0.658060	0.658060	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	Other	Yes	Online A (2014)
16	NAIST	zh-ja	2014/07/31 11:42:31	120	0.768190	0.768190	0.768190	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Travatar-based Forest-to-String SMT System
17	NAIST	zh-ja	2014/08/01 17:33:01	124	0.766270	0.766270	0.766270	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Travatar-based Forest-to-String SMT System (Tuned BLEU+RIBES)
18	Kyoto-U	zh-ja	2014/08/19 09:31:08	133	0.750310	0.750310	0.750310	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	EBMT	No	Using n-best parses and RNNLM.
19	Kyoto-U	zh-ja	2014/08/19 10:21:37	135	0.748200	0.748200	0.748200	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	EBMT	No	Our baseline system.
20	EIWA	zh-ja	2014/08/20 11:52:45	137	0.613730	0.613730	0.613730	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	RBMT	Yes	RBMT plus user dictionary
21	EIWA	zh-ja	2014/08/20 11:56:00	138	0.693330	0.693330	0.693330	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	RBMT with user dictionary plus SPE(statistical post editing)
22	Sense	zh-ja	2014/08/26 15:17:49	200	0.752890	0.752890	0.752890	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Character based SMT
23	ORGANIZER	zh-ja	2014/08/28 12:10:13	215	0.636930	0.636930	0.636930	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	Other	Yes	Online B (2014)
24	SAS_MT	zh-ja	2014/08/29 15:33:07	232	0.752170	0.752170	0.752170	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Syntactic reordering phrase-based SMT （SAS token tool）
25	ORGANIZER	zh-ja	2014/08/29 18:45:03	239	0.626070	0.626070	0.626070	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	RBMT	No	RBMT A (2014)
26	ORGANIZER	zh-ja	2014/08/29 18:48:29	242	0.586790	0.586790	0.586790	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	RBMT	No	RBMT D
27	Kyoto-U	zh-ja	2014/08/31 23:42:41	258	0.750370	0.750370	0.750370	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	EBMT	No	Our new baseline system after several modifications.
28	SAS_MT	zh-ja	2014/09/01 10:38:13	263	0.765730	0.765730	0.765730	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Syntactic reordering Hierarchical SMT （using SAS token tool）
29	Kyoto-U	zh-ja	2014/09/01 21:33:23	268	0.757610	0.757610	0.757610	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	EBMT	No	Our new baseline system after several modifications + 20-best parses, KN7, RNNLM reranking
30	WASUIPS	zh-ja	2014/09/17 00:43:38	369	0.711650	0.711650	0.711650	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Our baseline system (segmentation tools: urheen and mecab, moses: 1.0).
31	WASUIPS	zh-ja	2014/09/17 00:46:07	370	0.734620	0.734620	0.734620	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: urheen and mecab, moses: 1.0).
32	WASUIPS	zh-ja	2014/09/17 01:03:57	374	0.740650	0.740650	0.740650	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Our baseline system (segmentation tools: urheen and mecab, moses: 2.1.1).
33	WASUIPS	zh-ja	2014/09/17 01:05:38	375	0.740640	0.740640	0.740640	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: urheen and mecab, moses: 2.1.1).
34	WASUIPS	zh-ja	2014/09/17 10:07:44	379	0.725360	0.725360	0.725360	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Our baseline system (segmentation tools: kytea, moses: 1.0).
35	WASUIPS	zh-ja	2014/09/17 10:10:47	380	0.725250	0.725250	0.725250	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: kytea, moses: 1.0).
36	WASUIPS	zh-ja	2014/09/17 10:24:50	383	0.753750	0.753750	0.753750	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Our baseline system (segmentation tools: kytea, moses: 2.1.1).
37	WASUIPS	zh-ja	2014/09/17 10:26:43	384	0.753690	0.753690	0.753690	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: kytea, moses: 2.1.1).
38	WASUIPS	zh-ja	2014/09/17 11:03:46	387	0.743140	0.743140	0.743140	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Our baseline system (segmentation tools: stanford-ctb and juman, moses: 2.1.1).
39	WASUIPS	zh-ja	2014/09/17 12:00:46	388	0.744040	0.744040	0.744040	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: stanford-ctb and juman, moses: 2.1.1).
40	Kyoto-U	zh-ja	2015/07/17 09:01:42	490	0.757070	0.757070	0.757070	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	EBMT	No	WAT2015 baseline
41	Kyoto-U	zh-ja	2015/07/17 09:04:22	491	0.762180	0.762180	0.762180	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	EBMT	No	WAT2015 baseline with reranking
42	TOSHIBA	zh-ja	2015/07/23 15:14:53	508	0.752830	0.752830	0.752830	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	System combination SMT and RBMT(SPE) with RNNLM language model
43	TOSHIBA	zh-ja	2015/07/28 16:27:32	525	0.758110	0.758110	0.758110	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	RBMT with SPE(Statistical Post Editing) system
44	Sense	zh-ja	2015/07/29 07:20:20	533	0.733190	0.733190	0.733190	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Baseline-2015
45	Kyoto-U	zh-ja	2015/08/07 13:24:55	597	0.762430	0.762430	0.762430	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	EBMT	No	Updated JUMAN and added one reordering feature, w/ reranking
46	TOSHIBA	zh-ja	2015/08/17 12:11:52	669	0.654080	0.654080	0.654080	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	RBMT	Yes	RBMT
47	EHR	zh-ja	2015/08/19 11:23:36	720	0.765050	0.765050	0.765050	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	System combination of RBMT with user dictionary plus SPE and phrase based SMT with preordering. Candidate selection by language model score.
48	BJTUNLP	zh-ja	2015/08/25 14:55:20	769	0.744130	0.744130	0.744130	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No
49	ORGANIZER	zh-ja	2015/08/25 18:58:08	776	0.649860	0.649860	0.649860	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	Other	Yes	Online A (2015)
50	NAIST	zh-ja	2015/08/31 08:23:30	834	0.771010	0.771010	0.771010	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Travatar System with NeuralMT Reranking
51	NAIST	zh-ja	2015/08/31 08:26:31	835	0.764830	0.764830	0.764830	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Travatar System Baseline
52	Kyoto-U	zh-ja	2015/08/31 22:38:22	844	0.761960	0.761960	0.761960	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	EBMT	No	KyotoEBMT system without reranking
53	Kyoto-U	zh-ja	2015/08/31 22:39:36	845	0.769700	0.769700	0.769700	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	EBMT	No	KyotoEBMT system with bilingual RNNLM reranking
54	BJTUNLP	zh-ja	2015/09/01 21:08:10	862	0.744130	0.744130	0.744130	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	a dependency-to-string model for SMT
55	EHR	zh-ja	2015/09/02 17:00:16	867	0.707310	0.707310	0.707310	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Phrase based SMT with preordering.
56	EHR	zh-ja	2015/09/04 11:44:26	868	0.754180	0.754180	0.754180	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	RBMT with user dictionary plus SPE.
57	ORGANIZER	zh-ja	2015/09/10 14:00:33	879	0.754870	0.754870	0.754870	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Tree-to-String SMT (2015)
58	ORGANIZER	zh-ja	2015/09/10 14:30:56	885	0.626070	0.626070	0.626070	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	Other	Yes	RBMT A (2015)
59	ORGANIZER	zh-ja	2015/09/11 10:09:23	890	0.628290	0.628290	0.628290	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	Other	Yes	Online B (2015)
60	ORGANIZER	zh-ja	2016/07/26 11:54:14	1043	0.659540	0.659540	0.659540	-	-	-	-	0.000000	0.000000	0.000000	Other	Yes	Online A (2016)
61	EHR	zh-ja	2016/07/31 17:06:57	1063	0.769490	0.769490	0.769490	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	LM-based merging of outputs of preordered word-based PBSMT(DL=6) and preordered character-based PBSMT(DL=6).
62	NICT-2	zh-ja	2016/08/05 18:05:03	1099	0.768580	0.768580	0.768580	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	Phrase-based SMT with Preordering + Domain Adaptation (JPC and ASPEC) + Google 5-gram LM
63	Kyoto-U	zh-ja	2016/08/07 18:28:23	1110	0.767120	0.767120	0.767120	-	-	-	-	0.000000	0.000000	0.000000	EBMT	No	KyotoEBMT 2016 w/o reranking
64	bjtu_nlp	zh-ja	2016/08/12 12:50:38	1138	0.760840	0.760840	0.760840	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	RNN Encoder-Decoder with attention mechanism, single model
65	JAPIO	zh-ja	2016/08/19 16:44:49	1208	0.696770	0.696770	0.696770	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	Phrase-based SMT with Preordering + JAPIO corpus + rule-based posteditor
66	UT-KAY	zh-ja	2016/08/20 07:09:54	1220	0.753820	0.753820	0.753820	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	An end-to-end NMT with 512 dimensional single-layer LSTMs, UNK replacement, and domain adaptation
67	UT-KAY	zh-ja	2016/08/20 07:12:52	1221	0.765530	0.765530	0.765530	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	Ensemble of our NMT models with and without domain adaptation
68	Kyoto-U	zh-ja	2016/08/20 22:48:16	1255	0.784380	0.784380	0.784380	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	src: 200k tgt: 50k 2-layers self-ensembling
69	Kyoto-U	zh-ja	2016/08/20 22:50:33	1256	0.785910	0.785910	0.785910	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	voc: 30k ensemble of 3 independent model + reverse rescoring
70	Kyoto-U	zh-ja	2016/10/11 10:46:03	1324	0.787930	0.787930	0.787930	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	voc: 32k ensemble of 4 independent model + Chinese short unit
71	ORGANIZER	zh-ja	2016/11/16 11:28:00	1342	0.692820	0.692820	0.692820	-	-	-	-	0.000000	0.000000	0.000000	NMT	Yes	Online A (2016/11/14)
72	NICT-2	zh-ja	2017/07/26 13:58:44	1477	0.788940	0.788940	0.788940	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	NMT Single Model: BPE50k, Bi-LSTM(500*2) Encoder, LSTM(1000) Left-to-Right Decoder
73	NICT-2	zh-ja	2017/07/26 14:08:45	1481	0.799680	0.799680	0.799680	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	NMT 6 Ensembles * Bi-directional Reranking
74	Kyoto-U	zh-ja	2017/07/29 08:02:07	1577	0.799520	0.799520	0.799520	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	Ensemble of 5 Shared BPE 40k
75	Kyoto-U	zh-ja	2017/07/31 15:27:21	1643	0.793410	0.793410	0.793410	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	KW replacement without KW in the test set, BPE, 6 ensemble
76	Kyoto-U	zh-ja	2017/08/01 14:14:49	1720	0.799840	0.799840	0.799840	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	Ensemble of 7 shared BPE, averaged
77	ORGANIZER	zh-ja	2017/08/02 09:59:33	1740	0.798110	0.798110	0.798110	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	Google's "Attention Is All You Need"

Notice:

This table is sorted by the leftmost segmenters. You can change the segmenter used to sort by clicking each segmenter link.
Adequacy-Fluency Metrics (AMFM) is a two-dimensional automatic evaluation metric for machine translation, designed to operate at the sentence level. It is based on adequacy and fluency, to decouple semantic and syntactic components of the translation process to provide a balanced view on translation quality.
AMFM is calculated without tokenizers.
The detail of AMFM is shown on the following paper: "Adequacy–Fluency Metrics: Evaluating MT in the Continuous Space Model Framework" [pdf]. Invited Talk in WAT2015 also helps understanding [slide].

HUMAN (WAT2022)

Notice:

HUMAN (WAT2022) is the result of the Pairwise Crowdsourcing Evaluation on WAT2022.
HUMAN (WAT2022) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.

HUMAN (WAT2021)

Notice:

HUMAN (WAT2021) is the result of the Pairwise Crowdsourcing Evaluation on WAT2021.
HUMAN (WAT2021) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.

HUMAN (WAT2020)

Notice:

HUMAN (WAT2020) is the result of the Pairwise Crowdsourcing Evaluation on WAT2020.
HUMAN (WAT2020) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.

HUMAN (WAT2019)

Notice:

HUMAN (WAT2019) is the result of the Pairwise Crowdsourcing Evaluation on WAT2019.
HUMAN (WAT2019) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.

HUMAN (WAT2018)

Notice:

HUMAN (WAT2018) is the result of the Pairwise Crowdsourcing Evaluation on WAT2018.
HUMAN (WAT2018) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.

HUMAN (WAT2017)

Notice:

HUMAN (WAT2017) is the result of the Pairwise Crowdsourcing Evaluation on WAT2017.
HUMAN (WAT2017) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.

HUMAN (WAT2016)

#	Team	Task	Date/Time	DataID	HUMAN	Method	Other Resources	System Description
1	Kyoto-U	zh-ja	2016/08/20 22:50:33	1256	63.750	NMT	No	voc: 30k ensemble of 3 independent model + reverse rescoring
2	Kyoto-U	zh-ja	2016/08/20 22:48:16	1255	56.000	NMT	No	src: 200k tgt: 50k 2-layers self-ensembling
3	bjtu_nlp	zh-ja	2016/08/12 12:50:38	1138	49.000	NMT	No	RNN Encoder-Decoder with attention mechanism, single model
4	UT-KAY	zh-ja	2016/08/20 07:12:52	1221	47.250	NMT	No	Ensemble of our NMT models with and without domain adaptation
5	UT-KAY	zh-ja	2016/08/20 07:09:54	1220	41.000	NMT	No	An end-to-end NMT with 512 dimensional single-layer LSTMs, UNK replacement, and domain adaptation
6	NICT-2	zh-ja	2016/08/05 18:05:03	1099	36.500	SMT	Yes	Phrase-based SMT with Preordering + Domain Adaptation (JPC and ASPEC) + Google 5-gram LM
7	EHR	zh-ja	2016/07/31 17:06:57	1063	32.500	SMT	Yes	LM-based merging of outputs of preordered word-based PBSMT(DL=6) and preordered character-based PBSMT(DL=6).
8	ORGANIZER	zh-ja	2016/11/16 11:28:00	1342	22.500	NMT	Yes	Online A (2016/11/14)
9	JAPIO	zh-ja	2016/08/19 16:44:49	1208	16.500	SMT	Yes	Phrase-based SMT with Preordering + JAPIO corpus + rule-based posteditor
10	ORGANIZER	zh-ja	2016/07/26 11:54:14	1043	-51.250	Other	Yes	Online A (2016)

Notice:

HUMAN (WAT2016) is the result of the Pairwise Crowdsourcing Evaluation on WAT2016.
HUMAN (WAT2016) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.

HUMAN (WAT2015)

#	Team	Task	Date/Time	DataID	HUMAN	Method	Other Resources	System Description
1	NAIST	zh-ja	2015/08/31 08:23:30	834	35.750	SMT	No	Travatar System with NeuralMT Reranking
2	EHR	zh-ja	2015/08/19 11:23:36	720	25.750	SMT and RBMT	Yes	System combination of RBMT with user dictionary plus SPE and phrase based SMT with preordering. Candidate selection by language model score.
3	NAIST	zh-ja	2015/08/31 08:26:31	835	25.750	SMT	No	Travatar System Baseline
4	Kyoto-U	zh-ja	2015/08/31 22:39:36	845	18.500	EBMT	No	KyotoEBMT system with bilingual RNNLM reranking
5	TOSHIBA	zh-ja	2015/07/23 15:14:53	508	18.000	SMT and RBMT	Yes	System combination SMT and RBMT(SPE) with RNNLM language model
6	ORGANIZER	zh-ja	2015/09/10 14:00:33	879	17.250	SMT	No	Tree-to-String SMT (2015)
7	Kyoto-U	zh-ja	2015/08/31 22:38:22	844	16.750	EBMT	No	KyotoEBMT system without reranking
8	BJTUNLP	zh-ja	2015/09/01 21:08:10	862	6.500	SMT	No	a dependency-to-string model for SMT
9	TOSHIBA	zh-ja	2015/07/28 16:27:32	525	-1.000	SMT and RBMT	Yes	RBMT with SPE(Statistical Post Editing) system
10	ORGANIZER	zh-ja	2015/08/25 18:58:08	776	-19.000	Other	Yes	Online A (2015)
11	ORGANIZER	zh-ja	2015/09/10 14:30:56	885	-28.000	Other	Yes	RBMT A (2015)

Notice:

HUMAN (WAT2015) is the result of the Pairwise Crowdsourcing Evaluation on WAT2015.
HUMAN (WAT2015) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.
The detail of the evaluation can be found in the pdf document (PDF file).

HUMAN (WAT2014)

#	Team	Task	Date/Time	DataID	HUMAN	Method	Other Resources	System Description
1	NAIST	zh-ja	2014/07/31 11:42:31	120	50.750	SMT	No	Travatar-based Forest-to-String SMT System
2	NAIST	zh-ja	2014/08/01 17:33:01	124	38.000	SMT	No	Travatar-based Forest-to-String SMT System (Tuned BLEU+RIBES)
3	SAS_MT	zh-ja	2014/09/01 10:38:13	263	22.500	SMT	No	Syntactic reordering Hierarchical SMT （using SAS token tool）
4	ORGANIZER	zh-ja	2014/07/11 20:04:10	13	16.000	SMT	No	Tree-to-String SMT (2014)
5	EIWA	zh-ja	2014/08/20 11:56:00	138	15.000	SMT and RBMT	Yes	RBMT with user dictionary plus SPE(statistical post editing)
6	Kyoto-U	zh-ja	2014/09/01 21:33:23	268	7.500	EBMT	No	Our new baseline system after several modifications + 20-best parses, KN7, RNNLM reranking
7	Kyoto-U	zh-ja	2014/08/31 23:42:41	258	6.000	EBMT	No	Our new baseline system after several modifications.
8	ORGANIZER	zh-ja	2014/07/11 19:47:27	4	4.750	SMT	No	Hierarchical Phrase-based SMT (2014)
9	Sense	zh-ja	2014/08/26 15:17:49	200	-1.000	SMT	No	Character based SMT
10	ORGANIZER	zh-ja	2014/07/18 11:09:12	36	-21.750	Other	Yes	Online A (2014)
11	ORGANIZER	zh-ja	2014/08/29 18:45:03	239	-37.750	RBMT	No	RBMT A (2014)

Notice:

HUMAN (WAT2014) is the result of the Pairwise Crowdsourcing Evaluation on WAT2014.
HUMAN (WAT2014) was evaluated by 3 different workers and the final decision is made by the voting of the judgements.
The detail of the evaluation can be found in the pdf document (PDF file).

EVALUATION RESULTS USAGE POLICY

When you use the WAT evaluation results for any purpose such as:
- writing technical papers,
- making presentations about your system,
- advertising your MT system to the customers,
you can use the information about translation directions, scores (including both automatic and human evaluations) and ranks of your system among others. You can also use the scores of the other systems, but you MUST anonymize the other system's names. In addition, you can show the links (URLs) to the WAT evaluation result pages.

NICT (National Institute of Information and Communications Technology)
Kyoto University
Last Modified: 2018-08-02

WAT The Workshop on Asian Translation Evaluation Results

BLEU

RIBES

AMFM

HUMAN (WAT2022)

HUMAN (WAT2021)

HUMAN (WAT2020)

HUMAN (WAT2019)

HUMAN (WAT2018)

HUMAN (WAT2017)

HUMAN (WAT2016)

HUMAN (WAT2015)

HUMAN (WAT2014)

EVALUATION RESULTS USAGE POLICY

WAT

The Workshop on Asian Translation

Evaluation Results