WAT

The Workshop on Asian Translation

Evaluation Results

BLEU

#	Team	Task	Date/Time	DataID	BLEU										Method	Other Resources	System Description
#	Team	Task	Date/Time	DataID	juman	kytea	mecab	moses- tokenizer	stanford- segmenter- ctb	stanford- segmenter- pku	indic- tokenizer	unuse	myseg	kmseg	Method	Other Resources	System Description
1	Kyoto-U+ECNU	zh-ja	2020/09/17 18:41:34	3813	52.80	53.64	52.92	-	-	-	-	-	-	-	NMT	Yes	ensemble 9 models: structures(LSTM, Transformer, ConvS2S, Lightconv), training data(BT, out-of-domain parallel), S2S settings(deeper transformer, deep encoder shallow decoder)
2	Kyoto-U+ECNU	zh-ja	2020/09/18 17:38:55	3933	52.65	53.48	52.80	-	-	-	-	-	-	-	NMT	No	without out-of-domain parallel data; others same as DataID:3813
3	srcb	zh-ja	2019/07/27 15:48:24	3210	52.37	53.58	52.57	-	-	-	-	-	-	-	NMT	No	Transformer(Big) with relative position, sentence-wise smooth, deep transformer, back translation, ensemble of 7 models.
4	Kyoto-U+ECNU	zh-ja	2020/09/10 23:58:13	3677	50.37	51.27	50.60	-	-	-	-	-	-	-	NMT	No	back-translation by using ja monolingual data from ASPEC-JE; lightconv (pay less attention) single model without ensemble
5	KNU_Hyundai	zh-ja	2019/07/27 10:30:04	3179	50.02	50.84	50.23	-	-	-	-	-	-	-	NMT	No	Transformer(base) + Used ASPEC ja-en corpus with relative position, bt, multi source, r2l rerank, 6-model ensemble
6	srcb	zh-ja	2019/07/25 11:37:44	2917	49.83	50.89	50.00	-	-	-	-	-	-	-	NMT	No	Transformer (Big) with relative position, layer attention, sentence-wise smooth.
7	NICT-5	zh-ja	2018/09/10 14:14:05	2267	49.79	50.66	49.89	-	-	-	-	-	0.00	0.00	NMT	No	MLNMT
8	NICT-5	zh-ja	2018/08/27 14:40:35	2169	49.67	50.46	49.79	-	-	-	-	-	0.00	0.00	NMT	No	Combining En-Ja corpus with Zh-Ja as a multilingual model. ADDITIONAL ASPEC CORPUS USED
9	NICT-5	zh-ja	2018/08/22 18:51:44	2052	48.43	48.78	48.52	-	-	-	-	-	0.00	0.00	NMT	No	Mixed fine tuning by first pretraining on En-Ja ASPEC data and then continue on the En-Ja+Zh-Ja data. Transformer.
10	Kyoto-U	zh-ja	2017/07/29 08:02:07	1577	48.43	48.84	48.51	-	-	-	-	0.00	0.00	0.00	NMT	No	Ensemble of 5 Shared BPE 40k
11	Kyoto-U	zh-ja	2017/08/01 14:14:49	1720	48.34	48.76	48.40	-	-	-	-	0.00	0.00	0.00	NMT	No	Ensemble of 7 shared BPE, averaged
12	NICT-2	zh-ja	2017/07/26 14:08:45	1481	46.84	47.51	47.27	-	-	-	-	0.00	0.00	0.00	NMT	No	NMT 6 Ensembles * Bi-directional Reranking
13	ORGANIZER	zh-ja	2017/08/02 09:59:33	1740	46.87	47.30	47.00	-	-	-	-	0.00	0.00	0.00	NMT	No	Google's "Attention Is All You Need"
14	Kyoto-U	zh-ja	2017/07/31 15:27:21	1643	46.74	47.79	46.67	-	-	-	-	0.00	0.00	0.00	NMT	No	KW replacement without KW in the test set, BPE, 6 ensemble
15	Kyoto-U	zh-ja	2016/10/11 10:46:03	1324	46.36	47.02	46.50	-	-	-	-	0.00	0.00	0.00	NMT	No	voc: 32k ensemble of 4 independent model + Chinese short unit
16	Kyoto-U	zh-ja	2016/08/20 22:50:33	1256	46.04	46.70	46.05	-	-	-	-	0.00	0.00	0.00	NMT	No	voc: 30k ensemble of 3 independent model + reverse rescoring
17	NICT-2	zh-ja	2017/07/26 13:58:44	1477	44.26	44.90	44.50	-	-	-	-	0.00	0.00	0.00	NMT	No	NMT Single Model: BPE50k, Bi-LSTM(500*2) Encoder, LSTM(1000) Left-to-Right Decoder
18	Kyoto-U	zh-ja	2016/08/20 22:48:16	1255	44.29	45.05	44.32	-	-	-	-	0.00	0.00	0.00	NMT	No	src: 200k tgt: 50k 2-layers self-ensembling
19	ORGANIZER	zh-ja	2018/08/14 11:33:03	1902	43.31	43.53	43.34	-	-	-	-	-	0.00	0.00	NMT	No	NMT with Attention
20	NAIST	zh-ja	2015/08/31 08:23:30	834	41.75	42.95	41.93	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Travatar System with NeuralMT Reranking
21	UT-KAY	zh-ja	2016/08/20 07:12:52	1221	40.50	41.81	40.67	-	-	-	-	0.00	0.00	0.00	NMT	No	Ensemble of our NMT models with and without domain adaptation
22	NAIST	zh-ja	2014/07/31 11:42:31	120	40.11	41.29	40.30	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Travatar-based Forest-to-String SMT System
23	NICT-2	zh-ja	2016/08/05 18:05:03	1099	40.02	40.45	40.29	-	-	-	-	0.00	0.00	0.00	SMT	Yes	Phrase-based SMT with Preordering + Domain Adaptation (JPC and ASPEC) + Google 5-gram LM
24	NAIST	zh-ja	2014/08/01 17:33:01	124	40.21	40.82	40.15	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Travatar-based Forest-to-String SMT System (Tuned BLEU+RIBES)
25	EHR	zh-ja	2015/09/02 17:00:16	867	39.43	39.98	39.58	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Phrase based SMT with preordering.
26	NAIST	zh-ja	2015/08/31 08:26:31	835	39.36	40.51	39.47	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Travatar System Baseline
27	EHR	zh-ja	2016/07/31 17:06:57	1063	39.75	39.85	39.40	-	-	-	-	0.00	0.00	0.00	SMT	Yes	LM-based merging of outputs of preordered word-based PBSMT(DL=6) and preordered character-based PBSMT(DL=6).
28	bjtu_nlp	zh-ja	2016/08/12 12:50:38	1138	38.83	39.25	38.68	-	-	-	-	0.00	0.00	0.00	NMT	No	RNN Encoder-Decoder with attention mechanism, single model
29	Kyoto-U	zh-ja	2015/08/31 22:39:36	845	38.53	39.41	38.66	-	-	-	0.00	0.00	0.00	0.00	EBMT	No	KyotoEBMT system with bilingual RNNLM reranking
30	EHR	zh-ja	2015/08/19 11:23:36	720	37.90	38.68	37.98	-	-	-	0.00	0.00	0.00	0.00	SMT and RBMT	Yes	System combination of RBMT with user dictionary plus SPE and phrase based SMT with preordering. Candidate selection by language model score.
31	UT-KAY	zh-ja	2016/08/20 07:09:54	1220	37.63	39.07	37.82	-	-	-	-	0.00	0.00	0.00	NMT	No	An end-to-end NMT with 512 dimensional single-layer LSTMs, UNK replacement, and domain adaptation
32	Kyoto-U	zh-ja	2015/08/07 13:24:55	597	37.30	38.26	37.45	-	-	-	0.00	0.00	0.00	0.00	EBMT	No	Updated JUMAN and added one reordering feature, w/ reranking
33	TOSHIBA	zh-ja	2015/07/23 15:14:53	508	37.47	37.44	37.34	-	-	-	0.00	0.00	0.00	0.00	SMT and RBMT	Yes	System combination SMT and RBMT(SPE) with RNNLM language model
34	SAS_MT	zh-ja	2014/09/01 10:38:13	263	37.42	37.65	37.07	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Syntactic reordering Hierarchical SMT （using SAS token tool）
35	Kyoto-U	zh-ja	2015/07/17 09:04:22	491	36.76	37.82	36.94	-	-	-	0.00	0.00	0.00	0.00	EBMT	No	WAT2015 baseline with reranking
36	Kyoto-U	zh-ja	2016/08/07 18:28:23	1110	36.63	37.54	36.70	-	-	-	-	0.00	0.00	0.00	EBMT	No	KyotoEBMT 2016 w/o reranking
37	ORGANIZER	zh-ja	2014/07/11 20:04:10	13	36.52	37.07	36.64	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Tree-to-String SMT (2014)
38	ORGANIZER	zh-ja	2015/09/10 14:00:33	879	36.52	37.07	36.64	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Tree-to-String SMT (2015)
39	Kyoto-U	zh-ja	2015/08/31 22:38:22	844	36.30	37.22	36.44	-	-	-	0.00	0.00	0.00	0.00	EBMT	No	KyotoEBMT system without reranking
40	SAS_MT	zh-ja	2014/08/29 15:33:07	232	36.58	36.22	36.10	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Syntactic reordering phrase-based SMT （SAS token tool）
41	Kyoto-U	zh-ja	2015/07/17 09:01:42	490	35.66	36.71	35.81	-	-	-	0.00	0.00	0.00	0.00	EBMT	No	WAT2015 baseline
42	TOSHIBA	zh-ja	2015/07/28 16:27:32	525	35.85	36.02	35.73	-	-	-	0.00	0.00	0.00	0.00	SMT and RBMT	Yes	RBMT with SPE(Statistical Post Editing) system
43	ORGANIZER	zh-ja	2014/07/11 19:47:27	4	35.43	35.91	35.64	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Hierarchical Phrase-based SMT (2014)
44	EHR	zh-ja	2015/09/04 11:44:26	868	35.59	35.56	35.37	-	-	-	0.00	0.00	0.00	0.00	SMT and RBMT	Yes	RBMT with user dictionary plus SPE.
45	Kyoto-U	zh-ja	2014/09/01 21:33:23	268	34.75	35.89	34.83	-	-	-	0.00	0.00	0.00	0.00	EBMT	No	Our new baseline system after several modifications + 20-best parses, KN7, RNNLM reranking
46	BJTUNLP	zh-ja	2015/08/25 14:55:20	769	34.72	34.87	34.79	-	-	-	0.00	0.00	0.00	0.00	SMT	No
47	BJTUNLP	zh-ja	2015/09/01 21:08:10	862	34.72	34.87	34.79	-	-	-	0.00	0.00	0.00	0.00	SMT	No	a dependency-to-string model for SMT
48	ORGANIZER	zh-ja	2014/07/11 19:54:58	8	34.65	35.16	34.77	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Phrase-based SMT
49	EIWA	zh-ja	2014/08/20 11:56:00	138	33.53	33.74	33.87	-	-	-	0.00	0.00	0.00	0.00	SMT and RBMT	Yes	RBMT with user dictionary plus SPE(statistical post editing)
50	Kyoto-U	zh-ja	2014/08/19 09:31:08	133	33.26	35.09	33.62	-	-	-	0.00	0.00	0.00	0.00	EBMT	No	Using n-best parses and RNNLM.
51	Sense	zh-ja	2014/08/26 15:17:49	200	33.66	33.86	33.46	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Character based SMT
52	Kyoto-U	zh-ja	2014/08/31 23:42:41	258	33.57	34.43	33.45	-	-	-	0.00	0.00	0.00	0.00	EBMT	No	Our new baseline system after several modifications.
53	WASUIPS	zh-ja	2014/09/17 12:00:46	388	32.65	32.81	32.59	-	-	-	0.00	0.00	0.00	0.00	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: stanford-ctb and juman, moses: 2.1.1).
54	WASUIPS	zh-ja	2014/09/17 01:05:38	375	32.19	32.55	32.54	-	-	-	0.00	0.00	0.00	0.00	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: urheen and mecab, moses: 2.1.1).
55	WASUIPS	zh-ja	2014/09/17 10:26:43	384	32.43	33.36	32.48	-	-	-	0.00	0.00	0.00	0.00	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: kytea, moses: 2.1.1).
56	WASUIPS	zh-ja	2014/09/17 11:03:46	387	32.52	32.69	32.47	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Our baseline system (segmentation tools: stanford-ctb and juman, moses: 2.1.1).
57	Kyoto-U	zh-ja	2014/08/19 10:21:37	135	32.68	33.30	32.45	-	-	-	0.00	0.00	0.00	0.00	EBMT	No	Our baseline system.
58	WASUIPS	zh-ja	2014/09/17 01:03:57	374	31.87	32.26	32.26	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Our baseline system (segmentation tools: urheen and mecab, moses: 2.1.1).
59	WASUIPS	zh-ja	2014/09/17 10:24:50	383	32.08	33.09	32.18	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Our baseline system (segmentation tools: kytea, moses: 2.1.1).
60	WASUIPS	zh-ja	2014/09/17 00:46:07	370	30.44	30.92	30.86	-	-	-	0.00	0.00	0.00	0.00	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: urheen and mecab, moses: 1.0).
61	Sense	zh-ja	2015/07/29 07:20:20	533	29.29	30.52	29.45	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Baseline-2015
62	WASUIPS	zh-ja	2014/09/17 00:43:38	369	27.66	28.09	28.20	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Our baseline system (segmentation tools: urheen and mecab, moses: 1.0).
63	WASUIPS	zh-ja	2014/09/17 10:10:47	380	27.86	28.89	28.00	-	-	-	0.00	0.00	0.00	0.00	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: kytea, moses: 1.0).
64	WASUIPS	zh-ja	2014/09/17 10:07:44	379	27.37	28.28	27.43	-	-	-	0.00	0.00	0.00	0.00	SMT	No	Our baseline system (segmentation tools: kytea, moses: 1.0).
65	JAPIO	zh-ja	2016/08/19 16:44:49	1208	26.24	27.87	26.37	-	-	-	-	0.00	0.00	0.00	SMT	Yes	Phrase-based SMT with Preordering + JAPIO corpus + rule-based posteditor
66	TOSHIBA	zh-ja	2015/08/17 12:11:52	669	19.24	19.48	19.12	-	-	-	0.00	0.00	0.00	0.00	RBMT	Yes	RBMT
67	ORGANIZER	zh-ja	2016/11/16 11:28:00	1342	18.75	20.64	19.04	-	-	-	-	0.00	0.00	0.00	NMT	Yes	Online A (2016/11/14)
68	EIWA	zh-ja	2014/08/20 11:52:45	137	18.69	18.33	18.32	-	-	-	0.00	0.00	0.00	0.00	RBMT	Yes	RBMT plus user dictionary
69	ORGANIZER	zh-ja	2014/07/18 11:09:12	36	11.63	13.21	11.87	-	-	-	0.00	0.00	0.00	0.00	Other	Yes	Online A (2014)
70	ORGANIZER	zh-ja	2016/07/26 11:54:14	1043	11.56	12.87	11.69	-	-	-	-	0.00	0.00	0.00	Other	Yes	Online A (2016)
71	ORGANIZER	zh-ja	2015/08/25 18:58:08	776	11.53	12.82	11.68	-	-	-	0.00	0.00	0.00	0.00	Other	Yes	Online A (2015)
72	ORGANIZER	zh-ja	2014/08/28 12:10:13	215	10.48	11.26	10.47	-	-	-	0.00	0.00	0.00	0.00	Other	Yes	Online B (2014)
73	ORGANIZER	zh-ja	2015/09/11 10:09:23	890	10.41	11.03	10.36	-	-	-	0.00	0.00	0.00	0.00	Other	Yes	Online B (2015)
74	ORGANIZER	zh-ja	2014/08/29 18:45:03	239	9.37	9.87	9.35	-	-	-	0.00	0.00	0.00	0.00	RBMT	No	RBMT A (2014)
75	ORGANIZER	zh-ja	2015/09/10 14:30:56	885	9.37	9.87	9.35	-	-	-	0.00	0.00	0.00	0.00	Other	Yes	RBMT A (2015)
76	ORGANIZER	zh-ja	2014/08/29 18:48:29	242	8.39	8.70	8.30	-	-	-	0.00	0.00	0.00	0.00	RBMT	No	RBMT D
77	TMU	zh-ja	2018/09/14 17:30:33	2343	6.21	7.02	6.27	-	-	-	-	-	0.00	0.00	NMT	Yes	Unsupervised NMT with sub-character information. Both ASPEC and JPC 4.0 data (zh-ja) were also used as monolingual data in the training.

Notice:

This table is sorted by the leftmost segmenters. You can change the segmenter used to sort by clicking each segmenter link.

RIBES

#	Team	Task	Date/Time	DataID	RIBES										Method	Other Resources	System Description
#	Team	Task	Date/Time	DataID	juman	kytea	mecab	moses- tokenizer	stanford- segmenter- ctb	stanford- segmenter- pku	indic- tokenizer	unuse	myseg	kmseg	Method	Other Resources	System Description
1	Kyoto-U+ECNU	zh-ja	2020/09/17 18:41:34	3813	0.897053	0.894441	0.897199	-	-	-	-	-	-	-	NMT	Yes	ensemble 9 models: structures(LSTM, Transformer, ConvS2S, Lightconv), training data(BT, out-of-domain parallel), S2S settings(deeper transformer, deep encoder shallow decoder)
2	Kyoto-U+ECNU	zh-ja	2020/09/18 17:38:55	3933	0.896551	0.894073	0.896743	-	-	-	-	-	-	-	NMT	No	without out-of-domain parallel data; others same as DataID:3813
3	srcb	zh-ja	2019/07/27 15:48:24	3210	0.895231	0.891872	0.895663	-	-	-	-	-	-	-	NMT	No	Transformer(Big) with relative position, sentence-wise smooth, deep transformer, back translation, ensemble of 7 models.
4	NICT-5	zh-ja	2018/09/10 14:14:05	2267	0.889674	0.886490	0.889853	-	-	-	-	-	0.000000	0.000000	NMT	No	MLNMT
5	Kyoto-U+ECNU	zh-ja	2020/09/10 23:58:13	3677	0.889269	0.885500	0.889503	-	-	-	-	-	-	-	NMT	No	back-translation by using ja monolingual data from ASPEC-JE; lightconv (pay less attention) single model without ensemble
6	KNU_Hyundai	zh-ja	2019/07/27 10:30:04	3179	0.888530	0.886248	0.888706	-	-	-	-	-	-	-	NMT	No	Transformer(base) + Used ASPEC ja-en corpus with relative position, bt, multi source, r2l rerank, 6-model ensemble
7	srcb	zh-ja	2019/07/25 11:37:44	2917	0.886069	0.882712	0.886854	-	-	-	-	-	-	-	NMT	No	Transformer (Big) with relative position, layer attention, sentence-wise smooth.
8	NICT-5	zh-ja	2018/08/27 14:40:35	2169	0.886161	0.882989	0.886367	-	-	-	-	-	0.000000	0.000000	NMT	No	Combining En-Ja corpus with Zh-Ja as a multilingual model. ADDITIONAL ASPEC CORPUS USED
9	NICT-5	zh-ja	2018/08/22 18:51:44	2052	0.884426	0.879456	0.884782	-	-	-	-	-	0.000000	0.000000	NMT	No	Mixed fine tuning by first pretraining on En-Ja ASPEC data and then continue on the En-Ja+Zh-Ja data. Transformer.
10	Kyoto-U	zh-ja	2017/08/01 14:14:49	1720	0.884210	0.880069	0.884745	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	Ensemble of 7 shared BPE, averaged
11	Kyoto-U	zh-ja	2017/07/29 08:02:07	1577	0.883457	0.878964	0.884137	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	Ensemble of 5 Shared BPE 40k
12	NICT-2	zh-ja	2017/07/26 14:08:45	1481	0.882356	0.878580	0.882195	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	NMT 6 Ensembles * Bi-directional Reranking
13	ORGANIZER	zh-ja	2017/08/02 09:59:33	1740	0.880815	0.875511	0.880368	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	Google's "Attention Is All You Need"
14	Kyoto-U	zh-ja	2017/07/31 15:27:21	1643	0.878008	0.872944	0.878627	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	KW replacement without KW in the test set, BPE, 6 ensemble
15	Kyoto-U	zh-ja	2016/08/20 22:50:33	1256	0.876531	0.872904	0.876946	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	voc: 30k ensemble of 3 independent model + reverse rescoring
16	Kyoto-U	zh-ja	2016/10/11 10:46:03	1324	0.875279	0.870175	0.875564	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	voc: 32k ensemble of 4 independent model + Chinese short unit
17	NICT-2	zh-ja	2017/07/26 13:58:44	1477	0.871438	0.868359	0.871736	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	NMT Single Model: BPE50k, Bi-LSTM(500*2) Encoder, LSTM(1000) Left-to-Right Decoder
18	ORGANIZER	zh-ja	2018/08/14 11:33:03	1902	0.870734	0.866281	0.870886	-	-	-	-	-	0.000000	0.000000	NMT	No	NMT with Attention
19	Kyoto-U	zh-ja	2016/08/20 22:48:16	1255	0.869360	0.864748	0.869913	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	src: 200k tgt: 50k 2-layers self-ensembling
20	UT-KAY	zh-ja	2016/08/20 07:12:52	1221	0.860214	0.854690	0.860449	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	Ensemble of our NMT models with and without domain adaptation
21	NAIST	zh-ja	2015/08/31 08:23:30	834	0.855089	0.847746	0.854587	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Travatar System with NeuralMT Reranking
22	bjtu_nlp	zh-ja	2016/08/12 12:50:38	1138	0.852818	0.846301	0.852298	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	RNN Encoder-Decoder with attention mechanism, single model
23	UT-KAY	zh-ja	2016/08/20 07:09:54	1220	0.847407	0.842055	0.848040	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	An end-to-end NMT with 512 dimensional single-layer LSTMs, UNK replacement, and domain adaptation
24	NAIST	zh-ja	2014/08/01 17:33:01	124	0.845486	0.838092	0.845625	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Travatar-based Forest-to-String SMT System (Tuned BLEU+RIBES)
25	NICT-2	zh-ja	2016/08/05 18:05:03	1099	0.843941	0.837707	0.842513	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	Phrase-based SMT with Preordering + Domain Adaptation (JPC and ASPEC) + Google 5-gram LM
26	NAIST	zh-ja	2014/07/31 11:42:31	120	0.842477	0.834824	0.842235	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Travatar-based Forest-to-String SMT System
27	EHR	zh-ja	2016/07/31 17:06:57	1063	0.843723	0.836156	0.841952	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	LM-based merging of outputs of preordered word-based PBSMT(DL=6) and preordered character-based PBSMT(DL=6).
28	Kyoto-U	zh-ja	2015/08/31 22:39:36	845	0.840681	0.834451	0.839063	-	-	-	0.000000	0.000000	0.000000	0.000000	EBMT	No	KyotoEBMT system with bilingual RNNLM reranking
29	EHR	zh-ja	2015/09/02 17:00:16	867	0.837678	0.831682	0.837227	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Phrase based SMT with preordering.
30	NAIST	zh-ja	2015/08/31 08:26:31	835	0.834388	0.827148	0.834130	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Travatar System Baseline
31	SAS_MT	zh-ja	2014/09/01 10:38:13	263	0.834170	0.825551	0.833048	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Syntactic reordering Hierarchical SMT （using SAS token tool）
32	TOSHIBA	zh-ja	2015/07/23 15:14:53	508	0.827291	0.817395	0.825472	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	System combination SMT and RBMT(SPE) with RNNLM language model
33	ORGANIZER	zh-ja	2014/07/11 20:04:10	13	0.825292	0.820490	0.825025	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Tree-to-String SMT (2014)
34	ORGANIZER	zh-ja	2015/09/10 14:00:33	879	0.825292	0.820490	0.825025	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Tree-to-String SMT (2015)
35	EHR	zh-ja	2015/08/19 11:23:36	720	0.826003	0.818620	0.824806	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	System combination of RBMT with user dictionary plus SPE and phrase based SMT with preordering. Candidate selection by language model score.
36	TOSHIBA	zh-ja	2015/07/28 16:27:32	525	0.824740	0.815388	0.822423	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	RBMT with SPE(Statistical Post Editing) system
37	Kyoto-U	zh-ja	2015/08/07 13:24:55	597	0.822672	0.817037	0.822340	-	-	-	0.000000	0.000000	0.000000	0.000000	EBMT	No	Updated JUMAN and added one reordering feature, w/ reranking
38	Kyoto-U	zh-ja	2016/08/07 18:28:23	1110	0.820259	0.814661	0.819963	-	-	-	-	0.000000	0.000000	0.000000	EBMT	No	KyotoEBMT 2016 w/o reranking
39	Kyoto-U	zh-ja	2015/08/31 22:38:22	844	0.819743	0.814581	0.818794	-	-	-	0.000000	0.000000	0.000000	0.000000	EBMT	No	KyotoEBMT system without reranking
40	Kyoto-U	zh-ja	2015/07/17 09:04:22	491	0.818445	0.812910	0.817522	-	-	-	0.000000	0.000000	0.000000	0.000000	EBMT	No	WAT2015 baseline with reranking
41	SAS_MT	zh-ja	2014/08/29 15:33:07	232	0.822180	0.807535	0.817368	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Syntactic reordering phrase-based SMT （SAS token tool）
42	EHR	zh-ja	2015/09/04 11:44:26	868	0.815842	0.806726	0.813996	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	RBMT with user dictionary plus SPE.
43	Kyoto-U	zh-ja	2015/07/17 09:01:42	490	0.809395	0.803780	0.808692	-	-	-	0.000000	0.000000	0.000000	0.000000	EBMT	No	WAT2015 baseline
44	EIWA	zh-ja	2014/08/20 11:56:00	138	0.811350	0.800506	0.808504	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	RBMT with user dictionary plus SPE(statistical post editing)
45	ORGANIZER	zh-ja	2014/07/11 19:47:27	4	0.810406	0.798726	0.807665	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Hierarchical Phrase-based SMT (2014)
46	Kyoto-U	zh-ja	2014/09/01 21:33:23	268	0.802629	0.798631	0.802930	-	-	-	0.000000	0.000000	0.000000	0.000000	EBMT	No	Our new baseline system after several modifications + 20-best parses, KN7, RNNLM reranking
47	BJTUNLP	zh-ja	2015/08/25 14:55:20	769	0.807012	0.792488	0.802430	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No
48	BJTUNLP	zh-ja	2015/09/01 21:08:10	862	0.807012	0.792488	0.802430	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	a dependency-to-string model for SMT
49	Kyoto-U	zh-ja	2014/08/31 23:42:41	258	0.800949	0.795390	0.800986	-	-	-	0.000000	0.000000	0.000000	0.000000	EBMT	No	Our new baseline system after several modifications.
50	Kyoto-U	zh-ja	2014/08/19 09:31:08	133	0.791680	0.787105	0.791269	-	-	-	0.000000	0.000000	0.000000	0.000000	EBMT	No	Using n-best parses and RNNLM.
51	WASUIPS	zh-ja	2014/09/17 12:00:46	388	0.796777	0.781733	0.791219	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: stanford-ctb and juman, moses: 2.1.1).
52	WASUIPS	zh-ja	2014/09/17 11:03:46	387	0.796059	0.780402	0.790107	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Our baseline system (segmentation tools: stanford-ctb and juman, moses: 2.1.1).
53	WASUIPS	zh-ja	2014/09/17 10:26:43	384	0.796220	0.778075	0.789657	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: kytea, moses: 2.1.1).
54	WASUIPS	zh-ja	2014/09/17 10:24:50	383	0.793230	0.775168	0.787665	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Our baseline system (segmentation tools: kytea, moses: 2.1.1).
55	WASUIPS	zh-ja	2014/09/17 01:05:38	375	0.795838	0.780027	0.787591	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: urheen and mecab, moses: 2.1.1).
56	WASUIPS	zh-ja	2014/09/17 01:03:57	374	0.794303	0.777876	0.786422	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Our baseline system (segmentation tools: urheen and mecab, moses: 2.1.1).
57	Kyoto-U	zh-ja	2014/08/19 10:21:37	135	0.786229	0.783016	0.786352	-	-	-	0.000000	0.000000	0.000000	0.000000	EBMT	No	Our baseline system.
58	JAPIO	zh-ja	2016/08/19 16:44:49	1208	0.790553	0.780637	0.785917	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	Phrase-based SMT with Preordering + JAPIO corpus + rule-based posteditor
59	Sense	zh-ja	2014/08/26 15:17:49	200	0.789495	0.774338	0.784012	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Character based SMT
60	WASUIPS	zh-ja	2014/09/17 00:46:07	370	0.789824	0.773142	0.781475	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: urheen and mecab, moses: 1.0).
61	Sense	zh-ja	2015/07/29 07:20:20	533	0.774692	0.764847	0.772410	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Baseline-2015
62	ORGANIZER	zh-ja	2014/07/11 19:54:58	8	0.772498	0.766384	0.771005	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Phrase-based SMT
63	WASUIPS	zh-ja	2014/09/17 00:43:38	369	0.779183	0.762949	0.770846	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Our baseline system (segmentation tools: urheen and mecab, moses: 1.0).
64	WASUIPS	zh-ja	2014/09/17 10:10:47	380	0.776550	0.756721	0.769409	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: kytea, moses: 1.0).
65	WASUIPS	zh-ja	2014/09/17 10:07:44	379	0.774423	0.753749	0.767073	-	-	-	0.000000	0.000000	0.000000	0.000000	SMT	No	Our baseline system (segmentation tools: kytea, moses: 1.0).
66	TOSHIBA	zh-ja	2015/08/17 12:11:52	669	0.741665	0.727155	0.738298	-	-	-	0.000000	0.000000	0.000000	0.000000	RBMT	Yes	RBMT
67	EIWA	zh-ja	2014/08/20 11:52:45	137	0.740183	0.720281	0.732466	-	-	-	0.000000	0.000000	0.000000	0.000000	RBMT	Yes	RBMT plus user dictionary
68	ORGANIZER	zh-ja	2016/11/16 11:28:00	1342	0.719022	0.717173	0.720095	-	-	-	-	0.000000	0.000000	0.000000	NMT	Yes	Online A (2016/11/14)
69	ORGANIZER	zh-ja	2014/08/29 18:45:03	239	0.666277	0.652402	0.661730	-	-	-	0.000000	0.000000	0.000000	0.000000	RBMT	No	RBMT A (2014)
70	ORGANIZER	zh-ja	2015/09/10 14:30:56	885	0.666277	0.652402	0.661730	-	-	-	0.000000	0.000000	0.000000	0.000000	Other	Yes	RBMT A (2015)
71	ORGANIZER	zh-ja	2014/08/29 18:48:29	242	0.641189	0.626400	0.633319	-	-	-	0.000000	0.000000	0.000000	0.000000	RBMT	No	RBMT D
72	ORGANIZER	zh-ja	2014/08/28 12:10:13	215	0.600733	0.596006	0.600706	-	-	-	0.000000	0.000000	0.000000	0.000000	Other	Yes	Online B (2014)
73	ORGANIZER	zh-ja	2014/07/18 11:09:12	36	0.595925	0.598172	0.598573	-	-	-	0.000000	0.000000	0.000000	0.000000	Other	Yes	Online A (2014)
74	ORGANIZER	zh-ja	2015/09/11 10:09:23	890	0.597355	0.592841	0.597298	-	-	-	0.000000	0.000000	0.000000	0.000000	Other	Yes	Online B (2015)
75	ORGANIZER	zh-ja	2016/07/26 11:54:14	1043	0.589802	0.589397	0.593361	-	-	-	-	0.000000	0.000000	0.000000	Other	Yes	Online A (2016)
76	ORGANIZER	zh-ja	2015/08/25 18:58:08	776	0.588285	0.590393	0.592887	-	-	-	0.000000	0.000000	0.000000	0.000000	Other	Yes	Online A (2015)
77	TMU	zh-ja	2018/09/14 17:30:33	2343	0.570040	0.562331	0.565795	-	-	-	-	-	0.000000	0.000000	NMT	Yes	Unsupervised NMT with sub-character information. Both ASPEC and JPC 4.0 data (zh-ja) were also used as monolingual data in the training.

Notice:

This table is sorted by the leftmost segmenters. You can change the segmenter used to sort by clicking each segmenter link.

AMFM

#	Team	Task	Date/Time	DataID	AMFM										Method	Other Resources	System Description
#	Team	Task	Date/Time	DataID	unuse	unuse	unuse	unuse	unuse	unuse	unuse	unuse	unuse	unuse	Method	Other Resources	System Description
1	Kyoto-U+ECNU	zh-ja	2020/09/17 18:41:34	3813	0.823390	0.823390	0.823390	-	-	-	-	-	-	-	NMT	Yes	ensemble 9 models: structures(LSTM, Transformer, ConvS2S, Lightconv), training data(BT, out-of-domain parallel), S2S settings(deeper transformer, deep encoder shallow decoder)
2	Kyoto-U+ECNU	zh-ja	2020/09/18 17:38:55	3933	0.821660	0.821660	0.821660	-	-	-	-	-	-	-	NMT	No	without out-of-domain parallel data; others same as DataID:3813
3	srcb	zh-ja	2019/07/27 15:48:24	3210	0.819020	0.819020	0.819020	-	-	-	-	-	-	-	NMT	No	Transformer(Big) with relative position, sentence-wise smooth, deep transformer, back translation, ensemble of 7 models.
4	Kyoto-U+ECNU	zh-ja	2020/09/10 23:58:13	3677	0.817610	0.817610	0.817610	-	-	-	-	-	-	-	NMT	No	back-translation by using ja monolingual data from ASPEC-JE; lightconv (pay less attention) single model without ensemble
5	srcb	zh-ja	2019/07/25 11:37:44	2917	0.812450	0.812450	0.812450	-	-	-	-	-	-	-	NMT	No	Transformer (Big) with relative position, layer attention, sentence-wise smooth.
6	KNU_Hyundai	zh-ja	2019/07/27 10:30:04	3179	0.809990	0.809990	0.809990	-	-	-	-	-	-	-	NMT	No	Transformer(base) + Used ASPEC ja-en corpus with relative position, bt, multi source, r2l rerank, 6-model ensemble
7	NICT-5	zh-ja	2018/08/27 14:40:35	2169	0.805750	0.805750	0.805750	-	-	-	-	-	0.000000	0.000000	NMT	No	Combining En-Ja corpus with Zh-Ja as a multilingual model. ADDITIONAL ASPEC CORPUS USED
8	NICT-5	zh-ja	2018/09/10 14:14:05	2267	0.804920	0.804920	0.804920	-	-	-	-	-	0.000000	0.000000	NMT	No	MLNMT
9	NICT-5	zh-ja	2018/08/22 18:51:44	2052	0.800670	0.800670	0.800670	-	-	-	-	-	0.000000	0.000000	NMT	No	Mixed fine tuning by first pretraining on En-Ja ASPEC data and then continue on the En-Ja+Zh-Ja data. Transformer.
10	Kyoto-U	zh-ja	2017/08/01 14:14:49	1720	0.799840	0.799840	0.799840	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	Ensemble of 7 shared BPE, averaged
11	NICT-2	zh-ja	2017/07/26 14:08:45	1481	0.799680	0.799680	0.799680	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	NMT 6 Ensembles * Bi-directional Reranking
12	Kyoto-U	zh-ja	2017/07/29 08:02:07	1577	0.799520	0.799520	0.799520	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	Ensemble of 5 Shared BPE 40k
13	ORGANIZER	zh-ja	2017/08/02 09:59:33	1740	0.798110	0.798110	0.798110	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	Google's "Attention Is All You Need"
14	Kyoto-U	zh-ja	2017/07/31 15:27:21	1643	0.793410	0.793410	0.793410	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	KW replacement without KW in the test set, BPE, 6 ensemble
15	NICT-2	zh-ja	2017/07/26 13:58:44	1477	0.788940	0.788940	0.788940	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	NMT Single Model: BPE50k, Bi-LSTM(500*2) Encoder, LSTM(1000) Left-to-Right Decoder
16	Kyoto-U	zh-ja	2016/10/11 10:46:03	1324	0.787930	0.787930	0.787930	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	voc: 32k ensemble of 4 independent model + Chinese short unit
17	Kyoto-U	zh-ja	2016/08/20 22:50:33	1256	0.785910	0.785910	0.785910	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	voc: 30k ensemble of 3 independent model + reverse rescoring
18	Kyoto-U	zh-ja	2016/08/20 22:48:16	1255	0.784380	0.784380	0.784380	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	src: 200k tgt: 50k 2-layers self-ensembling
19	ORGANIZER	zh-ja	2018/08/14 11:33:03	1902	0.782100	0.782100	0.782100	-	-	-	-	-	0.000000	0.000000	NMT	No	NMT with Attention
20	NAIST	zh-ja	2015/08/31 08:23:30	834	0.771010	0.771010	0.771010	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Travatar System with NeuralMT Reranking
21	Kyoto-U	zh-ja	2015/08/31 22:39:36	845	0.769700	0.769700	0.769700	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	EBMT	No	KyotoEBMT system with bilingual RNNLM reranking
22	EHR	zh-ja	2016/07/31 17:06:57	1063	0.769490	0.769490	0.769490	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	LM-based merging of outputs of preordered word-based PBSMT(DL=6) and preordered character-based PBSMT(DL=6).
23	NICT-2	zh-ja	2016/08/05 18:05:03	1099	0.768580	0.768580	0.768580	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	Phrase-based SMT with Preordering + Domain Adaptation (JPC and ASPEC) + Google 5-gram LM
24	NAIST	zh-ja	2014/07/31 11:42:31	120	0.768190	0.768190	0.768190	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Travatar-based Forest-to-String SMT System
25	Kyoto-U	zh-ja	2016/08/07 18:28:23	1110	0.767120	0.767120	0.767120	-	-	-	-	0.000000	0.000000	0.000000	EBMT	No	KyotoEBMT 2016 w/o reranking
26	NAIST	zh-ja	2014/08/01 17:33:01	124	0.766270	0.766270	0.766270	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Travatar-based Forest-to-String SMT System (Tuned BLEU+RIBES)
27	SAS_MT	zh-ja	2014/09/01 10:38:13	263	0.765730	0.765730	0.765730	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Syntactic reordering Hierarchical SMT （using SAS token tool）
28	UT-KAY	zh-ja	2016/08/20 07:12:52	1221	0.765530	0.765530	0.765530	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	Ensemble of our NMT models with and without domain adaptation
29	EHR	zh-ja	2015/08/19 11:23:36	720	0.765050	0.765050	0.765050	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	System combination of RBMT with user dictionary plus SPE and phrase based SMT with preordering. Candidate selection by language model score.
30	NAIST	zh-ja	2015/08/31 08:26:31	835	0.764830	0.764830	0.764830	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Travatar System Baseline
31	Kyoto-U	zh-ja	2015/08/07 13:24:55	597	0.762430	0.762430	0.762430	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	EBMT	No	Updated JUMAN and added one reordering feature, w/ reranking
32	Kyoto-U	zh-ja	2015/07/17 09:04:22	491	0.762180	0.762180	0.762180	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	EBMT	No	WAT2015 baseline with reranking
33	Kyoto-U	zh-ja	2015/08/31 22:38:22	844	0.761960	0.761960	0.761960	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	EBMT	No	KyotoEBMT system without reranking
34	bjtu_nlp	zh-ja	2016/08/12 12:50:38	1138	0.760840	0.760840	0.760840	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	RNN Encoder-Decoder with attention mechanism, single model
35	TOSHIBA	zh-ja	2015/07/28 16:27:32	525	0.758110	0.758110	0.758110	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	RBMT with SPE(Statistical Post Editing) system
36	Kyoto-U	zh-ja	2014/09/01 21:33:23	268	0.757610	0.757610	0.757610	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	EBMT	No	Our new baseline system after several modifications + 20-best parses, KN7, RNNLM reranking
37	Kyoto-U	zh-ja	2015/07/17 09:01:42	490	0.757070	0.757070	0.757070	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	EBMT	No	WAT2015 baseline
38	ORGANIZER	zh-ja	2014/07/11 20:04:10	13	0.754870	0.754870	0.754870	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Tree-to-String SMT (2014)
39	ORGANIZER	zh-ja	2015/09/10 14:00:33	879	0.754870	0.754870	0.754870	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Tree-to-String SMT (2015)
40	EHR	zh-ja	2015/09/04 11:44:26	868	0.754180	0.754180	0.754180	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	RBMT with user dictionary plus SPE.
41	UT-KAY	zh-ja	2016/08/20 07:09:54	1220	0.753820	0.753820	0.753820	-	-	-	-	0.000000	0.000000	0.000000	NMT	No	An end-to-end NMT with 512 dimensional single-layer LSTMs, UNK replacement, and domain adaptation
42	WASUIPS	zh-ja	2014/09/17 10:24:50	383	0.753750	0.753750	0.753750	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Our baseline system (segmentation tools: kytea, moses: 2.1.1).
43	WASUIPS	zh-ja	2014/09/17 10:26:43	384	0.753690	0.753690	0.753690	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: kytea, moses: 2.1.1).
44	ORGANIZER	zh-ja	2014/07/11 19:54:58	8	0.753010	0.753010	0.753010	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Phrase-based SMT
45	Sense	zh-ja	2014/08/26 15:17:49	200	0.752890	0.752890	0.752890	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Character based SMT
46	TOSHIBA	zh-ja	2015/07/23 15:14:53	508	0.752830	0.752830	0.752830	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	System combination SMT and RBMT(SPE) with RNNLM language model
47	SAS_MT	zh-ja	2014/08/29 15:33:07	232	0.752170	0.752170	0.752170	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Syntactic reordering phrase-based SMT （SAS token tool）
48	ORGANIZER	zh-ja	2014/07/11 19:47:27	4	0.750950	0.750950	0.750950	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Hierarchical Phrase-based SMT (2014)
49	Kyoto-U	zh-ja	2014/08/31 23:42:41	258	0.750370	0.750370	0.750370	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	EBMT	No	Our new baseline system after several modifications.
50	Kyoto-U	zh-ja	2014/08/19 09:31:08	133	0.750310	0.750310	0.750310	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	EBMT	No	Using n-best parses and RNNLM.
51	Kyoto-U	zh-ja	2014/08/19 10:21:37	135	0.748200	0.748200	0.748200	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	EBMT	No	Our baseline system.
52	BJTUNLP	zh-ja	2015/08/25 14:55:20	769	0.744130	0.744130	0.744130	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No
53	BJTUNLP	zh-ja	2015/09/01 21:08:10	862	0.744130	0.744130	0.744130	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	a dependency-to-string model for SMT
54	WASUIPS	zh-ja	2014/09/17 12:00:46	388	0.744040	0.744040	0.744040	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: stanford-ctb and juman, moses: 2.1.1).
55	WASUIPS	zh-ja	2014/09/17 11:03:46	387	0.743140	0.743140	0.743140	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Our baseline system (segmentation tools: stanford-ctb and juman, moses: 2.1.1).
56	WASUIPS	zh-ja	2014/09/17 01:03:57	374	0.740650	0.740650	0.740650	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Our baseline system (segmentation tools: urheen and mecab, moses: 2.1.1).
57	WASUIPS	zh-ja	2014/09/17 01:05:38	375	0.740640	0.740640	0.740640	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: urheen and mecab, moses: 2.1.1).
58	WASUIPS	zh-ja	2014/09/17 00:46:07	370	0.734620	0.734620	0.734620	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: urheen and mecab, moses: 1.0).
59	Sense	zh-ja	2015/07/29 07:20:20	533	0.733190	0.733190	0.733190	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Baseline-2015
60	WASUIPS	zh-ja	2014/09/17 10:07:44	379	0.725360	0.725360	0.725360	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Our baseline system (segmentation tools: kytea, moses: 1.0).
61	WASUIPS	zh-ja	2014/09/17 10:10:47	380	0.725250	0.725250	0.725250	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	Yes	Our baseline system + additional quasi-parallel corpus (segmentation tools: kytea, moses: 1.0).
62	WASUIPS	zh-ja	2014/09/17 00:43:38	369	0.711650	0.711650	0.711650	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Our baseline system (segmentation tools: urheen and mecab, moses: 1.0).
63	EHR	zh-ja	2015/09/02 17:00:16	867	0.707310	0.707310	0.707310	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT	No	Phrase based SMT with preordering.
64	JAPIO	zh-ja	2016/08/19 16:44:49	1208	0.696770	0.696770	0.696770	-	-	-	-	0.000000	0.000000	0.000000	SMT	Yes	Phrase-based SMT with Preordering + JAPIO corpus + rule-based posteditor
65	EIWA	zh-ja	2014/08/20 11:56:00	138	0.693330	0.693330	0.693330	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	SMT and RBMT	Yes	RBMT with user dictionary plus SPE(statistical post editing)
66	ORGANIZER	zh-ja	2016/11/16 11:28:00	1342	0.692820	0.692820	0.692820	-	-	-	-	0.000000	0.000000	0.000000	NMT	Yes	Online A (2016/11/14)
67	ORGANIZER	zh-ja	2016/07/26 11:54:14	1043	0.659540	0.659540	0.659540	-	-	-	-	0.000000	0.000000	0.000000	Other	Yes	Online A (2016)
68	ORGANIZER	zh-ja	2014/07/18 11:09:12	36	0.658060	0.658060	0.658060	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	Other	Yes	Online A (2014)
69	TOSHIBA	zh-ja	2015/08/17 12:11:52	669	0.654080	0.654080	0.654080	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	RBMT	Yes	RBMT
70	ORGANIZER	zh-ja	2015/08/25 18:58:08	776	0.649860	0.649860	0.649860	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	Other	Yes	Online A (2015)
71	ORGANIZER	zh-ja	2014/08/28 12:10:13	215	0.636930	0.636930	0.636930	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	Other	Yes	Online B (2014)
72	ORGANIZER	zh-ja	2015/09/11 10:09:23	890	0.628290	0.628290	0.628290	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	Other	Yes	Online B (2015)
73	ORGANIZER	zh-ja	2014/08/29 18:45:03	239	0.626070	0.626070	0.626070	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	RBMT	No	RBMT A (2014)
74	ORGANIZER	zh-ja	2015/09/10 14:30:56	885	0.626070	0.626070	0.626070	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	Other	Yes	RBMT A (2015)
75	EIWA	zh-ja	2014/08/20 11:52:45	137	0.613730	0.613730	0.613730	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	RBMT	Yes	RBMT plus user dictionary
76	ORGANIZER	zh-ja	2014/08/29 18:48:29	242	0.586790	0.586790	0.586790	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	RBMT	No	RBMT D
77	TMU	zh-ja	2018/09/14 17:30:33	2343	0.512430	0.512430	0.512430	-	-	-	-	-	0.000000	0.000000	NMT	Yes	Unsupervised NMT with sub-character information. Both ASPEC and JPC 4.0 data (zh-ja) were also used as monolingual data in the training.

Notice:

This table is sorted by the leftmost segmenters. You can change the segmenter used to sort by clicking each segmenter link.
Adequacy-Fluency Metrics (AMFM) is a two-dimensional automatic evaluation metric for machine translation, designed to operate at the sentence level. It is based on adequacy and fluency, to decouple semantic and syntactic components of the translation process to provide a balanced view on translation quality.
AMFM is calculated without tokenizers.
The detail of AMFM is shown on the following paper: "Adequacy–Fluency Metrics: Evaluating MT in the Continuous Space Model Framework" [pdf]. Invited Talk in WAT2015 also helps understanding [slide].

HUMAN (WAT2022)

Notice:

HUMAN (WAT2022) is the result of the Pairwise Crowdsourcing Evaluation on WAT2022.
HUMAN (WAT2022) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.

HUMAN (WAT2021)

Notice:

HUMAN (WAT2021) is the result of the Pairwise Crowdsourcing Evaluation on WAT2021.
HUMAN (WAT2021) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.

HUMAN (WAT2020)

Notice:

HUMAN (WAT2020) is the result of the Pairwise Crowdsourcing Evaluation on WAT2020.
HUMAN (WAT2020) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.

HUMAN (WAT2019)

Notice:

HUMAN (WAT2019) is the result of the Pairwise Crowdsourcing Evaluation on WAT2019.
HUMAN (WAT2019) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.

HUMAN (WAT2018)

Notice:

HUMAN (WAT2018) is the result of the Pairwise Crowdsourcing Evaluation on WAT2018.
HUMAN (WAT2018) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.

HUMAN (WAT2017)

Notice:

HUMAN (WAT2017) is the result of the Pairwise Crowdsourcing Evaluation on WAT2017.
HUMAN (WAT2017) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.

HUMAN (WAT2016)

#	Team	Task	Date/Time	DataID	HUMAN	Method	Other Resources	System Description
1	Kyoto-U	zh-ja	2016/08/20 22:50:33	1256	63.750	NMT	No	voc: 30k ensemble of 3 independent model + reverse rescoring
2	Kyoto-U	zh-ja	2016/08/20 22:48:16	1255	56.000	NMT	No	src: 200k tgt: 50k 2-layers self-ensembling
3	bjtu_nlp	zh-ja	2016/08/12 12:50:38	1138	49.000	NMT	No	RNN Encoder-Decoder with attention mechanism, single model
4	UT-KAY	zh-ja	2016/08/20 07:12:52	1221	47.250	NMT	No	Ensemble of our NMT models with and without domain adaptation
5	UT-KAY	zh-ja	2016/08/20 07:09:54	1220	41.000	NMT	No	An end-to-end NMT with 512 dimensional single-layer LSTMs, UNK replacement, and domain adaptation
6	NICT-2	zh-ja	2016/08/05 18:05:03	1099	36.500	SMT	Yes	Phrase-based SMT with Preordering + Domain Adaptation (JPC and ASPEC) + Google 5-gram LM
7	EHR	zh-ja	2016/07/31 17:06:57	1063	32.500	SMT	Yes	LM-based merging of outputs of preordered word-based PBSMT(DL=6) and preordered character-based PBSMT(DL=6).
8	ORGANIZER	zh-ja	2016/11/16 11:28:00	1342	22.500	NMT	Yes	Online A (2016/11/14)
9	JAPIO	zh-ja	2016/08/19 16:44:49	1208	16.500	SMT	Yes	Phrase-based SMT with Preordering + JAPIO corpus + rule-based posteditor
10	ORGANIZER	zh-ja	2016/07/26 11:54:14	1043	-51.250	Other	Yes	Online A (2016)

Notice:

HUMAN (WAT2016) is the result of the Pairwise Crowdsourcing Evaluation on WAT2016.
HUMAN (WAT2016) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.

HUMAN (WAT2015)

#	Team	Task	Date/Time	DataID	HUMAN	Method	Other Resources	System Description
1	NAIST	zh-ja	2015/08/31 08:23:30	834	35.750	SMT	No	Travatar System with NeuralMT Reranking
2	EHR	zh-ja	2015/08/19 11:23:36	720	25.750	SMT and RBMT	Yes	System combination of RBMT with user dictionary plus SPE and phrase based SMT with preordering. Candidate selection by language model score.
3	NAIST	zh-ja	2015/08/31 08:26:31	835	25.750	SMT	No	Travatar System Baseline
4	Kyoto-U	zh-ja	2015/08/31 22:39:36	845	18.500	EBMT	No	KyotoEBMT system with bilingual RNNLM reranking
5	TOSHIBA	zh-ja	2015/07/23 15:14:53	508	18.000	SMT and RBMT	Yes	System combination SMT and RBMT(SPE) with RNNLM language model
6	ORGANIZER	zh-ja	2015/09/10 14:00:33	879	17.250	SMT	No	Tree-to-String SMT (2015)
7	Kyoto-U	zh-ja	2015/08/31 22:38:22	844	16.750	EBMT	No	KyotoEBMT system without reranking
8	BJTUNLP	zh-ja	2015/09/01 21:08:10	862	6.500	SMT	No	a dependency-to-string model for SMT
9	TOSHIBA	zh-ja	2015/07/28 16:27:32	525	-1.000	SMT and RBMT	Yes	RBMT with SPE(Statistical Post Editing) system
10	ORGANIZER	zh-ja	2015/08/25 18:58:08	776	-19.000	Other	Yes	Online A (2015)
11	ORGANIZER	zh-ja	2015/09/10 14:30:56	885	-28.000	Other	Yes	RBMT A (2015)

Notice:

HUMAN (WAT2015) is the result of the Pairwise Crowdsourcing Evaluation on WAT2015.
HUMAN (WAT2015) was evaluated by 5 different workers and the final decision is made by the voting of the judgements.
The detail of the evaluation can be found in the pdf document (PDF file).

HUMAN (WAT2014)

#	Team	Task	Date/Time	DataID	HUMAN	Method	Other Resources	System Description
1	NAIST	zh-ja	2014/07/31 11:42:31	120	50.750	SMT	No	Travatar-based Forest-to-String SMT System
2	NAIST	zh-ja	2014/08/01 17:33:01	124	38.000	SMT	No	Travatar-based Forest-to-String SMT System (Tuned BLEU+RIBES)
3	SAS_MT	zh-ja	2014/09/01 10:38:13	263	22.500	SMT	No	Syntactic reordering Hierarchical SMT （using SAS token tool）
4	ORGANIZER	zh-ja	2014/07/11 20:04:10	13	16.000	SMT	No	Tree-to-String SMT (2014)
5	EIWA	zh-ja	2014/08/20 11:56:00	138	15.000	SMT and RBMT	Yes	RBMT with user dictionary plus SPE(statistical post editing)
6	Kyoto-U	zh-ja	2014/09/01 21:33:23	268	7.500	EBMT	No	Our new baseline system after several modifications + 20-best parses, KN7, RNNLM reranking
7	Kyoto-U	zh-ja	2014/08/31 23:42:41	258	6.000	EBMT	No	Our new baseline system after several modifications.
8	ORGANIZER	zh-ja	2014/07/11 19:47:27	4	4.750	SMT	No	Hierarchical Phrase-based SMT (2014)
9	Sense	zh-ja	2014/08/26 15:17:49	200	-1.000	SMT	No	Character based SMT
10	ORGANIZER	zh-ja	2014/07/18 11:09:12	36	-21.750	Other	Yes	Online A (2014)
11	ORGANIZER	zh-ja	2014/08/29 18:45:03	239	-37.750	RBMT	No	RBMT A (2014)

Notice:

HUMAN (WAT2014) is the result of the Pairwise Crowdsourcing Evaluation on WAT2014.
HUMAN (WAT2014) was evaluated by 3 different workers and the final decision is made by the voting of the judgements.
The detail of the evaluation can be found in the pdf document (PDF file).

EVALUATION RESULTS USAGE POLICY

When you use the WAT evaluation results for any purpose such as:
- writing technical papers,
- making presentations about your system,
- advertising your MT system to the customers,
you can use the information about translation directions, scores (including both automatic and human evaluations) and ranks of your system among others. You can also use the scores of the other systems, but you MUST anonymize the other system's names. In addition, you can show the links (URLs) to the WAT evaluation result pages.

NICT (National Institute of Information and Communications Technology)
Kyoto University
Last Modified: 2018-08-02

WAT The Workshop on Asian Translation Evaluation Results

BLEU

RIBES

AMFM

HUMAN (WAT2022)

HUMAN (WAT2021)

HUMAN (WAT2020)

HUMAN (WAT2019)

HUMAN (WAT2018)

HUMAN (WAT2017)

HUMAN (WAT2016)

HUMAN (WAT2015)

HUMAN (WAT2014)

EVALUATION RESULTS USAGE POLICY

WAT

The Workshop on Asian Translation

Evaluation Results